Wiki source code of Continuous Integration

Last modified by Vincent Massol on 2023/04/25 14:02

Hide last authors
Vincent Massol 13.1 1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
Vincent Massol 12.1 4
Vincent Massol 13.1 5 = General =
Vincent Massol 3.2 6
Vincent Massol 23.2 7 XWiki has a [[Continuous Integration>>http://www.martinfowler.com/articles/continuousIntegration.html]] setup to ensure XWiki's code is built at all times (i.e at every code check in). This also allows developers to only build the module they want and they'll have the other XWiki module dependencies downloaded from the [[XWiki remote repository>>http://nexus.xwiki.org/nexus/#view-repositories]].
Jean-Vincent Drean 1.1 8
Vincent Massol 3.1 9 We use the following tools:
10
Vincent Massol 12.1 11 * [[GitHub>>https://github.com/xwiki]] is used as our [[SCM>>http://en.wikipedia.org/wiki/Source_Code_Management]]
12 * [[Maven>>http://maven.apache.org/]] is used for [[building XWiki>>Community.Building]]
13 * [[Jenkins>>http://jenkins-ci.org/]] is used as our Continuous Integration (CI) tool (set up at [[http://ci.xwiki.org/]])
Vincent Massol 21.1 14 * [[Docker>>https://www.docker.com/docker-community]] is used to run Jenkins agents (and incidentally to execute some functional tests over various configurations)
Vincent Massol 6.1 15
Vincent Massol 14.2 16 We use a technique called [[binary dependency build>>http://web.archive.org/web/20090423073100/http://blogs.codehaus.org/people/vmassol/archives/000953_binary_dependency_builds.html]] which allows to check out only the module a developer wishes to work on and he'll automatically get the latest fresh binary dependencies built by our CI tool.
Jean-Vincent Drean 1.1 17
Vincent Massol 32.1 18 = CI Requirements =
19
Vincent Massol 33.1 20 We're currently using Jenkins as our CI tool but it's possible to change. We're listing here our requirements for a CI tool, and with some of the solutions we've evaluated or are still evaluating.
Vincent Massol 32.1 21
Vincent Massol 33.1 22 == REQ1: Parallel Execution ==
Vincent Massol 32.1 23
Vincent Massol 34.1 24 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
25 |REQ1|Ability to run parallel and serial builds, with different parameters|(/)|(/)|(/)|(/)
Vincent Massol 33.1 26
27 * Jenkins: Using the ##parallel()## step
28 * GitLab CI: Using [[##trigger##, ##rules## and ##needs##>>https://docs.gitlab.com/ee/ci/pipelines/pipeline_architectures.html]]. See also [[multi-project triggering>>https://docs.gitlab.com/ee/ci/multi_project_pipelines.html]]. See also [[Child pipelines>>https://docs.gitlab.com/ee/ci/parent_child_pipelines.html]]. General doc for [[pipeline types>>https://docs.gitlab.com/ee/ci/pipelines/#types-of-pipelines]].
29 * GitHub Actions: Using different YML files in ##.github/workflows## (jobs run in parallel by default) and using [[##job.needs##>>https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobs]].
Vincent Massol 34.1 30 * CircleCI: See [[https://circleci.com/blog/decrease-your-build-times-by-running-jobs-in-parallel-with-workflows/]]
Vincent Massol 33.1 31
32 == REQ2: Scheduling ==
33
Vincent Massol 34.1 34 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
35 |REQ2|Ability to schedule builds (e.g. for the various docker tests) and to run the build at least once per month even if there's been no changes to the source|(!)|(!)|(/)|(/)
Vincent Massol 33.1 36
37 * Jenkins: Scheduling is supported but it's [[very complex to make it work automatically with a pipeline>>https://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]] since a repo can have only a single pipeline (i.e. a single ##Jenkinsfile##). Other solutions include using a Git submodule or another GitHub Organization Job.
38 * GitLab CI: It's possible to have [[schedules>>https://docs.gitlab.com/ee/ci/pipelines/schedules.html]] but note that the schedule is not part of the YML file but something you define in the GitLab CI configuration UI. However, it's also possible to [[use the API>>https://docs.gitlab.com/ee/api/pipeline_schedules.html#create-a-new-pipeline-schedule]].
39 * Github Actions: Using the [[##on## element with a ##schedule##>>https://help.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule]].
Vincent Massol 35.1 40 * CircleCI: See https://circleci.com/docs/2.0/configuration-reference/#schedule
Vincent Massol 33.1 41
42 == REQ3: Sharing Pipeline Code ==
43
Vincent Massol 34.1 44 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
45 |REQ3|Ability to share pipeline code between builds for the same repositoryy and for different repositories. And ability to pass properties to this common code.|(/)|(/)|(/)|(/)
Vincent Massol 33.1 46
47 * Jenkins: Using a shared pipeline
48 * GitLab CI: Using an [[##include##>>https://docs.gitlab.com/ee/ci/yaml/README.html#include]].
49 * GitHub Actions: With a [[custom action>>https://help.github.com/en/actions/building-actions]] at least (feels a bit complex).
50
51 == REQ4: View Test Results ==
52
Vincent Massol 34.1 53 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
54 |REQ4|Ability to view test results and failing test logs (test logs + console/xwiki logs).|(/)|(x)|(!)|(?)
Vincent Massol 33.1 55
56 * Jenkins: Built in.
57 * GitLab CI: Built in but [[not working ATM because of performance issues>>https://gitlab.com/groups/gitlab-org/-/epics/2854]].
Vincent Massol 35.2 58 * GitHub Actions: Needs a custom action ATM. Found 2 but didn't work when I tested them: [[check run reporter>>https://www.check-run-reporter.com/]] ([[problem reported>>https://github.com/check-run-reporter/feedback/issues/3]]) and [[Scope>>https://scope.dev/]] (2 problems: unknown cost + got stuck when I tested it on commons and I had to kill the workflow after 1 hour).
Vincent Massol 36.1 59 ** Update: [[new ScaCap surefire report action>>https://github.com/ScaCap/action-surefire-report]] to test.
Vincent Massol 33.1 60
61 == REQ5: Disable Concurrent Builds ==
62
Vincent Massol 34.1 63 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
64 |REQ5|(optional) Ability to aggregate commits before a build is executed, i.e. disable concurrent builds. Note that this is more a limitation of the number of agents we can have vs the duration of the full xwiki build. If we get a large number of agents, it may be a less important requirement (or not one at all).|(/)|(?)|?/)|(/)
Vincent Massol 33.1 65
Vincent Massol 34.1 66 * Jenkins: Using the
67 * GitLab: (?)
68 * GitHubActions: (?)
69 * CircleCI: In the Project CI's configuration. It's called "Auto-cancel redundant builds".
70
Vincent Massol 33.1 71 == REQ6: Custom Docker Image ==
72
Vincent Massol 34.1 73 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
74 |REQ6|Supports using a custom docker build image since [[XWiki has one>>https://github.com/xwiki/xwiki-docker-build/tree/master/build]]|(/)|(/)|(/)|(/)
Vincent Massol 33.1 75
76 * Jenkins: Using the Docker Cloud plugin.
77 * GitLab CI: Built in using the ##image## YML element.
78 * GitHub Actions: Buimt in using the [[#container##>>https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idcontainer]] YML element.
79
80 Notes:
81 * That this requirement allows use to control the Java, Maven and more generally our full build environment.
82 * With Jenkins and GitHub Actions the docker daemon socket is shared allowing our docker tests to work. Needs to be checked for GitLab CI.
83
84 == REQ7: Parse pom.xml ==
85
Vincent Massol 34.1 86 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
87 |REQ7|Ability to easily read/parse the current pom.xml to automatically find the version of Java to use and the memory parameters to set for the JVM.|(/)|(?)|(?)|(?)
Vincent Massol 33.1 88
89 * Jenkins: Using the ##readMavenPom## step.
90
91 == REQ8: Shell Scripts ==
92
Vincent Massol 34.1 93 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
94 |REQ8|Ability to execute shell scripts before/after the build, inside our custom build image. For example to execute VNC and export the DISPLAY (for the functional tests not based on Docker)|(/)|(/)|(/)|(/)
Vincent Massol 33.1 95
96 * Jenkins: In the Jenkinsfile/shared pipeline
97 * GitLab CI: Built in, using the ##script## step.
98 * GitHub Actions: Built in, using the ##run## step.
99
100 == REQ9: Archive Artifacts ==
101
Vincent Massol 34.1 102 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
103 |REQ9|Ability to archive artifacts and ability to save the failing test screenshots and videos as artifacts.|(/)|(/)|(/)|(/)
Vincent Massol 33.1 104
105 * Jenkins: Feature of the Maven plugin + ##archiveArtifacts## step.
106 * GitLab CI: Using the ##artifacts## step.
107 * GitHub Actions: Using the [[##upload-artifact## custom action>>https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts]].
108
109 == REQ10: Build Parameters ==
110
Vincent Massol 34.1 111 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
112 |REQ10|Ability to have different build parameters (different maven parameters for example) between master and branches/PR. For example, we want to run "mvn deploy" on master and "mvn install" on branches.|(/)|(/)|(/)|(?)
Vincent Massol 33.1 113
114 * Jenkins: Using method parameters in custom pipeline steps.
115 * GitLab CI: Using [[##variables##>>https://docs.gitlab.com/ee/ci/variables/]]
116 * GitHub Actions: Parameters of a custom action.
117
118 == REQ11: Supports GitHub ==
119
Vincent Massol 34.1 120 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
121 |REQ11|Must work with sources located on GitHub.|(/)|(/)|(/)|(/)
Vincent Massol 33.1 122
123 * Jenkins: GitHub webhook.
124 * GitLab CI: Automatically [[creates a mirrored GitHub repo>>https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html]].
125 * GitHub Actions: By design ;)
126
127 == REQ12: Build PRs ==
128
Vincent Massol 34.1 129 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
130 |REQ12|Ability to run the build on PRs.|(/)|(/)|(/)|(/)
Vincent Massol 33.1 131
132 == REQ13: Timeout ==
133
Vincent Massol 34.1 134 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
135 |REQ13|Ability to kill builds after a given timeout.|(/)|(/)|(?)|(?)
Vincent Massol 33.1 136
137 * Jenkins: Using the ##timeout## step.
138 * GitLab CI: Using the [[##timeout## step>>https://docs.gitlab.com/ee/ci/yaml/#timeout]].
139 * GitHub Actions: (?)
140
141 == REQ14: Secrets ==
142
Vincent Massol 34.1 143 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
144 |REQ14|Handle secrets so that they are not in the pipeline scripts (e.g. we need to push to sonarcloud or deploy to nexus.xwiki.org or to maven central).|(/)|(/)|(/)|(?)
Vincent Massol 33.1 145
146 == REQ15: Mail ==
147
Vincent Massol 34.1 148 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
149 |REQ15|Ability to send mail and customize the mail template.|(/)|(?)|(?)|?)
Vincent Massol 33.1 150
151 * Jenkins: Using the ##emailext## step from the Extended Mail plugin.
152 * GitLab CI: (?)
153 * GitHub Actions: (?)
154
155 == REQ16: Test Screenshots ==
156
Vincent Massol 34.1 157 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
158 |REQ16|(Nice to have) Ability to attach failing test screenshot to failing test report pages|(/)|(?)|(?)|(?)
Vincent Massol 33.1 159
160 == REQ17: Handle Flickers ==
161
Vincent Massol 34.1 162 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
163 |REQ17|Ability to check failing test (i.e. have an API to retrieve failing tests) so that we can compare with known flickers on our JIRA and decide to send a failing mail or not + mark the test as flickering in the CI Test Report UI.|(/)|(?)|(?)|(?)
Vincent Massol 33.1 164
Vincent Massol 23.1 165 = Jenkins Agent Image =
166
Vincent Massol 26.4 167 XWiki has its own [[Jenkins Agent Docker image>>https://github.com/xwiki/xwiki-jenkins-slave]] that is used by Jenkins master to spawn agents.
Vincent Massol 23.1 168
Thomas Mortagne 38.1 169 = Maven settings
170
171 The Maven settings used during Jenkins builds are located in each agent and need to be kept in sync.
172
Vincent Massol 38.2 173 Among other things the ##settings.xml## expose two "configurable" profiles:
174 * ##repository-all##: the default profile which enable all Maven repositories used by Maven. It can be disabled with ##-P-repository-all##
Thomas Mortagne 38.1 175 * ##repository-snapshots##: not enabled by default. It's generally used when you disabled ##repository-all## but still want to use XWiki own snapshots as dependencies
176
Vincent Massol 18.1 177 = Jenkins Pipelines =
Vincent Massol 16.2 178
Vincent Massol 18.1 179 == Main Pipeline ==
Vincent Massol 16.2 180
Vincent Massol 18.1 181 We have developed a {{scm project="xwiki-jenkins-pipeline" path="vars/xwikiModule.groovy"}}Jenkins Pipeline script{{/scm}} to build XWiki projects (can be used for core and contrib projects). It has the following features:
182
Vincent Massol 16.2 183 * Automatically use the right version of Java (and proper memory configuration)
184 * Use Jenkins's Xvnc plugin to have a Display for functional tests
185 * Use the "deploy" maven goal for "master" and "stable-*" branches only and "install" for the rest
186 * [[Attach the screenshot>>http://massol.myxwiki.org/xwiki/bin/view/Blog/AttachFailingTestPipeline]] of a failing XWiki Selenium test to the failed test's description
187 * Check for false positives for known cases of failures not related to code + [[check for test flickers>>http://massol.myxwiki.org/xwiki/bin/view/Blog/FlakyTestTool]]
188 * Send mails for build failures
Vincent Massol 29.1 189 * Supports passing custom job properties. See below for an example of how to use this to trigger nightly build of Docker-based configuration tests.
Vincent Massol 16.2 190
Vincent Massol 28.1 191 Note that if you use a "Github Organization" job type in Jenkins you'll get branch management for free, i.e. automatically build branches when new branches are added (or Pull Requests), and automatically stop building branches when they are removed from the SCM.
192
Vincent Massol 18.1 193 To use is for a project, add a ##Jenkinsfile## with the following minimal configuration:
Vincent Massol 16.2 194
Vincent Massol 18.1 195 {{code language="groovy"}}
196 /*
197 * See the NOTICE file distributed with this work for additional
198 * information regarding copyright ownership.
199 *
200 * This is free software; you can redistribute it and/or modify it
201 * under the terms of the GNU Lesser General Public License as
202 * published by the Free Software Foundation; either version 2.1 of
203 * the License, or (at your option) any later version.
204 *
205 * This software is distributed in the hope that it will be useful,
206 * but WITHOUT ANY WARRANTY; without even the implied warranty of
207 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
208 * Lesser General Public License for more details.
209 *
210 * You should have received a copy of the GNU Lesser General Public
211 * License along with this software; if not, write to the Free
212 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
213 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
214 */
Vincent Massol 16.2 215
Vincent Massol 18.1 216 // It's assumed that Jenkins has been configured to implicitly load the vars/xwikiModule.groovy library which exposes
217 // the "xwikiModule" global function/DSL.
218 // Note that the version used is the one defined in Jenkins but it can be overridden as follows:
219 // @Library("XWiki@<branch, tag, sha1>") _
220 // See https://github.com/jenkinsci/workflow-cps-global-lib-plugin for details.
221
222 xwikiModule {
223 }
Vincent Massol 18.2 224 {{/code}}
Vincent Massol 18.1 225
Vincent Massol 43.1 226 Example with SonarQube execution (XWiki < 14.0):
227
228 {{code language="groovy"}}
229 xwikiModule {
230 goals = 'clean deploy jacoco:report sonar:sonar'
231 sonar = true
232 // Force Java 11 even though we depend on XS requiring only Java 8 because we run SonarQube and this requires
233 // Java 11.
234 javaTool = 'java11'
235 }
236 {{/code}}
237
238 Example with SonarQube execution (XWiki >= 14.0):
239
240 {{code language="groovy"}}
241 xwikiModule {
242 goals = 'clean deploy jacoco:report sonar:sonar'
243 sonar = true
244 }
245 {{/code}}
246
247 Example with Docker tests:
248
249 {{code language="groovy"}}
250 node('docker') {
251 xwikiBuild {
252 xvnc = false
253 profiles = 'integration-tests,docker'
254 }
255 }
256 {{/code}}
257
Vincent Massol 18.1 258 For more elaborate configuration, see:
259 * {{scm project="xwiki-commons" path="Jenkinsfile"}}xwiki-commons's Jenkinsfile{{/scm}}
260 * {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}}
261
Vincent Massol 29.1 262 === Nightly Configuration Tests ===
263
Vincent Massol 41.1 264 Here's what we do in XWiki Platform to not only have a pipeline triggered whenever there's a commit but also to separate environment tests so that they are executed only during the night:
265 * Have 2 files, each containing the pipeline to execute, and define 2 "Organization Folder" jobs in Jenkins (each configured to execute a given Jenkins file):
266 ** {{scm project="xwiki-platform" path="Jenkinsfile"}}Jenkinsfile{{/scm}}: the main pipeline that we use to run the build whenever there's a commit.
267 ** {{scm project="xwiki-platform" path="JenkinsfileEnvironmentTests"}}JenkinsfileEnvironmentTests{{/scm}}: the pipeline that executes the environment tests using a crontab
268 * The strategy for the ##JenkinsfileEnvironmentTests## file is explained in details in [[this blog post>>http://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]]. In short:
269 ** Step 1: In the ##xwikiBuild## step, pass the ##jobProperties## configuration with some scheduler cron. For example:(((
Vincent Massol 29.1 270 {{code language="groovy"}}
271 xwikiBuild(map.name) {
272 ...
273 jobProperties = getCustomJobProperties()
274 ...
275 }
276
277 private def getCustomJobProperties()
278 {
279 // Define a scheduler job to execute the Docker-based functional tests at regular intervals. We do this since they
280 // take time to execute and thus we cannot run them all the time.
281 // This scheduler job will pass the "type" parameter to this Jenkinsfile when it executes, allowing us to decide if
282 // we run the standard builds or the docker ones.
283 // Note: it's the xwikiBuild() calls from the standard builds that will set the jobProperties and thus set up the
284 // job parameter + the crons. It would be better to set the properties directly in this Jenkinsfile but we haven't
285 // found a way to merge properties and calling the properties() step will override any pre-existing properties.
286 return [
287 parameters([string(defaultValue: 'standard', description: 'Job type', name: 'type')]),
288 pipelineTriggers([
289 parameterizedCron('''@midnight %type=docker-latest
290 @weekly %type=docker-all
291 @monthly %type=docker-unsupported'''),
292 cron("@monthly")
293 ])
294 ]
295 }
296 {{/code}}
297 )))
Vincent Massol 41.1 298 ** Step 2: Based on the parametrized ##type## value, decide to execute Docker-based tests:(((
Vincent Massol 29.1 299 {{code language="groovy"}}
300 ...
301 if (params.type && params.type == 'docker-latest') {
302 buildDocker('docker-latest')
303 }
304 ...
305 private void buildDocker(type)
306 {
307 def dockerConfigurationList
308 def dockerModuleList
309 def customJobProperties
310 node() {
311 // Checkout platform to find all docker configurations and test modules so that we can then parallelize executions
312 // of configs and modules across Jenkins agents.
313 checkout skipChangeLog: true, scm: scm
314 dockerConfigurationList = dockerConfigurations(type)
315 if (type == 'docker-unsupported') {
316 dockerModuleList = ['xwiki-platform-core/xwiki-platform-menu']
317 } else {
318 dockerModuleList = dockerModules()
319 }
320 customJobProperties = getCustomJobProperties()
321 }
322
323 xwikiDockerBuild {
324 configurations = dockerConfigurationList
325 modules = dockerModuleList
326 // Make sure that we don't reset the job properties!
327 jobProperties = customJobProperties
328 }
329 }
330 {{/code}}
331 )))
Vincent Massol 41.1 332 ** Step 3: Configure the job to suppress being triggered when there's a commit (so that it's only triggered based on its crontab):(((
333 {{image reference="scm-trigger-suppress.png"/}}
334 )))
Vincent Massol 29.1 335
Vincent Massol 41.1 336 Pros of this approach:
337 * No need to handle branches, done automatically by the Jenkins files (creation and deletion). Also works for PRs.
338 * No messy history that we would have if we were using only a single Jenkins file.
Vincent Massol 29.1 339
Vincent Massol 19.1 340 == Clover Pipeline ==
341
Vincent Massol 36.3 342 Used to generate the global test coverage for XWiki (over all XWiki Standard repositories). See [[Test Coverage>>Community.Testing.TestCoverage.WebHome#HUsingClover2BJenkins]].
Vincent Massol 19.1 343
Vincent Massol 13.1 344 = Maintainer's guide =
Jean-Vincent Drean 1.1 345
Vincent Massol 13.1 346 == Prerequisites ==
Jean-Vincent Drean 1.1 347
fciubotaru 14.3 348 * You need to have an account on [[maven.xwiki.org>>http://maven.xwiki.org/]] (this is the machine hosting XWiki's remote repository) and you'll need a key setup on your account there so that you can ssh to it without having to enter username or password.
Vincent Massol 12.1 349 * You need to have an administrator account on [[ci.xwiki.org>>http://ci.xwiki.org/signup]].
Jean-Vincent Drean 1.1 350
Vincent Massol 13.1 351 == Functional tests ==
Jean-Vincent Drean 1.1 352
Vincent Massol 13.1 353 === Debugging ===
Jean-Vincent Drean 2.1 354
Vincent Massol 30.1 355 **Start remote Firefox**
356
357 * For example, to start firefox on machine ##ks4##: {{code}}ssh -X <username>@<ks4 hostname>{{/code}}
358 ** For this to work you need to ensure that ##X11Forwarding## is set to ##yes## in ##/etc/ssh/sshd_config## on the remote machine:(((
359 {{code}}
360 sudo vi /etc/ssh/sshd_config
361 {{/code}}
Vincent Massol 31.1 362
Vincent Massol 31.2 363 Then reload sshd: {{code}}sudo /etc/init.d/ssh reload{{/code}}
Vincent Massol 30.1 364 )))
365
Vincent Massol 12.1 366 **Connect to the XVNC server**
367
Vincent Massol 27.1 368 * Establish a SSH tunnel between your computer and the server on port 5901 (//ssh -L 5901:localhost:5901 <user@ip address>//)
Jean-Vincent Drean 2.1 369 * Use your favorite VNC client to connect to the XVNC server
370 ** Host : localhost
371 ** Display : 1
372 ** Password : same password as for XWiki SAS wifi access points
373 * See the functional tests live.
374
Vincent Massol 27.1 375 * Work in progress to try to connect to an agent not exposing an IP address externally (by going through maven.xwiki.org):
376 ** {{code}}ssh -L 5901:192.168.1.119:5901 -o ProxyCommand="ssh [email protected] nc -w1 %h %p" [email protected] -i ~/.ssh/id_rsa_mavenxwikiorg{{/code}}
377 ** Where ##~/.ssh/id_rsa_mavenxwikiorg## is the private key on maven.xwiki.org, used to connect to 192.168.1.119 (ip of agent 2-4 in this example)
378 ** Not working yet since I can't connect with my local vnc client.
Simon Urli 37.1 379
380 = CI infrastructure =
381
382 == Nodes requirements ==
383
384 Here's the minimum requirements for each CI node:
Vincent Massol 37.3 385 * 2 CPUs
386 * 8GB of memory
387 * SSD hard drive with XXXGB space
Simon Urli 37.1 388
389 == Current configurations ==
390
Vincent Massol 40.1 391 Our [[current CI>>https://ci.xwiki.org]] is running with 7 docker nodes running each one docker build CI image (##a1## to ##a7##). Each agent has the following configuration:
392 * CPU: ((({{code language="none"}}
393 $ lscpu
394 Architecture: x86_64
395 CPU op-mode(s): 32-bit, 64-bit
396 Byte Order: Little Endian
397 Address sizes: 40 bits physical, 48 bits virtual
398 CPU(s): 4
399 On-line CPU(s) list: 0-3
400 Thread(s) per core: 1
401 Core(s) per socket: 1
402 Socket(s): 4
403 NUMA node(s): 1
404 Vendor ID: GenuineIntel
405 CPU family: 6
406 Model: 60
407 Model name: Intel Core Processor (Haswell, no TSX)
408 Stepping: 1
409 CPU MHz: 2599.990
410 BogoMIPS: 5199.98
411 Virtualization: VT-x
412 Hypervisor vendor: KVM
413 Virtualization type: full
414 L1d cache: 32K
415 L1i cache: 32K
416 L2 cache: 4096K
417 L3 cache: 16384K
418 NUMA node0 CPU(s): 0-3
419 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear
420 {{/code}})))
421 * Disk: ((({{code}}
422 $ lsblk
423 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
424 sda 8:0 0 160G 0 disk
425 └─sda1 8:1 0 160G 0 part /
426 {{/code}})))
Vincent Massol 37.2 427
Vincent Massol 39.1 428 Each agent needs to be [[properly configured>>https://github.com/xwiki/xwiki-docker-build/tree/master/build#setup-for-cixwikiorg]].
429
Simon Urli 37.1 430

Get Connected