Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 = General =
6
7 XWiki has a [[Continuous Integration>>http://www.martinfowler.com/articles/continuousIntegration.html]] setup to ensure XWiki's code is built at all times (i.e at every code check in). This also allows developers to only build the module they want and they'll have the other XWiki module dependencies downloaded from the [[XWiki remote repository>>http://nexus.xwiki.org/nexus/#view-repositories]].
8
9 We use the following tools:
10
11 * [[GitHub>>https://github.com/xwiki]] is used as our [[SCM>>http://en.wikipedia.org/wiki/Source_Code_Management]]
12 * [[Maven>>http://maven.apache.org/]] is used for [[building XWiki>>Community.Building]]
13 * [[Jenkins>>http://jenkins-ci.org/]] is used as our Continuous Integration (CI) tool (set up at [[http://ci.xwiki.org/]])
14 * [[Docker>>https://www.docker.com/docker-community]] is used to run Jenkins agents (and incidentally to execute some functional tests over various configurations)
15
16 We use a technique called [[binary dependency build>>http://web.archive.org/web/20090423073100/http://blogs.codehaus.org/people/vmassol/archives/000953_binary_dependency_builds.html]] which allows to check out only the module a developer wishes to work on and he'll automatically get the latest fresh binary dependencies built by our CI tool.
17
18 = CI Requirements =
19
20 We're currently using Jenkins as our CI tool but it's possible to change. We're listing here our requirements for a CI tool, and with some of the solutions we've evaluated or are still evaluating.
21
22 == REQ1: Parallel Execution ==
23
24 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
25 |REQ1|Ability to run parallel and serial builds, with different parameters|(/)|(/)|(/)|(/)
26
27 * Jenkins: Using the ##parallel()## step
28 * GitLab CI: Using [[##trigger##, ##rules## and ##needs##>>https://docs.gitlab.com/ee/ci/pipelines/pipeline_architectures.html]]. See also [[multi-project triggering>>https://docs.gitlab.com/ee/ci/multi_project_pipelines.html]]. See also [[Child pipelines>>https://docs.gitlab.com/ee/ci/parent_child_pipelines.html]]. General doc for [[pipeline types>>https://docs.gitlab.com/ee/ci/pipelines/#types-of-pipelines]].
29 * GitHub Actions: Using different YML files in ##.github/workflows## (jobs run in parallel by default) and using [[##job.needs##>>https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobs]].
30 * CircleCI: See [[https://circleci.com/blog/decrease-your-build-times-by-running-jobs-in-parallel-with-workflows/]]
31
32 == REQ2: Scheduling ==
33
34 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
35 |REQ2|Ability to schedule builds (e.g. for the various docker tests) and to run the build at least once per month even if there's been no changes to the source|(!)|(!)|(/)|(/)
36
37 * Jenkins: Scheduling is supported but it's [[very complex to make it work automatically with a pipeline>>https://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]] since a repo can have only a single pipeline (i.e. a single ##Jenkinsfile##). Other solutions include using a Git submodule or another GitHub Organization Job.
38 * GitLab CI: It's possible to have [[schedules>>https://docs.gitlab.com/ee/ci/pipelines/schedules.html]] but note that the schedule is not part of the YML file but something you define in the GitLab CI configuration UI. However, it's also possible to [[use the API>>https://docs.gitlab.com/ee/api/pipeline_schedules.html#create-a-new-pipeline-schedule]].
39 * Github Actions: Using the [[##on## element with a ##schedule##>>https://help.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule]].
40 * CircleCI: See https://circleci.com/docs/2.0/configuration-reference/#schedule
41
42 == REQ3: Sharing Pipeline Code ==
43
44 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
45 |REQ3|Ability to share pipeline code between builds for the same repositoryy and for different repositories. And ability to pass properties to this common code.|(/)|(/)|(/)|(/)
46
47 * Jenkins: Using a shared pipeline
48 * GitLab CI: Using an [[##include##>>https://docs.gitlab.com/ee/ci/yaml/README.html#include]].
49 * GitHub Actions: With a [[custom action>>https://help.github.com/en/actions/building-actions]] at least (feels a bit complex).
50
51 == REQ4: View Test Results ==
52
53 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
54 |REQ4|Ability to view test results and failing test logs (test logs + console/xwiki logs).|(/)|(x)|(!)|(?)
55
56 * Jenkins: Built in.
57 * GitLab CI: Built in but [[not working ATM because of performance issues>>https://gitlab.com/groups/gitlab-org/-/epics/2854]].
58 * GitHub Actions: Needs a custom action ATM. Found 2 but didn't work when I tested them: [[check run reporter>>https://www.check-run-reporter.com/]] ([[problem reported>>https://github.com/check-run-reporter/feedback/issues/3]]) and [[Scope>>https://scope.dev/]] (2 problems: unknown cost + got stuck when I tested it on commons and I had to kill the workflow after 1 hour).
59 ** Update: [[new ScaCap surefire report action>>https://github.com/ScaCap/action-surefire-report]] to test.
60
61 == REQ5: Disable Concurrent Builds ==
62
63 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
64 |REQ5|(optional) Ability to aggregate commits before a build is executed, i.e. disable concurrent builds. Note that this is more a limitation of the number of agents we can have vs the duration of the full xwiki build. If we get a large number of agents, it may be a less important requirement (or not one at all).|(/)|(?)|?/)|(/)
65
66 * Jenkins: Using the
67 * GitLab: (?)
68 * GitHubActions: (?)
69 * CircleCI: In the Project CI's configuration. It's called "Auto-cancel redundant builds".
70
71 == REQ6: Custom Docker Image ==
72
73 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
74 |REQ6|Supports using a custom docker build image since [[XWiki has one>>https://github.com/xwiki/xwiki-docker-build/tree/master/build]]|(/)|(/)|(/)|(/)
75
76 * Jenkins: Using the Docker Cloud plugin.
77 * GitLab CI: Built in using the ##image## YML element.
78 * GitHub Actions: Buimt in using the [[#container##>>https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idcontainer]] YML element.
79
80 Notes:
81 * That this requirement allows use to control the Java, Maven and more generally our full build environment.
82 * With Jenkins and GitHub Actions the docker daemon socket is shared allowing our docker tests to work. Needs to be checked for GitLab CI.
83
84 == REQ7: Parse pom.xml ==
85
86 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
87 |REQ7|Ability to easily read/parse the current pom.xml to automatically find the version of Java to use and the memory parameters to set for the JVM.|(/)|(?)|(?)|(?)
88
89 * Jenkins: Using the ##readMavenPom## step.
90
91 == REQ8: Shell Scripts ==
92
93 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
94 |REQ8|Ability to execute shell scripts before/after the build, inside our custom build image. For example to execute VNC and export the DISPLAY (for the functional tests not based on Docker)|(/)|(/)|(/)|(/)
95
96 * Jenkins: In the Jenkinsfile/shared pipeline
97 * GitLab CI: Built in, using the ##script## step.
98 * GitHub Actions: Built in, using the ##run## step.
99
100 == REQ9: Archive Artifacts ==
101
102 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
103 |REQ9|Ability to archive artifacts and ability to save the failing test screenshots and videos as artifacts.|(/)|(/)|(/)|(/)
104
105 * Jenkins: Feature of the Maven plugin + ##archiveArtifacts## step.
106 * GitLab CI: Using the ##artifacts## step.
107 * GitHub Actions: Using the [[##upload-artifact## custom action>>https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts]].
108
109 == REQ10: Build Parameters ==
110
111 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
112 |REQ10|Ability to have different build parameters (different maven parameters for example) between master and branches/PR. For example, we want to run "mvn deploy" on master and "mvn install" on branches.|(/)|(/)|(/)|(?)
113
114 * Jenkins: Using method parameters in custom pipeline steps.
115 * GitLab CI: Using [[##variables##>>https://docs.gitlab.com/ee/ci/variables/]]
116 * GitHub Actions: Parameters of a custom action.
117
118 == REQ11: Supports GitHub ==
119
120 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
121 |REQ11|Must work with sources located on GitHub.|(/)|(/)|(/)|(/)
122
123 * Jenkins: GitHub webhook.
124 * GitLab CI: Automatically [[creates a mirrored GitHub repo>>https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html]].
125 * GitHub Actions: By design ;)
126
127 == REQ12: Build PRs ==
128
129 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
130 |REQ12|Ability to run the build on PRs.|(/)|(/)|(/)|(/)
131
132 == REQ13: Timeout ==
133
134 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
135 |REQ13|Ability to kill builds after a given timeout.|(/)|(/)|(?)|(?)
136
137 * Jenkins: Using the ##timeout## step.
138 * GitLab CI: Using the [[##timeout## step>>https://docs.gitlab.com/ee/ci/yaml/#timeout]].
139 * GitHub Actions: (?)
140
141 == REQ14: Secrets ==
142
143 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
144 |REQ14|Handle secrets so that they are not in the pipeline scripts (e.g. we need to push to sonarcloud or deploy to nexus.xwiki.org or to maven central).|(/)|(/)|(/)|(?)
145
146 == REQ15: Mail ==
147
148 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
149 |REQ15|Ability to send mail and customize the mail template.|(/)|(?)|(?)|?)
150
151 * Jenkins: Using the ##emailext## step from the Extended Mail plugin.
152 * GitLab CI: (?)
153 * GitHub Actions: (?)
154
155 == REQ16: Test Screenshots ==
156
157 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
158 |REQ16|(Nice to have) Ability to attach failing test screenshot to failing test report pages|(/)|(?)|(?)|(?)
159
160 == REQ17: Handle Flickers ==
161
162 |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI
163 |REQ17|Ability to check failing test (i.e. have an API to retrieve failing tests) so that we can compare with known flickers on our JIRA and decide to send a failing mail or not + mark the test as flickering in the CI Test Report UI.|(/)|(?)|(?)|(?)
164
165 = Jenkins Agent Image =
166
167 XWiki has its own [[Jenkins Agent Docker image>>https://github.com/xwiki/xwiki-jenkins-slave]] that is used by Jenkins master to spawn agents.
168
169 = Maven settings
170
171 The Maven settings used during Jenkins builds are located in each agent and need to be kept in sync.
172
173 Among other things the ##settings.xml## expose two "configurable" profiles:
174 * ##repository-all##: the default profile which enable all Maven repositories used by Maven. It can be disabled with ##-P-repository-all##
175 * ##repository-snapshots##: not enabled by default. It's generally used when you disabled ##repository-all## but still want to use XWiki own snapshots as dependencies
176
177 = Jenkins Pipelines =
178
179 == Main Pipeline ==
180
181 We have developed a {{scm project="xwiki-jenkins-pipeline" path="vars/xwikiModule.groovy"}}Jenkins Pipeline script{{/scm}} to build XWiki projects (can be used for core and contrib projects). It has the following features:
182
183 * Automatically use the right version of Java (and proper memory configuration)
184 * Use Jenkins's Xvnc plugin to have a Display for functional tests
185 * Use the "deploy" maven goal for "master" and "stable-*" branches only and "install" for the rest
186 * [[Attach the screenshot>>http://massol.myxwiki.org/xwiki/bin/view/Blog/AttachFailingTestPipeline]] of a failing XWiki Selenium test to the failed test's description
187 * Check for false positives for known cases of failures not related to code + [[check for test flickers>>http://massol.myxwiki.org/xwiki/bin/view/Blog/FlakyTestTool]]
188 * Send mails for build failures
189 * Supports passing custom job properties. See below for an example of how to use this to trigger nightly build of Docker-based configuration tests.
190
191 Note that if you use a "Github Organization" job type in Jenkins you'll get branch management for free, i.e. automatically build branches when new branches are added (or Pull Requests), and automatically stop building branches when they are removed from the SCM.
192
193 To use is for a project, add a ##Jenkinsfile## with the following minimal configuration:
194
195 {{code language="groovy"}}
196 /*
197 * See the NOTICE file distributed with this work for additional
198 * information regarding copyright ownership.
199 *
200 * This is free software; you can redistribute it and/or modify it
201 * under the terms of the GNU Lesser General Public License as
202 * published by the Free Software Foundation; either version 2.1 of
203 * the License, or (at your option) any later version.
204 *
205 * This software is distributed in the hope that it will be useful,
206 * but WITHOUT ANY WARRANTY; without even the implied warranty of
207 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
208 * Lesser General Public License for more details.
209 *
210 * You should have received a copy of the GNU Lesser General Public
211 * License along with this software; if not, write to the Free
212 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
213 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
214 */
215
216 // It's assumed that Jenkins has been configured to implicitly load the vars/xwikiModule.groovy library which exposes
217 // the "xwikiModule" global function/DSL.
218 // Note that the version used is the one defined in Jenkins but it can be overridden as follows:
219 // @Library("[email protected]<branch, tag, sha1>") _
220 // See https://github.com/jenkinsci/workflow-cps-global-lib-plugin for details.
221
222 xwikiModule {
223 }
224 {{/code}}
225
226 For more elaborate configuration, see:
227 * {{scm project="xwiki-commons" path="Jenkinsfile"}}xwiki-commons's Jenkinsfile{{/scm}}
228 * {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}}
229
230 === Nightly Configuration Tests ===
231
232 Heres an example of how you can setup your ##Jenkinsfile## to automatically have execution of Docker-based configuration tests every night. This example is taken from {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}}. For more details see [[this blog post>>http://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]].
233
234 * Step 1: In the ##xwikiBuild## step, pass the ##jobProperties## configuration with some scheduler cron. For example:(((
235 {{code language="groovy"}}
236 xwikiBuild(map.name) {
237 ...
238 jobProperties = getCustomJobProperties()
239 ...
240 }
241
242 private def getCustomJobProperties()
243 {
244 // Define a scheduler job to execute the Docker-based functional tests at regular intervals. We do this since they
245 // take time to execute and thus we cannot run them all the time.
246 // This scheduler job will pass the "type" parameter to this Jenkinsfile when it executes, allowing us to decide if
247 // we run the standard builds or the docker ones.
248 // Note: it's the xwikiBuild() calls from the standard builds that will set the jobProperties and thus set up the
249 // job parameter + the crons. It would be better to set the properties directly in this Jenkinsfile but we haven't
250 // found a way to merge properties and calling the properties() step will override any pre-existing properties.
251 return [
252 parameters([string(defaultValue: 'standard', description: 'Job type', name: 'type')]),
253 pipelineTriggers([
254 parameterizedCron('''@midnight %type=docker-latest
255 @weekly %type=docker-all
256 @monthly %type=docker-unsupported'''),
257 cron("@monthly")
258 ])
259 ]
260 }
261 {{/code}}
262 )))
263 * Step 2: Based on the parametrized ##type## value, decide to execute Docker-based tests:(((
264 {{code language="groovy"}}
265 ...
266 if (params.type && params.type == 'docker-latest') {
267 buildDocker('docker-latest')
268 }
269 ...
270 private void buildDocker(type)
271 {
272 def dockerConfigurationList
273 def dockerModuleList
274 def customJobProperties
275 node() {
276 // Checkout platform to find all docker configurations and test modules so that we can then parallelize executions
277 // of configs and modules across Jenkins agents.
278 checkout skipChangeLog: true, scm: scm
279 dockerConfigurationList = dockerConfigurations(type)
280 if (type == 'docker-unsupported') {
281 dockerModuleList = ['xwiki-platform-core/xwiki-platform-menu']
282 } else {
283 dockerModuleList = dockerModules()
284 }
285 customJobProperties = getCustomJobProperties()
286 }
287
288 xwikiDockerBuild {
289 configurations = dockerConfigurationList
290 modules = dockerModuleList
291 // Make sure that we don't reset the job properties!
292 jobProperties = customJobProperties
293 }
294 }
295 {{/code}}
296 )))
297
298 Pros/Cons of this approach (which is needed because Jenkins doesn't support more than one ##Jenkinsfile##) vs using a separate job:
299 * Pros:
300 ** No need to handle branches, done automatically by the ##Jenkinsfile## (creation and deletion). Also works for PRs.
301 * Cons:
302 ** Jenkins build history gets a bit messed up, because you're reusing the same job name but running possibly different builds. For example, test failure age will get reset every time a different type of build is ran, but at least individual test history is kept.
303
304 == Clover Pipeline ==
305
306 Used to generate the global test coverage for XWiki (over all XWiki Standard repositories). See [[Test Coverage>>Community.Testing.TestCoverage.WebHome#HUsingClover2BJenkins]].
307
308 = Maintainer's guide =
309
310 == Prerequisites ==
311
312 * You need to have an account on [[maven.xwiki.org>>http://maven.xwiki.org/]] (this is the machine hosting XWiki's remote repository) and you'll need a key setup on your account there so that you can ssh to it without having to enter username or password.
313 * You need to have an administrator account on [[ci.xwiki.org>>http://ci.xwiki.org/signup]].
314
315 == Functional tests ==
316
317 === Debugging ===
318
319 **Start remote Firefox**
320
321 * For example, to start firefox on machine ##ks4##: {{code}}ssh -X <username>@<ks4 hostname>{{/code}}
322 ** For this to work you need to ensure that ##X11Forwarding## is set to ##yes## in ##/etc/ssh/sshd_config## on the remote machine:(((
323 {{code}}
324 sudo vi /etc/ssh/sshd_config
325 {{/code}}
326
327 Then reload sshd: {{code}}sudo /etc/init.d/ssh reload{{/code}}
328 )))
329
330 **Connect to the XVNC server**
331
332 * Establish a SSH tunnel between your computer and the server on port 5901 (//ssh -L 5901:localhost:5901 <[email protected] address>//)
333 * Use your favorite VNC client to connect to the XVNC server
334 ** Host : localhost
335 ** Display : 1
336 ** Password : same password as for XWiki SAS wifi access points
337 * See the functional tests live.
338
339 * Work in progress to try to connect to an agent not exposing an IP address externally (by going through maven.xwiki.org):
340 ** {{code}}ssh -L 5901:192.168.1.119:5901 -o ProxyCommand="ssh [email protected] nc -w1 %h %p" [email protected] -i ~/.ssh/id_rsa_mavenxwikiorg{{/code}}
341 ** Where ##~/.ssh/id_rsa_mavenxwikiorg## is the private key on maven.xwiki.org, used to connect to 192.168.1.119 (ip of agent 2-4 in this example)
342 ** Not working yet since I can't connect with my local vnc client.
343
344 = CI infrastructure =
345
346 == Nodes requirements ==
347
348 Here's the minimum requirements for each CI node:
349 * 2 CPUs
350 * 8GB of memory
351 * SSD hard drive with XXXGB space
352
353 == Current configurations ==
354
355 Our current CI is running with one master and 5 docker nodes split on 2 machines:
356 * a3 supports 2 nodes
357 * a4 supports 3 nodes
358
359 === a3 ===
360
361 * CPU: Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz
362 * RAM: 16G E3-1225v2
363 * Hard drive: SoftRaid 2x500 Go
364
365 === a4 ===
366
367 * CPU: Intel W3520 (8 cpus)
368 * RAM: 32GB
369 * Hard drive: SoftRaid 2x2To SATA

Get Connected