Wiki source code of Continuous Integration
Last modified by Vincent Massol on 2023/04/25 14:02
Hide last authors
author | version | line-number | content |
---|---|---|---|
13.1 | 1 | {{box cssClass="floatinginfobox" title="**Contents**"}} | |
2 | {{toc/}} | ||
3 | {{/box}} | ||
12.1 | 4 | ||
13.1 | 5 | = General = | |
3.2 | 6 | ||
23.2 | 7 | XWiki has a [[Continuous Integration>>http://www.martinfowler.com/articles/continuousIntegration.html]] setup to ensure XWiki's code is built at all times (i.e at every code check in). This also allows developers to only build the module they want and they'll have the other XWiki module dependencies downloaded from the [[XWiki remote repository>>http://nexus.xwiki.org/nexus/#view-repositories]]. | |
1.1 | 8 | ||
3.1 | 9 | We use the following tools: | |
10 | |||
12.1 | 11 | * [[GitHub>>https://github.com/xwiki]] is used as our [[SCM>>http://en.wikipedia.org/wiki/Source_Code_Management]] | |
12 | * [[Maven>>http://maven.apache.org/]] is used for [[building XWiki>>Community.Building]] | ||
13 | * [[Jenkins>>http://jenkins-ci.org/]] is used as our Continuous Integration (CI) tool (set up at [[http://ci.xwiki.org/]]) | ||
21.1 | 14 | * [[Docker>>https://www.docker.com/docker-community]] is used to run Jenkins agents (and incidentally to execute some functional tests over various configurations) | |
6.1 | 15 | ||
14.2 | 16 | We use a technique called [[binary dependency build>>http://web.archive.org/web/20090423073100/http://blogs.codehaus.org/people/vmassol/archives/000953_binary_dependency_builds.html]] which allows to check out only the module a developer wishes to work on and he'll automatically get the latest fresh binary dependencies built by our CI tool. | |
1.1 | 17 | ||
32.1 | 18 | = CI Requirements = | |
19 | |||
33.1 | 20 | We're currently using Jenkins as our CI tool but it's possible to change. We're listing here our requirements for a CI tool, and with some of the solutions we've evaluated or are still evaluating. | |
32.1 | 21 | ||
33.1 | 22 | == REQ1: Parallel Execution == | |
32.1 | 23 | ||
34.1 | 24 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
25 | |REQ1|Ability to run parallel and serial builds, with different parameters|(/)|(/)|(/)|(/) | ||
33.1 | 26 | ||
27 | * Jenkins: Using the ##parallel()## step | ||
28 | * GitLab CI: Using [[##trigger##, ##rules## and ##needs##>>https://docs.gitlab.com/ee/ci/pipelines/pipeline_architectures.html]]. See also [[multi-project triggering>>https://docs.gitlab.com/ee/ci/multi_project_pipelines.html]]. See also [[Child pipelines>>https://docs.gitlab.com/ee/ci/parent_child_pipelines.html]]. General doc for [[pipeline types>>https://docs.gitlab.com/ee/ci/pipelines/#types-of-pipelines]]. | ||
29 | * GitHub Actions: Using different YML files in ##.github/workflows## (jobs run in parallel by default) and using [[##job.needs##>>https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobs]]. | ||
34.1 | 30 | * CircleCI: See [[https://circleci.com/blog/decrease-your-build-times-by-running-jobs-in-parallel-with-workflows/]] | |
33.1 | 31 | ||
32 | == REQ2: Scheduling == | ||
33 | |||
34.1 | 34 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
35 | |REQ2|Ability to schedule builds (e.g. for the various docker tests) and to run the build at least once per month even if there's been no changes to the source|(!)|(!)|(/)|(/) | ||
33.1 | 36 | ||
37 | * Jenkins: Scheduling is supported but it's [[very complex to make it work automatically with a pipeline>>https://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]] since a repo can have only a single pipeline (i.e. a single ##Jenkinsfile##). Other solutions include using a Git submodule or another GitHub Organization Job. | ||
38 | * GitLab CI: It's possible to have [[schedules>>https://docs.gitlab.com/ee/ci/pipelines/schedules.html]] but note that the schedule is not part of the YML file but something you define in the GitLab CI configuration UI. However, it's also possible to [[use the API>>https://docs.gitlab.com/ee/api/pipeline_schedules.html#create-a-new-pipeline-schedule]]. | ||
39 | * Github Actions: Using the [[##on## element with a ##schedule##>>https://help.github.com/en/actions/reference/events-that-trigger-workflows#scheduled-events-schedule]]. | ||
35.1 | 40 | * CircleCI: See https://circleci.com/docs/2.0/configuration-reference/#schedule | |
33.1 | 41 | ||
42 | == REQ3: Sharing Pipeline Code == | ||
43 | |||
34.1 | 44 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
45 | |REQ3|Ability to share pipeline code between builds for the same repositoryy and for different repositories. And ability to pass properties to this common code.|(/)|(/)|(/)|(/) | ||
33.1 | 46 | ||
47 | * Jenkins: Using a shared pipeline | ||
48 | * GitLab CI: Using an [[##include##>>https://docs.gitlab.com/ee/ci/yaml/README.html#include]]. | ||
49 | * GitHub Actions: With a [[custom action>>https://help.github.com/en/actions/building-actions]] at least (feels a bit complex). | ||
50 | |||
51 | == REQ4: View Test Results == | ||
52 | |||
34.1 | 53 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
54 | |REQ4|Ability to view test results and failing test logs (test logs + console/xwiki logs).|(/)|(x)|(!)|(?) | ||
33.1 | 55 | ||
56 | * Jenkins: Built in. | ||
57 | * GitLab CI: Built in but [[not working ATM because of performance issues>>https://gitlab.com/groups/gitlab-org/-/epics/2854]]. | ||
35.2 | 58 | * GitHub Actions: Needs a custom action ATM. Found 2 but didn't work when I tested them: [[check run reporter>>https://www.check-run-reporter.com/]] ([[problem reported>>https://github.com/check-run-reporter/feedback/issues/3]]) and [[Scope>>https://scope.dev/]] (2 problems: unknown cost + got stuck when I tested it on commons and I had to kill the workflow after 1 hour). | |
36.1 | 59 | ** Update: [[new ScaCap surefire report action>>https://github.com/ScaCap/action-surefire-report]] to test. | |
33.1 | 60 | ||
61 | == REQ5: Disable Concurrent Builds == | ||
62 | |||
34.1 | 63 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
64 | |REQ5|(optional) Ability to aggregate commits before a build is executed, i.e. disable concurrent builds. Note that this is more a limitation of the number of agents we can have vs the duration of the full xwiki build. If we get a large number of agents, it may be a less important requirement (or not one at all).|(/)|(?)|?/)|(/) | ||
33.1 | 65 | ||
34.1 | 66 | * Jenkins: Using the | |
67 | * GitLab: (?) | ||
68 | * GitHubActions: (?) | ||
69 | * CircleCI: In the Project CI's configuration. It's called "Auto-cancel redundant builds". | ||
70 | |||
33.1 | 71 | == REQ6: Custom Docker Image == | |
72 | |||
34.1 | 73 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
74 | |REQ6|Supports using a custom docker build image since [[XWiki has one>>https://github.com/xwiki/xwiki-docker-build/tree/master/build]]|(/)|(/)|(/)|(/) | ||
33.1 | 75 | ||
76 | * Jenkins: Using the Docker Cloud plugin. | ||
77 | * GitLab CI: Built in using the ##image## YML element. | ||
78 | * GitHub Actions: Buimt in using the [[#container##>>https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idcontainer]] YML element. | ||
79 | |||
80 | Notes: | ||
81 | * That this requirement allows use to control the Java, Maven and more generally our full build environment. | ||
82 | * With Jenkins and GitHub Actions the docker daemon socket is shared allowing our docker tests to work. Needs to be checked for GitLab CI. | ||
83 | |||
84 | == REQ7: Parse pom.xml == | ||
85 | |||
34.1 | 86 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
87 | |REQ7|Ability to easily read/parse the current pom.xml to automatically find the version of Java to use and the memory parameters to set for the JVM.|(/)|(?)|(?)|(?) | ||
33.1 | 88 | ||
89 | * Jenkins: Using the ##readMavenPom## step. | ||
90 | |||
91 | == REQ8: Shell Scripts == | ||
92 | |||
34.1 | 93 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
94 | |REQ8|Ability to execute shell scripts before/after the build, inside our custom build image. For example to execute VNC and export the DISPLAY (for the functional tests not based on Docker)|(/)|(/)|(/)|(/) | ||
33.1 | 95 | ||
96 | * Jenkins: In the Jenkinsfile/shared pipeline | ||
97 | * GitLab CI: Built in, using the ##script## step. | ||
98 | * GitHub Actions: Built in, using the ##run## step. | ||
99 | |||
100 | == REQ9: Archive Artifacts == | ||
101 | |||
34.1 | 102 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
103 | |REQ9|Ability to archive artifacts and ability to save the failing test screenshots and videos as artifacts.|(/)|(/)|(/)|(/) | ||
33.1 | 104 | ||
105 | * Jenkins: Feature of the Maven plugin + ##archiveArtifacts## step. | ||
106 | * GitLab CI: Using the ##artifacts## step. | ||
107 | * GitHub Actions: Using the [[##upload-artifact## custom action>>https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts]]. | ||
108 | |||
109 | == REQ10: Build Parameters == | ||
110 | |||
34.1 | 111 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
112 | |REQ10|Ability to have different build parameters (different maven parameters for example) between master and branches/PR. For example, we want to run "mvn deploy" on master and "mvn install" on branches.|(/)|(/)|(/)|(?) | ||
33.1 | 113 | ||
114 | * Jenkins: Using method parameters in custom pipeline steps. | ||
115 | * GitLab CI: Using [[##variables##>>https://docs.gitlab.com/ee/ci/variables/]] | ||
116 | * GitHub Actions: Parameters of a custom action. | ||
117 | |||
118 | == REQ11: Supports GitHub == | ||
119 | |||
34.1 | 120 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
121 | |REQ11|Must work with sources located on GitHub.|(/)|(/)|(/)|(/) | ||
33.1 | 122 | ||
123 | * Jenkins: GitHub webhook. | ||
124 | * GitLab CI: Automatically [[creates a mirrored GitHub repo>>https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html]]. | ||
125 | * GitHub Actions: By design ;) | ||
126 | |||
127 | == REQ12: Build PRs == | ||
128 | |||
34.1 | 129 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
130 | |REQ12|Ability to run the build on PRs.|(/)|(/)|(/)|(/) | ||
33.1 | 131 | ||
132 | == REQ13: Timeout == | ||
133 | |||
34.1 | 134 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
135 | |REQ13|Ability to kill builds after a given timeout.|(/)|(/)|(?)|(?) | ||
33.1 | 136 | ||
137 | * Jenkins: Using the ##timeout## step. | ||
138 | * GitLab CI: Using the [[##timeout## step>>https://docs.gitlab.com/ee/ci/yaml/#timeout]]. | ||
139 | * GitHub Actions: (?) | ||
140 | |||
141 | == REQ14: Secrets == | ||
142 | |||
34.1 | 143 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
144 | |REQ14|Handle secrets so that they are not in the pipeline scripts (e.g. we need to push to sonarcloud or deploy to nexus.xwiki.org or to maven central).|(/)|(/)|(/)|(?) | ||
33.1 | 145 | ||
146 | == REQ15: Mail == | ||
147 | |||
34.1 | 148 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
149 | |REQ15|Ability to send mail and customize the mail template.|(/)|(?)|(?)|?) | ||
33.1 | 150 | ||
151 | * Jenkins: Using the ##emailext## step from the Extended Mail plugin. | ||
152 | * GitLab CI: (?) | ||
153 | * GitHub Actions: (?) | ||
154 | |||
155 | == REQ16: Test Screenshots == | ||
156 | |||
34.1 | 157 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
158 | |REQ16|(Nice to have) Ability to attach failing test screenshot to failing test report pages|(/)|(?)|(?)|(?) | ||
33.1 | 159 | ||
160 | == REQ17: Handle Flickers == | ||
161 | |||
34.1 | 162 | |=(% width=100px; %)Id|=Requirement|=(% width=100px; %)Jenkins|=(% width=100px; %)GitLab CI|=(% width=100px; %)GitHub Actions|=(% width=100px; %)CircleCI | |
163 | |REQ17|Ability to check failing test (i.e. have an API to retrieve failing tests) so that we can compare with known flickers on our JIRA and decide to send a failing mail or not + mark the test as flickering in the CI Test Report UI.|(/)|(?)|(?)|(?) | ||
33.1 | 164 | ||
23.1 | 165 | = Jenkins Agent Image = | |
166 | |||
26.4 | 167 | XWiki has its own [[Jenkins Agent Docker image>>https://github.com/xwiki/xwiki-jenkins-slave]] that is used by Jenkins master to spawn agents. | |
23.1 | 168 | ||
38.1 | 169 | = Maven settings | |
170 | |||
171 | The Maven settings used during Jenkins builds are located in each agent and need to be kept in sync. | ||
172 | |||
38.2 | 173 | Among other things the ##settings.xml## expose two "configurable" profiles: | |
174 | * ##repository-all##: the default profile which enable all Maven repositories used by Maven. It can be disabled with ##-P-repository-all## | ||
38.1 | 175 | * ##repository-snapshots##: not enabled by default. It's generally used when you disabled ##repository-all## but still want to use XWiki own snapshots as dependencies | |
176 | |||
18.1 | 177 | = Jenkins Pipelines = | |
16.2 | 178 | ||
18.1 | 179 | == Main Pipeline == | |
16.2 | 180 | ||
18.1 | 181 | We have developed a {{scm project="xwiki-jenkins-pipeline" path="vars/xwikiModule.groovy"}}Jenkins Pipeline script{{/scm}} to build XWiki projects (can be used for core and contrib projects). It has the following features: | |
182 | |||
16.2 | 183 | * Automatically use the right version of Java (and proper memory configuration) | |
184 | * Use Jenkins's Xvnc plugin to have a Display for functional tests | ||
185 | * Use the "deploy" maven goal for "master" and "stable-*" branches only and "install" for the rest | ||
186 | * [[Attach the screenshot>>http://massol.myxwiki.org/xwiki/bin/view/Blog/AttachFailingTestPipeline]] of a failing XWiki Selenium test to the failed test's description | ||
187 | * Check for false positives for known cases of failures not related to code + [[check for test flickers>>http://massol.myxwiki.org/xwiki/bin/view/Blog/FlakyTestTool]] | ||
188 | * Send mails for build failures | ||
29.1 | 189 | * Supports passing custom job properties. See below for an example of how to use this to trigger nightly build of Docker-based configuration tests. | |
16.2 | 190 | ||
28.1 | 191 | Note that if you use a "Github Organization" job type in Jenkins you'll get branch management for free, i.e. automatically build branches when new branches are added (or Pull Requests), and automatically stop building branches when they are removed from the SCM. | |
192 | |||
18.1 | 193 | To use is for a project, add a ##Jenkinsfile## with the following minimal configuration: | |
16.2 | 194 | ||
18.1 | 195 | {{code language="groovy"}} | |
196 | /* | ||
197 | * See the NOTICE file distributed with this work for additional | ||
198 | * information regarding copyright ownership. | ||
199 | * | ||
200 | * This is free software; you can redistribute it and/or modify it | ||
201 | * under the terms of the GNU Lesser General Public License as | ||
202 | * published by the Free Software Foundation; either version 2.1 of | ||
203 | * the License, or (at your option) any later version. | ||
204 | * | ||
205 | * This software is distributed in the hope that it will be useful, | ||
206 | * but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
207 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU | ||
208 | * Lesser General Public License for more details. | ||
209 | * | ||
210 | * You should have received a copy of the GNU Lesser General Public | ||
211 | * License along with this software; if not, write to the Free | ||
212 | * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA | ||
213 | * 02110-1301 USA, or see the FSF site: http://www.fsf.org. | ||
214 | */ | ||
16.2 | 215 | ||
18.1 | 216 | // It's assumed that Jenkins has been configured to implicitly load the vars/xwikiModule.groovy library which exposes | |
217 | // the "xwikiModule" global function/DSL. | ||
218 | // Note that the version used is the one defined in Jenkins but it can be overridden as follows: | ||
219 | // @Library("XWiki@<branch, tag, sha1>") _ | ||
220 | // See https://github.com/jenkinsci/workflow-cps-global-lib-plugin for details. | ||
221 | |||
222 | xwikiModule { | ||
223 | } | ||
18.2 | 224 | {{/code}} | |
18.1 | 225 | ||
43.1 | 226 | Example with SonarQube execution (XWiki < 14.0): | |
227 | |||
228 | {{code language="groovy"}} | ||
229 | xwikiModule { | ||
230 | goals = 'clean deploy jacoco:report sonar:sonar' | ||
231 | sonar = true | ||
232 | // Force Java 11 even though we depend on XS requiring only Java 8 because we run SonarQube and this requires | ||
233 | // Java 11. | ||
234 | javaTool = 'java11' | ||
235 | } | ||
236 | {{/code}} | ||
237 | |||
238 | Example with SonarQube execution (XWiki >= 14.0): | ||
239 | |||
240 | {{code language="groovy"}} | ||
241 | xwikiModule { | ||
242 | goals = 'clean deploy jacoco:report sonar:sonar' | ||
243 | sonar = true | ||
244 | } | ||
245 | {{/code}} | ||
246 | |||
247 | Example with Docker tests: | ||
248 | |||
249 | {{code language="groovy"}} | ||
250 | node('docker') { | ||
251 | xwikiBuild { | ||
252 | xvnc = false | ||
253 | profiles = 'integration-tests,docker' | ||
254 | } | ||
255 | } | ||
256 | {{/code}} | ||
257 | |||
18.1 | 258 | For more elaborate configuration, see: | |
259 | * {{scm project="xwiki-commons" path="Jenkinsfile"}}xwiki-commons's Jenkinsfile{{/scm}} | ||
260 | * {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}} | ||
261 | |||
29.1 | 262 | === Nightly Configuration Tests === | |
263 | |||
41.1 | 264 | Here's what we do in XWiki Platform to not only have a pipeline triggered whenever there's a commit but also to separate environment tests so that they are executed only during the night: | |
265 | * Have 2 files, each containing the pipeline to execute, and define 2 "Organization Folder" jobs in Jenkins (each configured to execute a given Jenkins file): | ||
266 | ** {{scm project="xwiki-platform" path="Jenkinsfile"}}Jenkinsfile{{/scm}}: the main pipeline that we use to run the build whenever there's a commit. | ||
267 | ** {{scm project="xwiki-platform" path="JenkinsfileEnvironmentTests"}}JenkinsfileEnvironmentTests{{/scm}}: the pipeline that executes the environment tests using a crontab | ||
268 | * The strategy for the ##JenkinsfileEnvironmentTests## file is explained in details in [[this blog post>>http://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]]. In short: | ||
269 | ** Step 1: In the ##xwikiBuild## step, pass the ##jobProperties## configuration with some scheduler cron. For example:((( | ||
29.1 | 270 | {{code language="groovy"}} | |
271 | xwikiBuild(map.name) { | ||
272 | ... | ||
273 | jobProperties = getCustomJobProperties() | ||
274 | ... | ||
275 | } | ||
276 | |||
277 | private def getCustomJobProperties() | ||
278 | { | ||
279 | // Define a scheduler job to execute the Docker-based functional tests at regular intervals. We do this since they | ||
280 | // take time to execute and thus we cannot run them all the time. | ||
281 | // This scheduler job will pass the "type" parameter to this Jenkinsfile when it executes, allowing us to decide if | ||
282 | // we run the standard builds or the docker ones. | ||
283 | // Note: it's the xwikiBuild() calls from the standard builds that will set the jobProperties and thus set up the | ||
284 | // job parameter + the crons. It would be better to set the properties directly in this Jenkinsfile but we haven't | ||
285 | // found a way to merge properties and calling the properties() step will override any pre-existing properties. | ||
286 | return [ | ||
287 | parameters([string(defaultValue: 'standard', description: 'Job type', name: 'type')]), | ||
288 | pipelineTriggers([ | ||
289 | parameterizedCron('''@midnight %type=docker-latest | ||
290 | @weekly %type=docker-all | ||
291 | @monthly %type=docker-unsupported'''), | ||
292 | cron("@monthly") | ||
293 | ]) | ||
294 | ] | ||
295 | } | ||
296 | {{/code}} | ||
297 | ))) | ||
41.1 | 298 | ** Step 2: Based on the parametrized ##type## value, decide to execute Docker-based tests:((( | |
29.1 | 299 | {{code language="groovy"}} | |
300 | ... | ||
301 | if (params.type && params.type == 'docker-latest') { | ||
302 | buildDocker('docker-latest') | ||
303 | } | ||
304 | ... | ||
305 | private void buildDocker(type) | ||
306 | { | ||
307 | def dockerConfigurationList | ||
308 | def dockerModuleList | ||
309 | def customJobProperties | ||
310 | node() { | ||
311 | // Checkout platform to find all docker configurations and test modules so that we can then parallelize executions | ||
312 | // of configs and modules across Jenkins agents. | ||
313 | checkout skipChangeLog: true, scm: scm | ||
314 | dockerConfigurationList = dockerConfigurations(type) | ||
315 | if (type == 'docker-unsupported') { | ||
316 | dockerModuleList = ['xwiki-platform-core/xwiki-platform-menu'] | ||
317 | } else { | ||
318 | dockerModuleList = dockerModules() | ||
319 | } | ||
320 | customJobProperties = getCustomJobProperties() | ||
321 | } | ||
322 | |||
323 | xwikiDockerBuild { | ||
324 | configurations = dockerConfigurationList | ||
325 | modules = dockerModuleList | ||
326 | // Make sure that we don't reset the job properties! | ||
327 | jobProperties = customJobProperties | ||
328 | } | ||
329 | } | ||
330 | {{/code}} | ||
331 | ))) | ||
41.1 | 332 | ** Step 3: Configure the job to suppress being triggered when there's a commit (so that it's only triggered based on its crontab):((( | |
333 | {{image reference="scm-trigger-suppress.png"/}} | ||
334 | ))) | ||
29.1 | 335 | ||
41.1 | 336 | Pros of this approach: | |
337 | * No need to handle branches, done automatically by the Jenkins files (creation and deletion). Also works for PRs. | ||
338 | * No messy history that we would have if we were using only a single Jenkins file. | ||
29.1 | 339 | ||
19.1 | 340 | == Clover Pipeline == | |
341 | |||
36.3 | 342 | Used to generate the global test coverage for XWiki (over all XWiki Standard repositories). See [[Test Coverage>>Community.Testing.TestCoverage.WebHome#HUsingClover2BJenkins]]. | |
19.1 | 343 | ||
13.1 | 344 | = Maintainer's guide = | |
1.1 | 345 | ||
13.1 | 346 | == Prerequisites == | |
1.1 | 347 | ||
14.3 | 348 | * You need to have an account on [[maven.xwiki.org>>http://maven.xwiki.org/]] (this is the machine hosting XWiki's remote repository) and you'll need a key setup on your account there so that you can ssh to it without having to enter username or password. | |
12.1 | 349 | * You need to have an administrator account on [[ci.xwiki.org>>http://ci.xwiki.org/signup]]. | |
1.1 | 350 | ||
13.1 | 351 | == Functional tests == | |
1.1 | 352 | ||
13.1 | 353 | === Debugging === | |
2.1 | 354 | ||
30.1 | 355 | **Start remote Firefox** | |
356 | |||
357 | * For example, to start firefox on machine ##ks4##: {{code}}ssh -X <username>@<ks4 hostname>{{/code}} | ||
358 | ** For this to work you need to ensure that ##X11Forwarding## is set to ##yes## in ##/etc/ssh/sshd_config## on the remote machine:((( | ||
359 | {{code}} | ||
360 | sudo vi /etc/ssh/sshd_config | ||
361 | {{/code}} | ||
31.1 | 362 | ||
31.2 | 363 | Then reload sshd: {{code}}sudo /etc/init.d/ssh reload{{/code}} | |
30.1 | 364 | ))) | |
365 | |||
12.1 | 366 | **Connect to the XVNC server** | |
367 | |||
27.1 | 368 | * Establish a SSH tunnel between your computer and the server on port 5901 (//ssh -L 5901:localhost:5901 <user@ip address>//) | |
2.1 | 369 | * Use your favorite VNC client to connect to the XVNC server | |
370 | ** Host : localhost | ||
371 | ** Display : 1 | ||
372 | ** Password : same password as for XWiki SAS wifi access points | ||
373 | * See the functional tests live. | ||
374 | |||
27.1 | 375 | * Work in progress to try to connect to an agent not exposing an IP address externally (by going through maven.xwiki.org): | |
376 | ** {{code}}ssh -L 5901:192.168.1.119:5901 -o ProxyCommand="ssh [email protected] nc -w1 %h %p" [email protected] -i ~/.ssh/id_rsa_mavenxwikiorg{{/code}} | ||
377 | ** Where ##~/.ssh/id_rsa_mavenxwikiorg## is the private key on maven.xwiki.org, used to connect to 192.168.1.119 (ip of agent 2-4 in this example) | ||
378 | ** Not working yet since I can't connect with my local vnc client. | ||
37.1 | 379 | ||
380 | = CI infrastructure = | ||
381 | |||
382 | == Nodes requirements == | ||
383 | |||
384 | Here's the minimum requirements for each CI node: | ||
37.3 | 385 | * 2 CPUs | |
386 | * 8GB of memory | ||
387 | * SSD hard drive with XXXGB space | ||
37.1 | 388 | ||
389 | == Current configurations == | ||
390 | |||
40.1 | 391 | Our [[current CI>>https://ci.xwiki.org]] is running with 7 docker nodes running each one docker build CI image (##a1## to ##a7##). Each agent has the following configuration: | |
392 | * CPU: ((({{code language="none"}} | ||
393 | $ lscpu | ||
394 | Architecture: x86_64 | ||
395 | CPU op-mode(s): 32-bit, 64-bit | ||
396 | Byte Order: Little Endian | ||
397 | Address sizes: 40 bits physical, 48 bits virtual | ||
398 | CPU(s): 4 | ||
399 | On-line CPU(s) list: 0-3 | ||
400 | Thread(s) per core: 1 | ||
401 | Core(s) per socket: 1 | ||
402 | Socket(s): 4 | ||
403 | NUMA node(s): 1 | ||
404 | Vendor ID: GenuineIntel | ||
405 | CPU family: 6 | ||
406 | Model: 60 | ||
407 | Model name: Intel Core Processor (Haswell, no TSX) | ||
408 | Stepping: 1 | ||
409 | CPU MHz: 2599.990 | ||
410 | BogoMIPS: 5199.98 | ||
411 | Virtualization: VT-x | ||
412 | Hypervisor vendor: KVM | ||
413 | Virtualization type: full | ||
414 | L1d cache: 32K | ||
415 | L1i cache: 32K | ||
416 | L2 cache: 4096K | ||
417 | L3 cache: 16384K | ||
418 | NUMA node0 CPU(s): 0-3 | ||
419 | Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear | ||
420 | {{/code}}))) | ||
421 | * Disk: ((({{code}} | ||
422 | $ lsblk | ||
423 | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | ||
424 | sda 8:0 0 160G 0 disk | ||
425 | └─sda1 8:1 0 160G 0 part / | ||
426 | {{/code}}))) | ||
37.2 | 427 | ||
39.1 | 428 | Each agent needs to be [[properly configured>>https://github.com/xwiki/xwiki-docker-build/tree/master/build#setup-for-cixwikiorg]]. | |
429 | |||
37.1 | 430 |