Wiki source code of Continuous Integration

Version 28.1 by Vincent Massol on 2019/04/02 15:46

Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 = General =
6
7 XWiki has a [[Continuous Integration>>http://www.martinfowler.com/articles/continuousIntegration.html]] setup to ensure XWiki's code is built at all times (i.e at every code check in). This also allows developers to only build the module they want and they'll have the other XWiki module dependencies downloaded from the [[XWiki remote repository>>http://nexus.xwiki.org/nexus/#view-repositories]].
8
9 We use the following tools:
10
11 * [[GitHub>>https://github.com/xwiki]] is used as our [[SCM>>http://en.wikipedia.org/wiki/Source_Code_Management]]
12 * [[Maven>>http://maven.apache.org/]] is used for [[building XWiki>>Community.Building]]
13 * [[Jenkins>>http://jenkins-ci.org/]] is used as our Continuous Integration (CI) tool (set up at [[http://ci.xwiki.org/]])
14 * [[Docker>>https://www.docker.com/docker-community]] is used to run Jenkins agents (and incidentally to execute some functional tests over various configurations)
15
16 We use a technique called [[binary dependency build>>http://web.archive.org/web/20090423073100/http://blogs.codehaus.org/people/vmassol/archives/000953_binary_dependency_builds.html]] which allows to check out only the module a developer wishes to work on and he'll automatically get the latest fresh binary dependencies built by our CI tool.
17
18 = Jenkins Agent Image =
19
20 XWiki has its own [[Jenkins Agent Docker image>>https://github.com/xwiki/xwiki-jenkins-slave]] that is used by Jenkins master to spawn agents.
21
22 = Jenkins Pipelines =
23
24 == Main Pipeline ==
25
26 We have developed a {{scm project="xwiki-jenkins-pipeline" path="vars/xwikiModule.groovy"}}Jenkins Pipeline script{{/scm}} to build XWiki projects (can be used for core and contrib projects). It has the following features:
27
28 * Automatically use the right version of Java (and proper memory configuration)
29 * Use Jenkins's Xvnc plugin to have a Display for functional tests
30 * Use the "deploy" maven goal for "master" and "stable-*" branches only and "install" for the rest
31 * [[Attach the screenshot>>http://massol.myxwiki.org/xwiki/bin/view/Blog/AttachFailingTestPipeline]] of a failing XWiki Selenium test to the failed test's description
32 * Check for false positives for known cases of failures not related to code + [[check for test flickers>>http://massol.myxwiki.org/xwiki/bin/view/Blog/FlakyTestTool]]
33 * Send mails for build failures
34
35 Note that if you use a "Github Organization" job type in Jenkins you'll get branch management for free, i.e. automatically build branches when new branches are added (or Pull Requests), and automatically stop building branches when they are removed from the SCM.
36
37 To use is for a project, add a ##Jenkinsfile## with the following minimal configuration:
38
39 {{code language="groovy"}}
40 /*
41 * See the NOTICE file distributed with this work for additional
42 * information regarding copyright ownership.
43 *
44 * This is free software; you can redistribute it and/or modify it
45 * under the terms of the GNU Lesser General Public License as
46 * published by the Free Software Foundation; either version 2.1 of
47 * the License, or (at your option) any later version.
48 *
49 * This software is distributed in the hope that it will be useful,
50 * but WITHOUT ANY WARRANTY; without even the implied warranty of
51 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
52 * Lesser General Public License for more details.
53 *
54 * You should have received a copy of the GNU Lesser General Public
55 * License along with this software; if not, write to the Free
56 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
57 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
58 */
59
60 // It's assumed that Jenkins has been configured to implicitly load the vars/xwikiModule.groovy library which exposes
61 // the "xwikiModule" global function/DSL.
62 // Note that the version used is the one defined in Jenkins but it can be overridden as follows:
63 // @Library("XWiki@<branch, tag, sha1>") _
64 // See https://github.com/jenkinsci/workflow-cps-global-lib-plugin for details.
65
66 xwikiModule {
67 }
68 {{/code}}
69
70 For more elaborate configuration, see:
71 * {{scm project="xwiki-commons" path="Jenkinsfile"}}xwiki-commons's Jenkinsfile{{/scm}}
72 * {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}}
73
74 == Clover Pipeline ==
75
76 Used to generate the global test coverage for XWiki (over all XWiki Standard repositories). See [[Test Coverage>>Community.Testing.TestCoverage.WebHome]].
77
78 Example usage:
79
80 {{code language="groovy"}}
81 import org.xwiki.jenkins.Clover
82 node('docker') {
83 new Clover().generateGlobalCoverage([
84 [baseline: "20171222-1835", fail: false],
85 [baseline: "20190101-2330", fail: true]
86 ])
87 }
88 {{/code}}
89
90 == Docker Pipeline ==
91
92 Used to run functional tests using Docker in various supported configurations. See the [[defined Jobs>>Community.Testing.DockerTesting.WebHome#HCIJobs]].
93
94 Example usage:
95
96 {{code language="groovy"}}
97 import org.xwiki.jenkins.DockerTests
98
99 def branchNames = ['master', 'stable-10.11.x']
100 def branches = [:]
101
102 branchNames.each() {
103 branches[it] = {
104 def label
105 if (it.contains('10.11.x')) {
106 label = 'docker-outside-docker'
107 } else {
108 label = 'docker'
109 }
110 node(label) {
111 new DockerTests().executeDockerSupportedTests(it)
112 }
113 }
114 }
115
116 parallel branches
117 {{/code}}
118
119 = Maintainer's guide =
120
121 == Prerequisites ==
122
123 * You need to have an account on [[maven.xwiki.org>>http://maven.xwiki.org/]] (this is the machine hosting XWiki's remote repository) and you'll need a key setup on your account there so that you can ssh to it without having to enter username or password.
124 * You need to have an administrator account on [[ci.xwiki.org>>http://ci.xwiki.org/signup]].
125
126 == Functional tests ==
127
128 === Debugging ===
129
130 **Connect to the XVNC server**
131
132 * Establish a SSH tunnel between your computer and the server on port 5901 (//ssh -L 5901:localhost:5901 <user@ip address>//)
133 * Use your favorite VNC client to connect to the XVNC server
134 ** Host : localhost
135 ** Display : 1
136 ** Password : same password as for XWiki SAS wifi access points
137 * See the functional tests live.
138
139 * Work in progress to try to connect to an agent not exposing an IP address externally (by going through maven.xwiki.org):
140 ** {{code}}ssh -L 5901:192.168.1.119:5901 -o ProxyCommand="ssh [email protected] nc -w1 %h %p" [email protected] -i ~/.ssh/id_rsa_mavenxwikiorg{{/code}}
141 ** Where ##~/.ssh/id_rsa_mavenxwikiorg## is the private key on maven.xwiki.org, used to connect to 192.168.1.119 (ip of agent 2-4 in this example)
142 ** Not working yet since I can't connect with my local vnc client.

Get Connected