Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 = General =
6
7 XWiki has a [[Continuous Integration>>http://www.martinfowler.com/articles/continuousIntegration.html]] setup to ensure XWiki's code is built at all times (i.e at every code check in). This also allows developers to only build the module they want and they'll have the other XWiki module dependencies downloaded from the [[XWiki remote repository>>http://nexus.xwiki.org/nexus/#view-repositories]].
8
9 We use the following tools:
10
11 * [[GitHub>>https://github.com/xwiki]] is used as our [[SCM>>http://en.wikipedia.org/wiki/Source_Code_Management]]
12 * [[Maven>>http://maven.apache.org/]] is used for [[building XWiki>>Community.Building]]
13 * [[Jenkins>>http://jenkins-ci.org/]] is used as our Continuous Integration (CI) tool (set up at [[http://ci.xwiki.org/]])
14 * [[Docker>>https://www.docker.com/docker-community]] is used to run Jenkins agents (and incidentally to execute some functional tests over various configurations)
15
16 We use a technique called [[binary dependency build>>http://web.archive.org/web/20090423073100/http://blogs.codehaus.org/people/vmassol/archives/000953_binary_dependency_builds.html]] which allows to check out only the module a developer wishes to work on and he'll automatically get the latest fresh binary dependencies built by our CI tool.
17
18 = Jenkins Agent Image =
19
20 XWiki has its own [[Jenkins Agent Docker image>>https://github.com/xwiki/xwiki-jenkins-slave]] that is used by Jenkins master to spawn agents.
21
22 = Jenkins Pipelines =
23
24 == Main Pipeline ==
25
26 We have developed a {{scm project="xwiki-jenkins-pipeline" path="vars/xwikiModule.groovy"}}Jenkins Pipeline script{{/scm}} to build XWiki projects (can be used for core and contrib projects). It has the following features:
27
28 * Automatically use the right version of Java (and proper memory configuration)
29 * Use Jenkins's Xvnc plugin to have a Display for functional tests
30 * Use the "deploy" maven goal for "master" and "stable-*" branches only and "install" for the rest
31 * [[Attach the screenshot>>http://massol.myxwiki.org/xwiki/bin/view/Blog/AttachFailingTestPipeline]] of a failing XWiki Selenium test to the failed test's description
32 * Check for false positives for known cases of failures not related to code + [[check for test flickers>>http://massol.myxwiki.org/xwiki/bin/view/Blog/FlakyTestTool]]
33 * Send mails for build failures
34 * Supports passing custom job properties. See below for an example of how to use this to trigger nightly build of Docker-based configuration tests.
35
36 Note that if you use a "Github Organization" job type in Jenkins you'll get branch management for free, i.e. automatically build branches when new branches are added (or Pull Requests), and automatically stop building branches when they are removed from the SCM.
37
38 To use is for a project, add a ##Jenkinsfile## with the following minimal configuration:
39
40 {{code language="groovy"}}
41 /*
42 * See the NOTICE file distributed with this work for additional
43 * information regarding copyright ownership.
44 *
45 * This is free software; you can redistribute it and/or modify it
46 * under the terms of the GNU Lesser General Public License as
47 * published by the Free Software Foundation; either version 2.1 of
48 * the License, or (at your option) any later version.
49 *
50 * This software is distributed in the hope that it will be useful,
51 * but WITHOUT ANY WARRANTY; without even the implied warranty of
52 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
53 * Lesser General Public License for more details.
54 *
55 * You should have received a copy of the GNU Lesser General Public
56 * License along with this software; if not, write to the Free
57 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
58 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
59 */
60
61 // It's assumed that Jenkins has been configured to implicitly load the vars/xwikiModule.groovy library which exposes
62 // the "xwikiModule" global function/DSL.
63 // Note that the version used is the one defined in Jenkins but it can be overridden as follows:
64 // @Library("[email protected]<branch, tag, sha1>") _
65 // See https://github.com/jenkinsci/workflow-cps-global-lib-plugin for details.
66
67 xwikiModule {
68 }
69 {{/code}}
70
71 For more elaborate configuration, see:
72 * {{scm project="xwiki-commons" path="Jenkinsfile"}}xwiki-commons's Jenkinsfile{{/scm}}
73 * {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}}
74
75 === Nightly Configuration Tests ===
76
77 Heres an example of how you can setup your ##Jenkinsfile## to automatically have execution of Docker-based configuration tests every night. This example is taken from {{scm project="xwiki-platform" path="Jenkinsfile"}}xwiki-platform's Jenkinsfile{{/scm}}. For more details see [[this blog post>>http://massol.myxwiki.org/xwiki/bin/view/Blog/ScheduledJenkinsfile]].
78
79 * Step 1: In the ##xwikiBuild## step, pass the ##jobProperties## configuration with some scheduler cron. For example:(((
80 {{code language="groovy"}}
81 xwikiBuild(map.name) {
82 ...
83 jobProperties = getCustomJobProperties()
84 ...
85 }
86
87 private def getCustomJobProperties()
88 {
89 // Define a scheduler job to execute the Docker-based functional tests at regular intervals. We do this since they
90 // take time to execute and thus we cannot run them all the time.
91 // This scheduler job will pass the "type" parameter to this Jenkinsfile when it executes, allowing us to decide if
92 // we run the standard builds or the docker ones.
93 // Note: it's the xwikiBuild() calls from the standard builds that will set the jobProperties and thus set up the
94 // job parameter + the crons. It would be better to set the properties directly in this Jenkinsfile but we haven't
95 // found a way to merge properties and calling the properties() step will override any pre-existing properties.
96 return [
97 parameters([string(defaultValue: 'standard', description: 'Job type', name: 'type')]),
98 pipelineTriggers([
99 parameterizedCron('''@midnight %type=docker-latest
100 @weekly %type=docker-all
101 @monthly %type=docker-unsupported'''),
102 cron("@monthly")
103 ])
104 ]
105 }
106 {{/code}}
107 )))
108 * Step 2: Based on the parametrized ##type## value, decide to execute Docker-based tests:(((
109 {{code language="groovy"}}
110 ...
111 if (params.type && params.type == 'docker-latest') {
112 buildDocker('docker-latest')
113 }
114 ...
115 private void buildDocker(type)
116 {
117 def dockerConfigurationList
118 def dockerModuleList
119 def customJobProperties
120 node() {
121 // Checkout platform to find all docker configurations and test modules so that we can then parallelize executions
122 // of configs and modules across Jenkins agents.
123 checkout skipChangeLog: true, scm: scm
124 dockerConfigurationList = dockerConfigurations(type)
125 if (type == 'docker-unsupported') {
126 dockerModuleList = ['xwiki-platform-core/xwiki-platform-menu']
127 } else {
128 dockerModuleList = dockerModules()
129 }
130 customJobProperties = getCustomJobProperties()
131 }
132
133 xwikiDockerBuild {
134 configurations = dockerConfigurationList
135 modules = dockerModuleList
136 // Make sure that we don't reset the job properties!
137 jobProperties = customJobProperties
138 }
139 }
140 {{/code}}
141 )))
142
143 Pros/Cons of this approach (which is needed because Jenkins doesn't support more than one ##Jenkinsfile##) vs using a separate job:
144 * Pros:
145 ** No need to handle branches, done automatically by the ##Jenkinsfile## (creation and deletion). Also works for PRs.
146 * Cons:
147 ** Jenkins build history gets a bit messed up, because you're reusing the same job name but running possibly different builds. For example, test failure age will get reset every time a different type of build is ran, but at least individual test history is kept.
148
149 == Clover Pipeline ==
150
151 Used to generate the global test coverage for XWiki (over all XWiki Standard repositories). See [[Test Coverage>>Community.Testing.TestCoverage.WebHome]].
152
153 Example usage:
154
155 {{code language="groovy"}}
156 import org.xwiki.jenkins.Clover
157 node('docker') {
158 new Clover().generateGlobalCoverage([
159 [baseline: "20171222-1835", fail: false],
160 [baseline: "20190101-2330", fail: true]
161 ])
162 }
163 {{/code}}
164
165 = Maintainer's guide =
166
167 == Prerequisites ==
168
169 * You need to have an account on [[maven.xwiki.org>>http://maven.xwiki.org/]] (this is the machine hosting XWiki's remote repository) and you'll need a key setup on your account there so that you can ssh to it without having to enter username or password.
170 * You need to have an administrator account on [[ci.xwiki.org>>http://ci.xwiki.org/signup]].
171
172 == Functional tests ==
173
174 === Debugging ===
175
176 **Start remote Firefox**
177
178 * For example, to start firefox on machine ##ks4##: {{code}}ssh -X <username>@<ks4 hostname>{{/code}}
179 ** For this to work you need to ensure that ##X11Forwarding## is set to ##yes## in ##/etc/ssh/sshd_config## on the remote machine:(((
180 {{code}}
181 sudo vi /etc/ssh/sshd_config
182 {{/code}}
183
184 Then reload sshd: {{code}}sudo /etc/init.d/ssh reload{{/code}}
185 )))
186
187 **Connect to the XVNC server**
188
189 * Establish a SSH tunnel between your computer and the server on port 5901 (//ssh -L 5901:localhost:5901 <[email protected] address>//)
190 * Use your favorite VNC client to connect to the XVNC server
191 ** Host : localhost
192 ** Display : 1
193 ** Password : same password as for XWiki SAS wifi access points
194 * See the functional tests live.
195
196 * Work in progress to try to connect to an agent not exposing an IP address externally (by going through maven.xwiki.org):
197 ** {{code}}ssh -L 5901:192.168.1.119:5901 -o ProxyCommand="ssh [email protected] nc -w1 %h %p" [email protected] -i ~/.ssh/id_rsa_mavenxwikiorg{{/code}}
198 ** Where ##~/.ssh/id_rsa_mavenxwikiorg## is the private key on maven.xwiki.org, used to connect to 192.168.1.119 (ip of agent 2-4 in this example)
199 ** Not working yet since I can't connect with my local vnc client.

Get Connected