General

XWiki has a Continuous Integration setup to ensure XWiki's code is built at all times (i.e at every code check in). This also allows developers to only build the module they want and they'll have the other XWiki module dependencies downloaded from the XWiki remote repository.

We use the following tools:

We use a technique called binary dependency build which allows to check out only the module a developer wishes to work on and he'll automatically get the latest fresh binary dependencies built by our CI tool.

CI Requirements

We're currently using Jenkins as our CI tool but it's possible to change. We're listing here our requirements for a CI tool, and with some of the solutions we've evaluated or are still evaluating.

REQ1: Parallel Execution

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ1Ability to run parallel and serial builds, with different parametersacceptacceptacceptaccept

REQ2: Scheduling

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ2Ability to schedule builds (e.g. for the various docker tests) and to run the build at least once per month even if there's been no changes to the sourceerrorerroracceptaccept

REQ3: Sharing Pipeline Code

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ3Ability to share pipeline code between builds for the same repositoryy and for different repositories. And ability to pass properties to this common code.acceptacceptacceptaccept
  • Jenkins: Using a shared pipeline
  • GitLab CI: Using an include.
  • GitHub Actions: With a custom action at least (feels a bit complex).

REQ4: View Test Results

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ4Ability to view test results and failing test logs (test logs + console/xwiki logs).acceptcancelerrorhelp

REQ5: Disable Concurrent Builds

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ5(optional) Ability to aggregate commits before a build is executed, i.e. disable concurrent builds. Note that this is more a limitation of the number of agents we can have vs the duration of the full xwiki build. If we get a large number of agents, it may be a less important requirement (or not one at all).accepthelp?/)accept
  • Jenkins: Using the
  • GitLab: help
  • GitHubActions: help
  • CircleCI: In the Project CI's configuration. It's called "Auto-cancel redundant builds".

REQ6: Custom Docker Image

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ6Supports using a custom docker build image since XWiki has oneacceptacceptacceptaccept
  • Jenkins: Using the Docker Cloud plugin.
  • GitLab CI: Built in using the image YML element.
  • GitHub Actions: Buimt in using the #container YML element.

Notes:

  • That this requirement allows use to control the Java, Maven and more generally our full build environment.
  • With Jenkins and GitHub Actions the docker daemon socket is shared allowing our docker tests to work. Needs to be checked for GitLab CI.

REQ7: Parse pom.xml

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ7Ability to easily read/parse the current pom.xml to automatically find the version of Java to use and the memory parameters to set for the JVM.accepthelphelphelp
  • Jenkins: Using the readMavenPom step.

REQ8: Shell Scripts

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ8Ability to execute shell scripts before/after the build, inside our custom build image. For example to execute VNC and export the DISPLAY (for the functional tests not based on Docker)acceptacceptacceptaccept
  • Jenkins: In the Jenkinsfile/shared pipeline
  • GitLab CI: Built in, using the script step.
  • GitHub Actions: Built in, using the run step.

REQ9: Archive Artifacts

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ9Ability to archive artifacts and ability to save the failing test screenshots and videos as artifacts.acceptacceptacceptaccept
  • Jenkins: Feature of the Maven plugin + archiveArtifacts step.
  • GitLab CI: Using the artifacts step.
  • GitHub Actions: Using the upload-artifact custom action.

REQ10: Build Parameters

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ10Ability to have different build parameters (different maven parameters for example) between master and branches/PR. For example, we want to run "mvn deploy" on master and "mvn install" on branches.acceptacceptaccepthelp
  • Jenkins: Using method parameters in custom pipeline steps.
  • GitLab CI: Using variables
  • GitHub Actions: Parameters of a custom action.

REQ11: Supports GitHub

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ11Must work with sources located on GitHub.acceptacceptacceptaccept

REQ12: Build PRs

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ12Ability to run the build on PRs.acceptacceptacceptaccept

REQ13: Timeout

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ13Ability to kill builds after a given timeout.acceptaccepthelphelp
  • Jenkins: Using the timeout step.
  • GitLab CI: Using the timeout step.
  • GitHub Actions: help

REQ14: Secrets

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ14Handle secrets so that they are not in the pipeline scripts (e.g. we need to push to sonarcloud or deploy to nexus.xwiki.org or to maven central).acceptacceptaccepthelp

REQ15: Mail

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ15Ability to send mail and customize the mail template.accepthelphelp?)
  • Jenkins: Using the emailext step from the Extended Mail plugin.
  • GitLab CI: help
  • GitHub Actions: help

REQ16: Test Screenshots

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ16(Nice to have) Ability to attach failing test screenshot to failing test report pagesaccepthelphelphelp

REQ17: Handle Flickers

IdRequirementJenkinsGitLab CIGitHub ActionsCircleCI
REQ17Ability to check failing test (i.e. have an API to retrieve failing tests) so that we can compare with known flickers on our JIRA and decide to send a failing mail or not + mark the test as flickering in the CI Test Report UI.accepthelphelphelp

Jenkins Agent Image

XWiki has its own Jenkins Agent Docker image that is used by Jenkins master to spawn agents.

Jenkins Pipelines

Main Pipeline

We have developed a Jenkins Pipeline script to build XWiki projects (can be used for core and contrib projects). It has the following features:

  • Automatically use the right version of Java (and proper memory configuration)
  • Use Jenkins's Xvnc plugin to have a Display for functional tests
  • Use the "deploy" maven goal for "master" and "stable-*" branches only and "install" for the rest
  • Attach the screenshot of a failing XWiki Selenium test to the failed test's description
  • Check for false positives for known cases of failures not related to code + check for test flickers
  • Send mails for build failures
  • Supports passing custom job properties. See below for an example of how to use this to trigger nightly build of Docker-based configuration tests.

Note that if you use a "Github Organization" job type in Jenkins you'll get branch management for free, i.e. automatically build branches when new branches are added (or Pull Requests), and automatically stop building branches when they are removed from the SCM.

To use is for a project, add a Jenkinsfile with the following minimal configuration:

/*
 * See the NOTICE file distributed with this work for additional
 * information regarding copyright ownership.
 *
 * This is free software; you can redistribute it and/or modify it
 * under the terms of the GNU Lesser General Public License as
 * published by the Free Software Foundation; either version 2.1 of
 * the License, or (at your option) any later version.
 *
 * This software is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this software; if not, write to the Free
 * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
 * 02110-1301 USA, or see the FSF site: http://www.fsf.org.
 */


// It's assumed that Jenkins has been configured to implicitly load the vars/xwikiModule.groovy library which exposes
// the "xwikiModule" global function/DSL.
// Note that the version used is the one defined in Jenkins but it can be overridden as follows:
// @Library("[email protected]<branch, tag, sha1>") _
// See https://github.com/jenkinsci/workflow-cps-global-lib-plugin for details.

xwikiModule {
}

For more elaborate configuration, see:

Nightly Configuration Tests

Heres an example of how you can setup your Jenkinsfile to automatically have execution of Docker-based configuration tests every night. This example is taken from xwiki-platform's Jenkinsfile. For more details see this blog post.

  • Step 1: In the xwikiBuild step, pass the jobProperties configuration with some scheduler cron. For example:
    xwikiBuild(map.name) {
     ...
      jobProperties = getCustomJobProperties()
     ...
    }

    private def getCustomJobProperties()
    {
     // Define a scheduler job to execute the Docker-based functional tests at regular intervals. We do this since they
     // take time to execute and thus we cannot run them all the time.
     // This scheduler job will pass the "type" parameter to this Jenkinsfile when it executes, allowing us to decide if
     // we run the standard builds or the docker ones.
     // Note: it's the xwikiBuild() calls from the standard builds that will set the jobProperties and thus set up the
     // job parameter + the crons. It would be better to set the properties directly in this Jenkinsfile but we haven't
     // found a way to merge properties and calling the properties() step will override any pre-existing properties.
     return [
        parameters([string(defaultValue: 'standard', description: 'Job type', name: 'type')]),
        pipelineTriggers([
          parameterizedCron('''@midnight %type=docker-latest
    @weekly %type=docker-all
    @monthly %type=docker-unsupported'''
    ),
          cron("@monthly")
       ])
     ]
    }
  • Step 2: Based on the parametrized type value, decide to execute Docker-based tests:
    ...
    if (params.type && params.type == 'docker-latest') {
      buildDocker('docker-latest')
    }
    ...
    private void buildDocker(type)
    {
     def dockerConfigurationList
     def dockerModuleList
     def customJobProperties
     node() {
       // Checkout platform to find all docker configurations and test modules so that we can then parallelize executions
       // of configs and modules across Jenkins agents.
       checkout skipChangeLog: true, scm: scm
        dockerConfigurationList = dockerConfigurations(type)
       if (type == 'docker-unsupported') {
          dockerModuleList = ['xwiki-platform-core/xwiki-platform-menu']
       } else {
          dockerModuleList = dockerModules()
       }
        customJobProperties = getCustomJobProperties()
     }

      xwikiDockerBuild {
        configurations = dockerConfigurationList
        modules = dockerModuleList
       // Make sure that we don't reset the job properties!
       jobProperties = customJobProperties
     }
    }

Pros/Cons of this approach (which is needed because Jenkins doesn't support more than one Jenkinsfile) vs using a separate job:

  • Pros:
    • No need to handle branches, done automatically by the Jenkinsfile (creation and deletion). Also works for PRs.
  • Cons:
    • Jenkins build history gets a bit messed up, because you're reusing the same job name but running possibly different builds. For example, test failure age will get reset every time a different type of build is ran, but at least individual test history is kept.

Clover Pipeline

Used to generate the global test coverage for XWiki (over all XWiki Standard repositories). See Test Coverage.

Example usage:

import org.xwiki.jenkins.Clover
node('docker') {
   new Clover().generateGlobalCoverage([
       [baseline: "20171222-1835", fail: false],
       [baseline: "20190101-2330", fail: true]
   ])
}

Maintainer's guide

Prerequisites

  • You need to have an account on maven.xwiki.org (this is the machine hosting XWiki's remote repository) and you'll need a key setup on your account there so that you can ssh to it without having to enter username or password.
  • You need to have an administrator account on ci.xwiki.org.

Functional tests

Debugging

Start remote Firefox

  • For example, to start firefox on machine ks4: ssh -X <username>@<ks4 hostname>
    • For this to work you need to ensure that X11Forwarding is set to yes in /etc/ssh/sshd_config on the remote machine:
      sudo vi /etc/ssh/sshd_config

      Then reload sshd: sudo /etc/init.d/ssh reload

Connect to the XVNC server

  • Establish a SSH tunnel between your computer and the server on port 5901 (ssh -L 5901:localhost:5901 <[email protected] address>)
  • Use your favorite VNC client to connect to the XVNC server
    • Host : localhost
    • Display : 1
    • Password : same password as for XWiki SAS wifi access points
  • See the functional tests live.
  • Work in progress to try to connect to an agent not exposing an IP address externally (by going through maven.xwiki.org):
    • ssh -L 5901:192.168.1.119:5901 -o ProxyCommand="ssh [email protected] nc -w1 %h %p" hudsonagent@192.168.1.119 -i ~/.ssh/id_rsa_mavenxwikiorg
    • Where /.ssh/id_rsa_mavenxwikiorg is the private key on maven.xwiki.org, used to connect to 192.168.1.119 (ip of agent 2-4 in this example)
    • Not working yet since I can't connect with my local vnc client.
Tags:
Created by Jean-Vincent Drean on 2008/10/14 18:54
   

Get Connected