Code Coverage while running Maven tests

We support both Jacoco and Clover to generate test coverage reports.

To generate test coverage reports make sure you can build your module and then pick one of the following strategies below depending on what you wish to generate.

Single Maven Reactor

Using Jacoco

• Go in the first top level module (e.g. xwiki-commons, also works for extensions) and run: mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=false -Plegacy,integration-tests -Dxwiki.revapi.skip=true -Dmaven.test.failure.ignore=true
• Go in all the other top level modules you wish to add and run: mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=true -Plegacy,integration-tests  -Dxwiki.revapi.skip=true -Dmaven.test.failure.ignore=true
• Then whenever you wish to generate the full test report, run: mvn jacoco:report -Djacoco.dataFile=/tmp/jacoco.exec

  TODO

  Jacoco supports generating report from a single module, it even supports generating aggregated report from several modules using the report-aggregate mojo introduced in 0.7.7 (but note that I don't know if that includes coverage induced by module B on module A). However jacoco doesn't seem to support the ability to execute several maven reactors (for example for building code located in various github repositories), using the same jacoco exec file and then generate a report for that. See also https://groups.google.com/forum/#!topic/jacoco/odVzr7P5i6w

Using Clover

Go to the top level module containing children modules for which to generate a Clover report. Note that you won't be able to aggregate Clover data across different Maven runs with this strategy so you really need a single parent module.

1. Run the following command (adjust the local repo path):
   mvn clean clover:setup install clover:aggregate clover:clover -Pclover,integration-tests,dev,jetty,hsqldb,docker -Dxwiki.revapi.skip=true -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -nsu

   • You might need to run the "install" goal instead of the "test" one if your local Maven repository doesn't already contain some test jars (apparently and for some reason Maven won't download them from the remote repository under some conditions).
   • Note that we use -Dmaven.repo.local to use a different Maven local repository so that instrumented source code and built binaries don't find their way in your standard local repository (which you could then deploy by mistake, or that'll simply fail your tests later on when you don't run with the Clover profile since instrumented artifacts require the Clover JAR to be present in the classpath at runtime...).

Multiple Maven Reactors

Using Jacoco

Using Clover

Use a single Clover database to which you add coverage information as you build modules one after another. This strategy is especially useful when you wish to manually run some modules and ensure that coverage data aggregate in a single place so that when you generate the report you have the result of all your runs.

1. Instrument the source code with Clover for all modules that you want to include in the report, using (adjust the paths):
   mvn clover:setup install -Pclover,integration-tests,docker,dev,jetty,hsqldb -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dxwiki.revapi.skip=true -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu

   When tests are executed they'll generate coverage data in the specified Clover database. Since there's a single Clover there's no need to merge Clover databases as in strategy 1 above.

2. To generate the Clover report, execute the following from any module (adjust path):
   mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
3. Remember to clean your clover database when you're done.

Here are typical steps you'd follow to generate full TPC for XWiki:

• Clean your local repo and remove any previous clover DBs:
  rm -R ~/.m2/repository-clover/org/xwiki
  rm -R ~/.m2/repository-clover/com/xpn
  rm -R ~/.xwiki/clover
• Generate coverage data for XWiki Commons:
  cd xwiki-commons
  mvn clean -Pclover,integration-tests,docker -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
  mvn clover:setup install -Pclover,integration-tests,docker -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true  -Dxwiki.revapi.skip=true -nsu
• Generate Clover report just for Commons:
  mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
• Generate coverage data for XWiki Rendering:
  cd xwiki-rendering
  mvn clean -Pclover,integration-tests,docker -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
  mvn clover:setup install -Pclover,integration-tests,docker -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true  -Dxwiki.revapi.skip=true -nsu
• Generate Clover report for Commons and Rendering:
  mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
• Generate coverage data for XWiki Platform:
  cd xwiki-platform
  mvn clean -Pclover,integration-tests,docker -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
  mvn clover:setup install -Pclover,integration-tests,docker -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true  -Dxwiki.revapi.skip=true -nsu
• Generate full Clover report (for Commons, Rendering and Platform):
  mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu

Using Clover + Jenkins

• Install Jenkins LTS and the following plugins:
  • Pipeline plugin
  • Docker Cloud Plugin
• Setup Jenkins to spawn Docker agents using the XWiki Build image.
• Setup a Global Pipeline Library in Jenkins that points to https://github.com/xwiki/xwiki-jenkins-pipeline (make sure it's loaded implicitly).
• Setup a Jenkins Clover pipeline job with a script such as:
  node('docker') {
      xwikiClover([
         [baseline: "20171222-1835", fail: false],
         [baseline: "20190106-0207", fail: false],
         [baseline: "latest", fail: true]
     ])
  }
• You can check the code for the xwikiClover step.
• Check the results.

Note that the Clover pipeline script will generate a report with differences from the previous passing report showing the contributions (positive and negative) of each module to the global TPC.

Example Reports

• All Reports
• XWiki Commons/Rendering (2nd December 2016)
• XWiki Commons/Rendering (21st October 2015)
• XWiki Commons/Rendering/Platform/Enterprise (1st of July 2012)
• XWiki Commons (12 March 2012), XWiki Rendering (12 March 2012)
• XWiki Enterprise + Platform (17 October 2010) (includes functional test coverage)
• XWiki Rendering  (12 October 2010)
• XWiki Rendering  (4 May 2009)
• XWiki Core (12 August 2008)

Code Coverage on a live XWiki instance

Using JaCoCo

You can run any Java program and check its code coverage as it runs. That includes XWiki.

Download JaCoCo from https://www.jacoco.org/jacoco/ and extract it wherever convenient. In the reminder of this section, we will assume it is extracted at $JACOCO_PATH. You can set this variable like this (in every terminal session you will use) if you want to copy paste from this doc.

JaCoCo has two parts that concerns us:

• A Java agent, running with the program of which you want to cover the code. Java agents are specified with their options when running the program with the java command.
• A command line tool that will speak to this agent and generate reports

First, run XWiki with the agent. We fully recommand you to read the JaCoCo agent documentation. We will assume you use the demo package downloaded from Download or XTool. If you run XWiki differently, adapt in consequence.

With the demo package, XWiki is started using start_xwiki.sh (or start_xwiki_debug.sh), and XTool uses it too. You can pass options to Java using the XWIKI_OPTS environment variable, which defaults to -Xmx1024m if you don't set it. Will specify the agent using this environment variable.

Using start_xwiki.sh:

Or using XTool:

This means that XWiki will be run with JaCoCo producing its data file at /tmp/jacoco.exec, listening for orders on TCP port 6300 as instructed by the output=tcpserver. If you wonder where this 6300 comes from, please read its doc :-). This has security consequences, you can also choose that you don't want to enable the TCP listening. Then JaCoCo will dump its analysis to /tmp/jacoco.exec when you stop XWiki.

Then, when you want to see the report, use JaCoCo's command line tool ask JaCoCo to dump its analysis and then to produce an HTML report. See JaCoCo's cli documentation.

Dump the analysis:

Produce the report: