Wiki source code of Test Coverage

Last modified by Vincent Massol on 2023/04/14 15:19

Hide last authors
Vincent Massol 7.1 1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
Raphaël Jakse 8.1 5 = Code Coverage while running Maven tests =
6
Vincent Massol 1.1 7 {{info}}
8 We now have a [[SonarQube instance>>http://sonar.xwiki.org]] showing Test coverage for both unit tests and integration tests. However it doesn't aggregate coverage data across top level modules (commons, rendering, platform, enterprise, etc).
9 {{/info}}
10
11 We support both [[Jacoco>>http://www.eclemma.org/jacoco/]] and [[Clover>>http://www.atlassian.com/software/clover/overview]] to generate test coverage reports.
12
Vincent Massol 1.2 13 To generate test coverage reports make sure you can [[build>>Community.Building]] your module and then pick one of the following strategies below depending on what you wish to generate.
Vincent Massol 1.1 14
Raphaël Jakse 8.1 15 == Single Maven Reactor ==
Vincent Massol 1.1 16
Raphaël Jakse 8.1 17 === Using Jacoco ===
Vincent Massol 1.1 18
Raphaël Jakse 7.3 19 * Go in the first top level module (e.g. ##xwiki-commons##, also works for extensions) and run: {{code}}mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=false -Plegacy,integration-tests -Dxwiki.revapi.skip=true -Dmaven.test.failure.ignore=true{{/code}}
Vincent Massol 1.1 20 * Go in all the other top level modules you wish to add and run: {{code}}mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=true -Plegacy,integration-tests -Dxwiki.revapi.skip=true -Dmaven.test.failure.ignore=true{{/code}}
21 * Then whenever you wish to generate the full test report, run: {{code}}mvn jacoco:report -Djacoco.dataFile=/tmp/jacoco.exec{{/code}}(((
22 {{todo}}
23 Jacoco supports generating report from a single module, it even supports generating aggregated report from several modules using the report-aggregate mojo introduced in 0.7.7 (but note that I don't know if that includes coverage induced by module B on module A). However jacoco doesn't seem to support the ability to execute several maven reactors (for example for building code located in various github repositories), using the same jacoco exec file and then generate a report for that. See also https://groups.google.com/forum/#!topic/jacoco/odVzr7P5i6w
24 {{/todo}}
25 )))
26
Raphaël Jakse 8.1 27 === Using Clover ===
Vincent Massol 1.1 28
29 Go to the top level module containing children modules for which to generate a Clover report. Note that you won't be able to aggregate Clover data across different Maven runs with this strategy so you really need a single parent module.
30
31 1. Run the following command (adjust the local repo path):(((
32 {{code language="none"}}
Vincent Massol 8.9 33 mvn clean clover:setup install clover:aggregate clover:clover -Pclover,integration-tests,dev,jetty,hsqldb,docker -Dxwiki.revapi.skip=true -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -nsu
Vincent Massol 1.1 34 {{/code}}
35
36 {{info}}
37 * You might need to run the "install" goal instead of the "test" one if your local Maven repository doesn't already contain some test jars (apparently and for some reason Maven won't download them from the remote repository under some conditions).
38 * Note that we use ##-Dmaven.repo.local## to use a different Maven local repository so that instrumented source code and built binaries don't find their way in your standard local repository (which you could then deploy by mistake, or that'll simply fail your tests later on when you don't run with the Clover profile since instrumented artifacts require the Clover JAR to be present in the classpath at runtime...).
39 {{/info}}
40 )))
41
Raphaël Jakse 8.1 42 == Multiple Maven Reactors ==
Vincent Massol 1.1 43
Raphaël Jakse 8.1 44 === Using Jacoco ===
Vincent Massol 1.1 45
46 {{todo/}}
47
Raphaël Jakse 8.1 48 === Using Clover ===
Vincent Massol 1.1 49
50 Use a single Clover database to which you add coverage information as you build modules one after another. This strategy is especially useful when you wish to manually run some modules and ensure that coverage data aggregate in a single place so that when you generate the report you have the result of all your runs.
51
52 1. Instrument the source code with Clover for all modules that you want to include in the report, using (adjust the paths):(((
53 {{code language="none"}}
54 mvn clover:setup install -Pclover,integration-tests,dev,jetty,hsqldb -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dxwiki.revapi.skip=true -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
55 {{/code}}
56
57 When tests are executed they'll generate coverage data in the specified Clover database. Since there's a single Clover there's no need to merge Clover databases as in strategy 1 above.
58 )))
59 1. To generate the Clover report, execute the following from any module (adjust path):(((
60 {{code language="none"}}
61 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
62 {{/code}}
63 )))
64 1. Remember to clean your clover database when you're done.
65
66 {{info}}
67 If you don't wish failing tests to stop the generation of the coverage report, you should pass ##-Dmaven.test.failure.ignore=true## on the command line.
68 {{/info}}
69
70 Here are typical steps you'd follow to generate full TPC for XWiki:
71
72 * Clean your local repo and remove any previous clover DBs:(((
73 {{code}}
74 rm -R ~/.m2/repository-clover/org/xwiki
75 rm -R ~/.m2/repository-clover/com/xpn
76 rm -R ~/.xwiki/clover
77 {{/code}}
78 )))
79 * Generate coverage data for XWiki Commons:(((
80 {{code}}
81 cd xwiki-commons
82 mvn clean -Pclover,integration-tests -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
83 mvn clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true -nsu
84 {{/code}}
85 )))
86 * Generate Clover report just for Commons:(((
87 {{code}}
88 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
89 {{/code}}
90 )))
91 * Generate coverage data for XWiki Rendering:(((
92 {{code}}
93 cd xwiki-rendering
94 mvn clean -Pclover,integration-tests -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
95 mvn clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true -nsu
96 {{/code}}
97 )))
98 * Generate Clover report for Commons and Rendering:(((
99 {{code}}
100 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
101 {{/code}}
102 )))
103 * Generate coverage data for XWiki Platform:(((
104 {{code}}
105 cd xwiki-platform
106 mvn clean -Pclover,integration-tests -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
107 mvn clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true -nsu
108 {{/code}}
109 )))
110 * Generate full Clover report (for Commons, Rendering and Platform):(((
111 {{code}}
112 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
113 {{/code}}
114 )))
115
Raphaël Jakse 8.1 116 === Using Clover + Jenkins ===
Vincent Massol 1.1 117
Vincent Massol 6.1 118 * Install Jenkins LTS and the following plugins:
Vincent Massol 1.1 119 ** Pipeline plugin
Vincent Massol 6.1 120 ** Docker Cloud Plugin
121 * Setup Jenkins to spawn Docker agents using the [[XWiki Build image>>https://github.com/xwiki/xwiki-docker-build/tree/master/build]].
122 * Setup a Global Pipeline Library in Jenkins that points to https://github.com/xwiki/xwiki-jenkins-pipeline (make sure it's loaded implicitly).
123 * Setup a Jenkins Clover pipeline job with a script such as:(((
Raphaël Jakse 7.3 124 {{code language="groovy"}}
Vincent Massol 6.1 125 node('docker') {
126 xwikiClover([
127 [baseline: "20171222-1835", fail: false],
128 [baseline: "20190106-0207", fail: false],
129 [baseline: "latest", fail: true]
130 ])
131 }
132 {{/code}}
133 )))
134 * You can check the {{scm project="xwiki-jenkins-pipeline" path="vars/xwikiClover.groovy"}}code for the ##xwikiClover## step{{/scm}}.
135 * Check the [[results>>http://maven.xwiki.org/site/clover/]].
Vincent Massol 1.1 136
137 Note that the Clover pipeline script will generate a report with differences from the previous passing report showing the contributions (positive and negative) of each module to the global TPC.
138
Raphaël Jakse 8.1 139 === Example Reports ===
Vincent Massol 1.1 140
141 * [[All Reports>>http://maven.xwiki.org/site/clover/]]
142 * [[XWiki Commons/Rendering (2nd December 2016)>>http://maven.xwiki.org/site/clover/20161202/]]
143 * [[XWiki Commons/Rendering (21st October 2015)>>http://maven.xwiki.org/site/clover/20151021/]]
144 * [[XWiki Commons/Rendering/Platform/Enterprise (1st of July 2012)>>http://maven.xwiki.org/site/clover/20120701/]]
145 * [[XWiki Commons (12 March 2012)>>http://maven.xwiki.org/site/clover/20120312/xwiki-commons/]], [[XWiki Rendering (12 March 2012)>>http://maven.xwiki.org/site/clover/20120312/xwiki-rendering/]]
146 * [[XWiki Enterprise + Platform (17 October 2010)>>http://maven.xwiki.org/site/clover/20101017/]] (includes functional test coverage)
Vincent Massol 5.2 147 * [[XWiki Rendering (12 October 2010)>>attach:xwiki-rendering-clover-20101012.zip]]
148 * [[XWiki Rendering (4 May 2009)>>attach:xwiki-rendering-clover-20090504.zip]]
149 * [[XWiki Core (12 August 2008)>>attach:xwiki-core-clover-20080812.zip]]
Raphaël Jakse 8.1 150
151 = Code Coverage on a live XWiki instance =
152
Raphaël Jakse 8.2 153 == Using JaCoCo ==
Raphaël Jakse 8.1 154
155 {{info}}
Raphaël Jakse 8.6 156 JaCoCo comes with the [[EclEmma Eclipse plugin>>https://www.jacoco.org/]] which probably provides a more user-friendly / integrated way to get code coverage if you use Eclipse. If you use Eclipse, you may want to look into it, and document how to use the EclEmma here. Here, we describe how to use JaCoCo using the command line.
Raphaël Jakse 8.1 157
Raphaël Jakse 8.7 158 We will assume you use a POSIX shell (bash, zsh... likely the case if you run GNU/Linux, macOS or WSL. Other users, including people using Powershell or CMD on Windows, will need to adapt the steps to make them work on their setup).
Raphaël Jakse 8.1 159 {{/info}}
160
Raphaël Jakse 8.8 161 You can run any Java program and check its code coverage as it runs. That includes XWiki.
Raphaël Jakse 8.1 162
163 Download JaCoCo from https://www.jacoco.org/jacoco/ and extract it wherever convenient. In the reminder of this section, we will assume it is extracted at ##$JACOCO_PATH##. You can set this variable like this (in every terminal session you will use) if you want to copy paste from this doc.
164
165 {{code}}
166 JACOCO_PATH=/home/raph/dl/jacoco-0.8.8/
167 {{/code}}
168
169 JaCoCo has two parts that concerns us:
170
171 * A Java agent, running with the program of which you want to cover the code. Java agents are specified with their options when running the program with the ##java## command.
172 * A command line tool that will speak to this agent and generate reports
173
Raphaël Jakse 8.5 174 First, run XWiki with the agent. We fully recommand you to read the [[JaCoCo agent documentation>>https://www.jacoco.org/jacoco/trunk/doc/agent.html]]. We will assume you use the demo package downloaded from [[xwiki:Download.WebHome]] or [[XTool>>https://github.com/aubincleme/xtool]]. If you run XWiki differently, adapt in consequence.
Raphaël Jakse 8.1 175
176 With the demo package, XWiki is started using ##start_xwiki.sh## (or ##start_xwiki_debug.sh##), and XTool uses it too. You can pass options to Java using the ##XWIKI_OPTS## environment variable, which defaults to ##-Xmx1024m## if you don't set it. Will specify the agent using this environment variable.
177
178 Using ##start_xwiki.sh##:
179
180 {{code}}
181 XWIKI_OPTS="-Xmx1024m -javaagent:$JACOCO_PATH=destfile=/tmp/jacoco.exec,append=false,output=tcpserver" ./start_xwiki.sh
182 {{/code}}
183
184 Or using XTool:
185
186 {{code}}
187 XWIKI_OPTS="-Xmx1024m -javaagent:$JACOCO_PATH=destfile=/tmp/jacoco.exec,append=false,output=tcpserver" x start myinstance
188 {{/code}}
189
Raphaël Jakse 8.3 190 This means that XWiki will be run with JaCoCo producing its data file at ##/tmp/jacoco.exec##, listening for orders on TCP port 6300 as instructed by the ##output=tcpserver##. If you wonder where this 6300 comes from, please read its doc :-). This has security consequences, you can also choose that you don't want to enable the TCP listening. Then JaCoCo will dump its analysis to ##/tmp/jacoco.exec## when you stop XWiki.
Raphaël Jakse 8.1 191
192 {{info}}
193 It is fully expected that the JaCoCo exec file (here ##/tmp/jacoco.exec##) is present as soon as XWiki starts but empty until JaCoCo dumps its analysis. Which happens when we ask it to, or, if you didn't provide the ##output## parameter, at the end of XWiki's execution.
194 {{/info}}
195
196 {{info}}
Raphaël Jakse 8.4 197 if you want to keep JaCoCo's analysis accross reboots for later report generation, or if you are low on space in ##/tmp##, which can happen if ##/tmp## is a ##tmpfs## (in RAM) and you don't have a lot of it, pick another location.
Raphaël Jakse 8.1 198 {{/info}}
199
200 Then, when you want to see the report, use JaCoCo's command line tool ask JaCoCo to dump its analysis and then to produce an HTML report. See [[JaCoCo's cli documentation>>https://www.jacoco.org/jacoco/trunk/doc/cli.html]].
201
202 Dump the analysis:
203
204 {{code}}
205 java -jar "$JACOCO_PATH/lib/jacococli.jar" dump --destfile /tmp/jacoco.exec
206 {{/code}}
207
208 Produce the report:
209
210 {{code}}
211 mkdir -p "$HOME/jacocohtml" # you can pick another location
212 java -jar "$JACOCO_PATH/lib/jacococli.jar" report /tmp/jacoco.exec --classfiles=$HOME/.xtool/instances/myinstance/data/extension/repository/ --sourcefiles /path/to/your-source-code/src/main/java/ --html "$HOME/jacocohtml"
213 {{/code}}
214
215 {{info}}
216 JaCoCo needs your class files and the path to your source code. Here we provide an example where we are using XTool and we named our XWiki instance ##myinstance##. Change this path to point to the right ##data/extension/repository## directory. If you (also) want to analyse classes provided with XWiki, try to provide the path to ##WEB-INF/lib## (I did not try this). you can provide several paths by using a colon separator (##:##)
217
218 {{code}}
219 java -jar "$JACOCO_PATH/lib/jacococli.jar" report /tmp/jacoco.exec --classfiles=$HOME/.xtool/instances/myinstance/data/extension/repository/:$HOME/.xtool/instances/myinstance/webapps/xwiki/WEB-INF/lib/ --sourcefiles /path/to/your-source-code/src/main/java/:/path/to/xwiki-source-code/xwiki-platform/xwiki-platform-xxx/xxx/src/main/java/ --html "$HOME/jacocohtml"
220 {{/code}}
221 {{/info}}

Get Connected