Wiki source code of Testing

Version 131.2 by Vincent Massol on 2018/11/25 11:30

Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 There are various types of tests and everyone has its own terminology. You'll find below the XWiki terminology and best practices related to testing.
6
7 Check the [[general development flow>>dev:Community.DevelopmentPractices||anchor="HGeneralDevelopmentFlow"]] to understand better how testing fits in the larger picture.
8
9 Here's the general methodology used:
10
11 * Code committed must have associated automated tests. There's [[a check in the build>>Community.Building||anchor="HAutomaticChecks"]] (in the ##quality## profile) to ensure that committed code do not reduce the total test coverage for that module. The CI runs the ##quality## profile for each commit and fails the build if the test coverage is reduced.
12 ** In addition we also check for Test quality by computing Mutation Testing scores for all tests and fail the build if the global quality in a given module is reduced by a new commit (this is checked by some CI jobs every day).
13 * Developers run unit and functional tests on their machines daily and frequently.
14 * Functional tests are executed by the CI, at each commit.
15 * Performance tests are executed manually several times per year (usually for LTS releases and sometimes for stable releases too).
16 * Most automated functional tests are currently running in a single environment (HSQLDB/Jetty/Firefox). However, we've started to use Docked-based testing to test automatically using several configurations and are migrating more and more tests to that. There are jobs in XWiki's CI that execute tests in all configurations regularly (not at each commit though but as much as possible).
17 ** Manual tests are also executed at each release by dedicated contributors and they manually execute tests that are not covered by the automated tests.
18 * Responsiveness tests (e.g. verify that XWiki displays fine on a Mobile) are currently executed manually from time to time by dedicated contributors.
19 * Accessibility tests are performed automatically at each commit.
20 * HTML5 validity tests are performed automatically at each commit.
21
22 = Unit Testing =
23
24 == Java Unit Testing ==
25
26 See [[Java Unit Testing>>Community.Testing.JavaUnitTesting.WebHome]]
27
28 == JavaScript Unit Testing ==
29
30 * These are **tests that do not rely on the DOM**, written as "behavioral specifications", using the [[Jasmine>>http://pivotal.github.io/jasmine/]] test framework.
31 * In development mode, you can start the test runner process by running {{code}}mvn jasmine:bdd{{/code}} in your command line. This will start a test runner at ##http:~/~/localhost:8234##, that will run all tests in the ##src/test/javascript## directory. Write your tests there and hit refresh to see the results directly in your browser.
32 * For tests that need a DOM, see [[Functional Testing>>||anchor="HFunctionalTesting"]]
33
34 == Java Rendering Testing ==
35
36 We have a special framework for making it easy to write Rendering tests, see the [[Rendering Testing Framework>>rendering:Main.Extending||anchor="HAddingTests"]]
37
38 == XAR Testing ==
39
40 See [[XAR Unit Testing>>Community.Testing.XARUnitTesting.WebHome]]
41
42 = Functional Testing =
43
44 A functional test requires a running XWiki instance.
45
46 == GUI tests ==
47
48 We now have 3 frameworks for running GUI tests:
49
50 * One based on Selenium3 and Docker. This is now the recommended framework to use for new tests.
51 * One based on Selenium2/Webdriver which is now deprecated and shouldn't be used. We encourage committers to port tests written for it to the Selenium3 framework. Especially when committers bring modification to the old tests we encourage them to rewrite the tests as new Selenium3 tests.
52 * A last one based on Selenium1 which is also deprecated and shouldn't be used. We encourage committers to port tests written for it to the Selenium3 framework. Especially when committers bring modification to the old tests we encourage them to rewrite the tests as new Selenium3 tests.
53
54 === Selenium3-based Framework ===
55
56 * Based on [[TestContainers>>https://www.testcontainers.org/]] and uses Docker to execute the tests under various Databases, Servlet Engines and Browsers.
57 * The only requirements for running these tests is to have Docker installed locally and to have the user under which you run your IDE and the Maven build be able to use the docker executable.
58 * Configurations options:(((
59 {{info}}
60 Note:
61 * When passing the options as system properties, the format is ##-Dxwiki.test.ui.<Option Name>=value##
62 * When passing as Java annotation attributes, in functional tests, the format is ##@UITest(<Option Name> = <Value>)##
63 {{/info}}
64
65 (% style="overflow-x: auto;" %)
66 (((
67 |=Option Name|=Default Value|=Valid Values|=Description
68 |##browser##|##firefox##|(((
69 * ##firefox## (##Browser.FIREFOX##)
70 * ##chrome## (##Browser.CHROME##)
71 )))|The browser used in the tests. Note that the version of the browser used is controlled by the version of Selenium that is defined in the ##pom.xml## of ##xwiki-platform-test-docker##.
72 |##database##|##hsqldb_embedded##|(((
73 * ##mysql## (##Database.MYSQL##)
74 * ##postgres## (##Database.POSTGRESQL##)
75 * ##hsqldb_embedded## (##Database.HSQLDB_EMBEDDED##)
76 )))|The database used in the tests
77 |##servletEngine##|##jetty_standalone##|(((
78 * ##tomcat## (##ServletEngine.TOMCAT##)
79 * ##jetty## (##ServletEngine.JETTY##)
80 * ##jetty_standalone## (##ServletEngine.JETTY_STANDALONE##)
81 * ##external## (##ServletEngine.EXTERNAL##)
82 )))|The Servlet Engine used in the tests. Use ##external## to use your own started and alreayd provisioned XWiki instance.
83 |##verbose##|##false##|(((
84 * ##true##
85 * ##false##
86 )))|When active, displays more logs in the console (especially container startup logs)
87 |##saveDatabaseData##|##false##|(((
88 * ##true##
89 * ##false##
90 )))|By default, database data is not saved between test executions. {{warning}}Note that if you decide to save database data, they'll be saved under a ##docker## user by Docker and you need your local user to be able to remove them if you want to clean your Maven ##target## directory.{{/warning}}
91 |##offline##|##false##|(((
92 * ##true##
93 * ##false##
94 )))|When offline, the custom XWiki WAR generation and the XWiki provisioning are done solely from your local Maven repository. Otherwise, when artifacts are not present locally or newer SNAPSHOT versions are available, they'll be fetched from Maven remote repositories.
95 |##databaseTag##|[[Latest supported version by XWiki>>Community.DatabaseSupportStrategy]]|Any Docker tag available from Dockerhub for the container image|Version of the database to use. Isn't supported for HSQLDB Embedded since it doesn't run in a Docker container.
96 |##servletEngineTag##|[[Latest supported version by XWiki>>Community.ServletContainerSupportStrategy.WebHome]]|Any Docker tag available from Dockerhub for the container image|Version of the Servlet Engine to use. Isn't supported for Jetty Standalone since it doesn't run in a Docker container.
97 |##jdbcDriverVersion##|Latest version validated by the XWiki dev team|Any version for the specified database that is available on Maven Central, e.g. for MySQL the groupId/artifactId is ##mysql##/##mysql-connector-java##.|
98 |##vnc##|##true##|(((
99 * ##true##
100 * ##false##
101 )))|When active a VNC container is started to record a video of the tests and more generally to allow connecting to the UI running the tests. Useful when debugging.
102 |##properties##|No default (empty)|Check the ##*.vm## files for the configuration files to see the list of possible values.|(% style="word-break: break-all;" %) Velocity properties that are applied when generating XWiki's configuration files: ##xwiki.cfg##, ##xwiki.properties## and ##hibernate.cfg.xml##. Example: Tell XWiki that minification is off: ##-Dxwiki.test.ui.properties.xwikiPropertiesAdditionalProperties=debug.minify=false## or ##@UITest(properties = { "xwikiPropertiesAdditionalProperties=debug.minify=false" })##.
103 )))
104 )))
105 * Tests get injected ##XWikiWebDriver## and ##TestUtils## instances as test method parameters.
106 * Test results is recording in a FLV file in the ##target## directory.
107 * A screenshot of the UI when the test is failing is also taken in the ##target## directory.
108 * When test execute it's possible to connect to the running VNC server and see the UI by using a VNC client and connect to the VNC URL printed in the console, e.g. ##vnc:~/~/vnc:secret@localhost:32936##.
109 * The version for the platform dependencies is specified using the ##platform.version## property in the ##pom.xml## file executing the tests.
110
111 General implementation architecture:
112
113 {{image reference="testcontainers-xwiki.png"/}}
114 //See [[Vincent's blog>>https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperimentations#HExperimentation5:inJavaTestsusingTestContainers]] for more details//
115
116 Example:
117
118 {{code language="java"}}
119 @UITest
120 public class SeleniumTest
121 {
122 @Test
123 public void test(XWikiWebDriver driver, TestUtils setup)
124 {
125 driver.get("http://xwiki.org");
126 assertThat(driver.getTitle(),
127 containsString("XWiki - The Advanced Open Source Enterprise and Application Wiki"));
128 driver.findElement(By.linkText("XWiki's concept")).click();
129 }
130 }
131 {{/code}}
132
133 === Selenium2-based Framework ===
134
135 Using:
136
137 * To debug a test simply start XE somewhere and then debug your JUnit tests as a normal JUnit test in your IDE.
138 ** Note that functional test Maven modules will create an XWiki instance in ##target/xwiki## thanks to the execution of the XWiki Packager plugin, so it's simpler to start this XWiki instance when debugging.
139 * In order to debug more easily flickering tests you can simply add the {{{@Intermittent}}} annotation on your test method and it'll be executed 100 times in a row (you can also specify {{{@Intermittent(repetition=N)}}}to repeat it N times). This is achieved thanks to the [[Tempus Fugit>>http://tempusfugitlibrary.org/documentation/junit/intermittent/]] framework that we've integrated.
140 * To run a specific test, pass the ##pattern## property (it's a regex on the test class name) as in: ##mvn install -Dpattern=TestClass#testName## (this will run the ##testName## test from ##TestClass##)
141 ** More details on the usage of the ##pattern## property and examples can be found in the [[documentation of the XWikiExecutorTestMethodFilter class>>https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwiki-platform-test/xwiki-platform-test-integration/src/main/java/org/xwiki/test/integration/XWikiExecutorTestMethodFilter.java#L25]].
142 * To run the tests on your own running instance (instead of letting Maven close your instance and start a fresh one), use ##-Dxwiki.test.verifyRunningXWikiAtStart=true##. It could be useful to verify that you have not broken the tests on your instance before committing your changes.
143 * By default the Firefox browser will be used but if you wish to run with another browser just pass the ##browser## parameter as in:
144 ** Firefox (default): ##-Dbrowser=*firefox##
145 ** Internet Explorer: ##-Dbrowser=*iexplore##
146 ** Chrome: ##-Dbrowser=*chrome##
147 ** PhantomJS: ##-Dbrowser=*phantomjs##
148 * You may reach a compatibility problem between the version of the browser and the version of Selenium. For example, at the moment, Firefox 32.0 is required. You may install it on your computer and refer to it with ##-Dwebdriver.firefox.bin="/path/to/your/firefox-32.0"##
149 * We check for pages that require Programming Rights automatically by bundling a Listener component that listens to the ##ScriptEvaluatingEvent## event and that drops Programming Rights, in an effort to make the tests fail and so that the developer can notice he requires PR and fix that. In some cases where it's necessary, this can be disabled by setting the configuration {{code}}< testProgrammingRights>false</testProgrammingRights>{{/code}} in the configuration of the Package Mojo, and/or by using a System Property to control where the check is performed, e.g. {{code}}<xwikiPropertiesAdditionalProperties>test.prchecker.excludePattern=.*:XWiki\.XWikiPreferences</xwikiPropertiesAdditionalProperties>{{/code}}. {{info}}Since XWiki 9.8RC1{{/info}}
150 * If the ##xwiki.test.startXWiki## system property is set to ##true## then the test itself will start/stop XWiki. If set to ##false## then it's then the responsibility of the build to start/stop XWiki. Useful when starting.stopping XWiki inside a Docker container handled by the Maven build for example. {{info}}Since XWiki 10.0{{/info}}
151
152 ==== Best practices ====
153
154 * Tests are located in ##xwiki-platform## inside the module that they are testing. Note that in the past we were putting functional tests in ##[[xwiki-platform-distribution/xwiki-platform-distribution-flavor/xwiki-platform-distribution-flavor-test>>https://github.com/xwiki/xwiki-platform/tree/master/xwiki-platform-distribution/xwiki-platform-distribution-flavor/xwiki-platform-distribution-flavor-test]]## but we have started to move them inside the specific modules they are testing.
155 * Name the tests using ##<prefix>IT.java##.
156 * Use a ##AllITs.java## test suite to ensure that XWiki is started/stopped only once for all tests in the module:(((
157 {{code language="java"}}
158 @RunWith(PageObjectSuite.class)
159 public class AllITs
160 {
161 }
162 {{/code}}
163 )))
164 * Use the ##maven-failsafe-plugin## for functional UI tests:(((
165 {{code language="xml"}}
166 <plugin>
167 <groupId>org.apache.maven.plugins</groupId>
168 <artifactId>maven-failsafe-plugin</artifactId>
169 </plugin>
170 {{/code}}
171 )))
172 * Locate tests in ##src/test/java## (i.e the default maven location for tests)
173 * Tests should be written using the Page Objects Pattern:
174 ** [[Original article from Simon Steward>>https://code.google.com/p/selenium/wiki/PageObjects]]
175 ** [[Article from Martin Fowler>>http://martinfowler.com/bliki/PageObject.html]]
176 * Since functional tests take a long time to execute (XWiki to start, Browser to start, navigation, etc) it's important to write tests that execute as fast as possible. Here are some tips:
177 ** Write scenarios (i.e. write only a few test methods, even only one if you can so that you can have a single fixture) instead of a lot of individual tests as you'd do for unit tests.
178 ** Use ##getUtil()## to perform quick actions that are not part of what you wish to test (i.e that are part of the test fixture). For example to create a page you need in your fixture, write:(((
179 {{code language="java"}}
180 getUtil().createPage(...);
181 {{/code}}
182 )))instead of(((
183 {{code language="java"}}
184 WikiEditPage wep = new WikiEditPage();
185 wep.switchToEdit(SPACE_NAME, DOC_NAME);
186 wep.setTitle(DOC_TITLE);
187 wep.setContent(CONTENT);
188 ViewPage vp = wep.clickSaveAndView();
189 {{/code}}
190 )))
191 ** Your Maven module for test may depend on a specific profile (as a best practice, you can use ##integration-tests##). Doing this, it will be built only when specifically asked with ##-Pintegration-tests## (see below for an example of the ##pom.xml##)(((
192 {{code language="xml"}}
193 <profiles>
194 <profile>
195 <id>integration-tests</id>
196 <modules>
197 <module>application-task-test</module>
198 </modules>
199 </profile>
200 </profiles>
201 {{/code}}
202 )))
203 * {{warning}}Never, ever, wait on a timer!{{/warning}} Instead wait on elements.
204 * If you need to create, update or delete a page during your tests, use a space name specific to your test scenario with ##getTestClassName()##. For example:(((
205 {{code language="java"}}
206 getUtil().createPage(getTestClassName(), "Page", "Content", "Title");
207 {{/code}}
208 )))
209 * Tests must not depend on one another. In other words, it should be possible to execute tests in any order and running only one test class should work fine.
210 * Tests that need to change existing configuration (e.g. change the default language, set specific rights, etc) must put back the configuration as it was. This is true only in flavor tests or when several functional tests of different domains are executed one after another. However functional tests located in xwiki-platform specific modules are only running the tests for their module and thus it's not important, and saves times, if they don't clean up.
211 * Tests are allowed to create new documents and don't need to remove them at the end of the test.
212
213 Examples of functional tests:
214
215 * {{scm path="xwiki-platform-core/xwiki-platform-activeinstalls/xwiki-platform-activeinstalls-test"}}Active Installs Application Functional Tests{{/scm}}
216 * {{scm path="xwiki-platform-core/xwiki-platform-administration/xwiki-platform-administration-test"}}Administration Application Functional Tests{{/scm}}
217 * {{scm path="xwiki-platform-core/xwiki-platform-faq/xwiki-platform-faq-test"}}FAQ Application Functional Tests{{/scm}}
218 * {{scm path="xwiki-platform-core/xwiki-platform-ircbot/xwiki-platform-ircbot-test"}}IRCBot Application Functional Tests{{/scm}}
219 * {{scm path="xwiki-platform-core/xwiki-platform-linkchecker/xwiki-platform-linkchecker-test"}}Link Checker Application Functional Tests{{/scm}}
220 * {{scm path="xwiki-platform-core/xwiki-platform-panels/xwiki-platform-panels-test"}}Panels Application Functional Tests{{/scm}}
221
222 === The Office Importer tests ===
223
224 In XWiki 7.3, we have introduced in ##xwiki-platform## some functional tests for the [[extensions:Extension.Office Importer Application]]. To enable them, you need to enable the profile ##office-tests##. An OpenOffice (or LibreOffice) server is needed on your system. You might also need to set an environment variable that points to the office home path, if not standard. This variable is called ##XWIKI_OFFICE_HOME## and can be set like this:
225
226 (((
227 {{code language="sh"}}
228 ## For Unix systems:
229 export XWIKI_OFFICE_HOME="/opt/libreoffice3.6"
230 {{/code}}
231 )))
232
233 You should set this environment variable in your CI agents.
234
235 === Old Selenium1-based Framework ===
236
237 * We were using Selenium RC to perform functional tests for GUI. We had created some JUnit extension to easily write Selenium tests in Java.
238 ** Existing tests can be found in ##[[xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium>>https://github.com/xwiki/xwiki-enterprise/tree/master/xwiki-enterprise-test/xwiki-enterprise-test-selenium]]##
239 * To run these tests on your local machine go to ##xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium## and type ##mvn install##.
240 * To run a specific test, pass the ##pattern## property as in: ##mvn install -Dpattern=DeletePageTest## (this will run the ##DeletePageTest## - Note that you don't have to specify the extension). In addition if you wish to execute only a specific method from a Test Case class, you can pass the ##patternMethod## property as in: ##mvn install -Dpattern=DeletePageTest -DpatternMethod=testDeletePageCanDoRedirect##.
241 * To enable debug output from the selenium server start maven with the ##-Ddebug=true## switch and then all messages from the selenium server are saved to: ##xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium/target/selenium/server.log##.
242 * To debug a functional Selenium test in your favourite Java IDE go in ##xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium## and run maven with ##-Dmaven.surefire.debug##. Maven will wait till you connect your IDE to the running JVM on port 5005. You can put a breakpoint in your IDE and debug the test.
243
244 === Browser version ===
245
246 Currently we only run our functional tests on the ##Firefox## browser.
247 The browser you use for functional tests needs to match the selenium version, otherwise unpredictable results may occur.
248
249 * The current ##Selenium## version we are using is ##2.44.0##
250 ** (valid for 18.March.2015. Actual version used can be verified [[here>>https://github.com/xwiki/xwiki-commons/blob/master/pom.xml]] under the ##selenium.version## property)
251 * The ##Firefox## version we use in you continuous integration agents is ##32.0.1##.
252 ** (valid for 18.March.2015. Ask on the [[list>>dev:Community.MailingLists]] or on [[IRC>>dev:Community.IRC]] for the actual used version since it's not publicly verifiable)
253 ** To determine browser compatibility with the Selenium version used, scan [[Selenium's changelog>>https://code.google.com/p/selenium/source/browse/java/CHANGELOG]] and look for entries like "* Updating Native events to support Firefox 24, 31, 32 and 33". That shows the supported browser version for the particular Selenium version.
254
255 If you wish to run tests with the exact configuration as XWiki's [[Continous Integration server>>http://ci.xwiki.org]] uses, you need to install and use locally the same Firefox version. To do so, you have to:
256
257 1. [[Download>>https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/]] the corresponding Firefox release
258 1. Unzip it locally
259 1. Use the ##webdriver.firefox.bin## java system property to specify the location of your firefox version
260 11. Depending on how you are starting the functional tests, you`d have to either add the system property in your maven build (surefire plugin configuration) or in your IDE (run configuration)
261 11. Read [[Selenium's FirefoxDriver documentation>>https://code.google.com/p/selenium/wiki/FirefoxDriver]] for more information and options
262
263 = XHTML, CSS & WCAG Validations =
264
265 * We are using JUnit to validate that all XE pages produce valid XHTML.
266 ** Existing tests can be found in ##enterprise/distribution-test/misc-tests/##
267 * [[WCAG Testing Strategy>>WCAGTesting]]
268 * CSS validation must be verified manually using the [[W3C validator>>http://jigsaw.w3.org/css-validator/]]. See [[XHTML & CSS Coding Style>>XhtmlCssCodeStyle]].
269
270 = Performance Testing =
271
272 * These are memory leakage tests, load tests and speed of execution tests.
273 * They are performed manually and in an ad hoc fashion for now. They are executed for some stable versions and for all super stable versions.
274 * See [[Methodology and reports>>test:Performances.WebHome]].
275
276 See the [[Profiling topic>>Profiling]] for details on how to use a profiler for detecting performance issues.
277
278 = Manual testing =
279
280 {{info}}
281 Here's the spirit we'd like to have from the XWiki Manual Testing Team: [[The Black Team>>http://www.t3.org/tangledwebs/07/tw0706.html]].
282 {{/info}}
283
284 Besides automated testing, ensuring software quality requires that features be tested by actual users (see [[Manual testing>>http://en.wikipedia.org/wiki/Manual_testing]]). In order to manage manual testing, a [[test plan>>http://en.wikipedia.org/wiki/Test_plan]] is required.
285
286 * [[Manual test reports>>test:Main.WebHome]]
287 * [[Manual Test Strategy>>ManualTestStrategy]]
288 * [[Manual Test Reports>>xwiki:TestReports.WebHome]]
289
290 = Tools Reference =
291
292 We use the following tools in our automated tests:
293
294 * JUnit: test framework
295 * Mockito, JMock: mocking environment of unit test to isolate it
296 * Hamcrest: extra assertions beyond what JUnit provides
297 * [[GreenMail>>http://www.icegreen.com/greenmail/]]: for testing email sending
298 * [[WireMock>>http://wiremock.org/]]: for simulating HTTP connections to remote servers
299 * [[JMeter>>http://jmeter.apache.org/]]: for performance tests
300 * [[Dumbbench>>https://github.com/tsee/dumbbench]]: for manual performance tests
301
302 = Test Coverage =
303
304 {{info}}
305 We now have a [[SonarQube instance>>http://sonar.xwiki.org]] showing Test coverage for both unit tests and integration tests. However it doesn't aggregate coverage data across top level modules (commons, rendering, platform, enterprise, etc).
306 {{/info}}
307
308 We support both [[Jacoco>>http://www.eclemma.org/jacoco/]] and [[Clover>>http://www.atlassian.com/software/clover/overview]] to generate test coverage reports.
309
310 To generate test coverage reports make sure you can [[build>>Building]] your module and then pick one of the following strategies below depending on what you wish to generate.
311
312 == Single Maven Reactor ==
313
314 === Using Jacoco ===
315
316 * Go in the first top level module (e.g. ##xwiki-commons##) and run: {{code}}mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=false -Plegacy,integration-tests -Dxwiki.revapi.skip=true -Dmaven.test.failure.ignore=true{{/code}}
317 * Go in all the other top level modules you wish to add and run: {{code}}mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=true -Plegacy,integration-tests -Dxwiki.revapi.skip=true -Dmaven.test.failure.ignore=true{{/code}}
318 * Then whenever you wish to generate the full test report, run: {{code}}mvn jacoco:report -Djacoco.dataFile=/tmp/jacoco.exec{{/code}}(((
319 {{todo}}
320 Jacoco supports generating report from a single module, it even supports generating aggregated report from several modules using the report-aggregate mojo introduced in 0.7.7 (but note that I don't know if that includes coverage induced by module B on module A). However jacoco doesn't seem to support the ability to execute several maven reactors (for example for building code located in various github repositories), using the same jacoco exec file and then generate a report for that. See also https://groups.google.com/forum/#!topic/jacoco/odVzr7P5i6w
321 {{/todo}}
322 )))
323
324 === Using Clover ===
325
326 Go to the top level module containing children modules for which to generate a Clover report. Note that you won't be able to aggregate Clover data across different Maven runs with this strategy so you really need a single parent module.
327
328 1. Run the following command (adjust the local repo path):(((
329 {{code language="none"}}
330 mvn clean clover:setup install clover:aggregate clover:clover -Pclover,integration-tests,dev,jetty,hsqldb -Dxwiki.revapi.skip=true -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -nsu
331 {{/code}}
332
333 {{info}}
334 * You might need to run the "install" goal instead of the "test" one if your local Maven repository doesn't already contain some test jars (apparently and for some reason Maven won't download them from the remote repository under some conditions).
335 * Note that we use ##-Dmaven.repo.local## to use a different Maven local repository so that instrumented source code and built binaries don't find their way in your standard local repository (which you could then deploy by mistake, or that'll simply fail your tests later on when you don't run with the Clover profile since instrumented artifacts require the Clover JAR to be present in the classpath at runtime...).
336 {{/info}}
337 )))
338
339 == Multiple Maven Reactors ==
340
341 === Using Jacoco ===
342
343 {{todo/}}
344
345 === Using Clover ===
346
347 Use a single Clover database to which you add coverage information as you build modules one after another. This strategy is especially useful when you wish to manually run some modules and ensure that coverage data aggregate in a single place so that when you generate the report you have the result of all your runs.
348
349 1. Instrument the source code with Clover for all modules that you want to include in the report, using (adjust the paths):(((
350 {{code language="none"}}
351 mvn clover:setup install -Pclover,integration-tests,dev,jetty,hsqldb -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dxwiki.revapi.skip=true -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
352 {{/code}}
353
354 When tests are executed they'll generate coverage data in the specified Clover database. Since there's a single Clover there's no need to merge Clover databases as in strategy 1 above.
355 )))
356 1. To generate the Clover report, execute the following from any module (adjust path):(((
357 {{code language="none"}}
358 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
359 {{/code}}
360 )))
361 1. Remember to clean your clover database when you're done.
362
363 {{info}}
364 If you don't wish failing tests to stop the generation of the coverage report, you should pass ##-Dmaven.test.failure.ignore=true## on the command line.
365 {{/info}}
366
367 Here are typical steps you'd follow to generate full TPC for XWiki:
368
369 * Clean your local repo and remove any previous clover DBs:(((
370 {{code}}
371 rm -R ~/.m2/repository-clover/org/xwiki
372 rm -R ~/.m2/repository-clover/com/xpn
373 rm -R ~/.xwiki/clover
374 {{/code}}
375 )))
376 * Generate coverage data for XWiki Commons:(((
377 {{code}}
378 cd xwiki-commons
379 mvn clean -Pclover,integration-tests -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
380 mvn clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true -nsu
381 {{/code}}
382 )))
383 * Generate Clover report just for Commons:(((
384 {{code}}
385 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
386 {{/code}}
387 )))
388 * Generate coverage data for XWiki Rendering:(((
389 {{code}}
390 cd xwiki-rendering
391 mvn clean -Pclover,integration-tests -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
392 mvn clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true -nsu
393 {{/code}}
394 )))
395 * Generate Clover report for Commons and Rendering:(((
396 {{code}}
397 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
398 {{/code}}
399 )))
400 * Generate coverage data for XWiki Platform:(((
401 {{code}}
402 cd xwiki-platform
403 mvn clean -Pclover,integration-tests -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
404 mvn clover:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -Dmaven.test.failure.ignore=true -Dxwiki.revapi.skip=true -nsu
405 {{/code}}
406 )))
407 * Generate full Clover report (for Commons, Rendering and Platform):(((
408 {{code}}
409 mvn clover:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -Dmaven.repo.local=/Users/vmassol/.m2/repository-clover -nsu
410 {{/code}}
411 )))
412
413 === Using Clover + Jenkins ===
414
415 * Install Jenkins 2.0+ and the following plugins:
416 ** Pipeline plugin
417 ** XVnc plugin
418 * Make sure to have Maven ("mvn" executable) + VNCServer ("vncserver" executable) installed on the build agent
419 * Build agent must be running unix
420 * Setup a Pipeline job and make it point to the {{scm project="xwiki-jenkins-pipeline" path="scripts/clover.groovy"}}following pipeline script{{/scm}}.
421
422 Note that the Clover pipeline script will generate a report with differences from the previous passing report showing the contributions (positive and negative) of each module to the global TPC.
423
424 == Example Reports ==
425
426 {{velocity}}
427 #set ($url1 = $xwiki.getPlugin("zipexplorer").getFileLink($doc, "xwiki-core-clover-20080812.zip", "index.html"))
428 #set ($url2 = $xwiki.getPlugin("zipexplorer").getFileLink($doc, "xwiki-rendering-clover-20090504.zip", "index.html"))
429 #set ($url3 = $xwiki.getPlugin("zipexplorer").getFileLink($doc, "xwiki-rendering-clover-20101012.zip", "index.html"))
430
431 * [[All Reports>>http://maven.xwiki.org/site/clover/]]
432 * [[XWiki Commons/Rendering (2nd December 2016)>>http://maven.xwiki.org/site/clover/20161202/]]
433 * [[XWiki Commons/Rendering (21st October 2015)>>http://maven.xwiki.org/site/clover/20151021/]]
434 * [[XWiki Commons/Rendering/Platform/Enterprise (1st of July 2012)>>http://maven.xwiki.org/site/clover/20120701/]]
435 * [[XWiki Commons (12 March 2012)>>http://maven.xwiki.org/site/clover/20120312/xwiki-commons/]], [[XWiki Rendering (12 March 2012)>>http://maven.xwiki.org/site/clover/20120312/xwiki-rendering/]]
436 * [[XWiki Enterprise + Platform (17 October 2010)>>http://maven.xwiki.org/site/clover/20101017/]] (includes functional test coverage)
437 * [[XWiki Rendering (12 October 2010)>>path:$url3]]
438 * [[XWiki Rendering (4 May 2009)>>path:$url2]]
439 * [[XWiki Core (12 August 2008)>>path:$url1]]
440 {{/velocity}}

Get Connected