Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 There are various types of tests and everyone has its own terminology. You'll find below the XWiki terminology and best practices related to testing.
6
7 Check the [[general development flow>>dev:Community.DevelopmentPractices||anchor="HGeneralDevelopmentFlow"]] to understand better how testing fits in the larger picture.
8
9 = Testing Strategy =
10
11 Here's the general testing methodology used by the XWiki project:
12
13 * Code committed must have associated automated tests.
14 * There are [[checks in the build>>Community.Building||anchor="HAutomaticChecks"]] (in the ##quality## profile - executed automatically on the CI) to ensure that committed code:
15 ** do not reduce the total test coverage for that module
16 ** do not decrease the mutation score for that module. Note: We use mutation score as a metric to gauge the quality of our tests.
17 * Developers run unit and functional tests on their machines daily and frequently.
18 * Unit and functional tests are executed by the CI, at each commit.
19 * Most automated functional tests are currently running in a single environment (HSQLDB/Jetty/Firefox). However, we've started to use Docked-based testing to test automatically using several configurations and are migrating more and more tests to that. There are jobs in XWiki's CI that execute tests in all configurations regularly (not at each commit though but as much as possible).
20 ** Manual tests are also executed at each release by dedicated contributors and they manually execute tests that are not covered by the automated tests.
21 * Performance tests are executed manually several times per year (usually for LTS releases and sometimes for stable releases too).
22 * Responsiveness tests (e.g. verify that XWiki displays fine on a Mobile) are currently executed manually from time to time by dedicated contributors.
23 * Accessibility (WCAG) tests are performed automatically at each commit.
24 * HTML5 validity tests are performed automatically at each commit.
25
26 = Unit Testing =
27
28 == Java Unit Testing ==
29
30 See [[Java Unit Testing>>Community.Testing.JavaUnitTesting.WebHome]]
31
32 == JavaScript Unit Testing ==
33
34 * These are **tests that do not rely on the DOM**, written as "behavioral specifications", using the [[Jasmine>>http://pivotal.github.io/jasmine/]] test framework.
35 * In development mode, you can start the test runner process by running {{code}}mvn jasmine:bdd{{/code}} in your command line. This will start a test runner at ##http:~/~/localhost:8234##, that will run all tests in the ##src/test/javascript## directory. Write your tests there and hit refresh to see the results directly in your browser.
36 * For tests that need a DOM, see [[Functional Testing>>||anchor="HFunctionalTesting"]]
37
38 == Java Rendering Testing ==
39
40 We have a special framework for making it easy to write Rendering tests, see the [[Rendering Testing Framework>>rendering:Main.Extending||anchor="HAddingTests"]]
41
42 == XAR Testing ==
43
44 See [[XAR Unit Testing>>Community.Testing.XARUnitTesting.WebHome]]
45
46 = Functional Testing =
47
48 A functional test requires a running XWiki instance.
49
50 == Functional UI Testing ==
51
52 We now have 3 frameworks for running GUI tests:
53
54 * One based on Docker (and using Selenium 3+). This is now the recommended framework to use for new tests.
55 * One based on Selenium2/Webdriver which is now deprecated and shouldn't be used. We encourage committers to port tests written for it to Docker tests. Especially when committers bring modification to the old tests we encourage them to rewrite the tests as new Docker tests.
56 * A last one based on Selenium1 which is also deprecated and shouldn't be used. We encourage committers to port tests written for it to Docker tests. Especially when committers bring modification to the old tests we encourage them to rewrite the tests as new Docker tests.
57
58 {{id name="HSelenium3-basedFramework"/}}
59
60 === Docker-based Testing ===
61
62 This is currently the official way to write UI functional tests.
63
64 See [[DockerTesting]].
65
66 === Selenium2-based Framework ===
67
68 Using:
69
70 * To debug a test simply start XWiki somewhere and then debug your JUnit tests as a normal JUnit test in your IDE.
71 ** Note that functional test Maven modules will create an XWiki instance in ##target/xwiki## thanks to the execution of the XWiki Packager plugin, so it's simpler to start this XWiki instance when debugging.
72 * In order to debug more easily flickering tests you can simply add the {{{@Intermittent}}} annotation on your test method and it'll be executed 100 times in a row (you can also specify {{{@Intermittent(repetition=N)}}}to repeat it N times). This is achieved thanks to the [[Tempus Fugit>>http://tempusfugitlibrary.org/documentation/junit/intermittent/]] framework that we've integrated.
73 * To run a specific test, pass the ##pattern## property (it's a regex on the test class name) as in: ##mvn install -Dpattern=TestClass#testName## (this will run the ##testName## test from ##TestClass##)
74 ** More details on the usage of the ##pattern## property and examples can be found in the [[documentation of the XWikiExecutorTestMethodFilter class>>https://github.com/xwiki/xwiki-platform/blob/master/xwiki-platform-core/xwiki-platform-test/xwiki-platform-test-integration/src/main/java/org/xwiki/test/integration/XWikiExecutorTestMethodFilter.java#L25]].
75 * To run the tests on your own running instance (instead of letting Maven close your instance and start a fresh one), use ##-Dxwiki.test.verifyRunningXWikiAtStart=true##. It could be useful to verify that you have not broken the tests on your instance before committing your changes.
76 * By default the Firefox browser will be used but if you wish to run with another browser just pass the ##browser## parameter as in:
77 ** Firefox (default): ##-Dbrowser=*firefox##
78 ** Internet Explorer: ##-Dbrowser=*iexplore##
79 ** Chrome: ##-Dbrowser=*chrome##
80 ** PhantomJS: ##-Dbrowser=*phantomjs##
81 * In case of compatibility problem between the version of the browser and the version of Selenium, you can specify which install of firefox to use thanks to ##-Dwebdriver.firefox.bin##. For example, we use to need Firefox 32 in the past, you could install it locally and refer to it with ##-Dwebdriver.firefox.bin="/path/to/your/firefox-32.0"##.
82 * We check for pages that require Programming Rights automatically by bundling a Listener component that listens to the ##ScriptEvaluatingEvent## event and that drops Programming Rights, in an effort to make the tests fail and so that the developer can notice he requires PR and fix that. In some cases where it's necessary, this can be disabled by setting the configuration {{code}}< testProgrammingRights>false</testProgrammingRights>{{/code}} in the configuration of the Package Mojo, and/or by using a System Property to control where the check is performed, e.g. {{code}}<xwikiPropertiesAdditionalProperties>test.prchecker.excludePattern=.*:XWiki\.XWikiPreferences</xwikiPropertiesAdditionalProperties>{{/code}}. {{info}}Since XWiki 9.8RC1{{/info}}
83 * If the ##xwiki.test.startXWiki## system property is set to ##true## then the test itself will start/stop XWiki. If set to ##false## then it's then the responsibility of the build to start/stop XWiki. Useful when starting.stopping XWiki inside a Docker container handled by the Maven build for example. {{info}}Since XWiki 10.0{{/info}}
84
85 ==== Best practices ====
86
87 * Tests are located in ##xwiki-platform## inside the module that they are testing. Note that in the past we were putting functional tests in ##[[xwiki-platform-distribution/xwiki-platform-distribution-flavor/xwiki-platform-distribution-flavor-test>>https://github.com/xwiki/xwiki-platform/tree/master/xwiki-platform-distribution/xwiki-platform-distribution-flavor/xwiki-platform-distribution-flavor-test]]## but we have started to move them inside the specific modules they are testing.
88 * Name the tests using ##<prefix>IT.java##.
89 * Use a ##AllITs.java## test suite to ensure that XWiki is started/stopped only once for all tests in the module:(((
90 {{code language="java"}}
91 @RunWith(PageObjectSuite.class)
92 public class AllITs
93 {
94 }
95 {{/code}}
96 )))
97 * Use the ##maven-failsafe-plugin## for functional UI tests:(((
98 {{code language="xml"}}
99 <plugin>
100 <groupId>org.apache.maven.plugins</groupId>
101 <artifactId>maven-failsafe-plugin</artifactId>
102 </plugin>
103 {{/code}}
104 )))
105 * Locate tests in ##src/test/java## (i.e the default maven location for tests)
106 * Tests should be written using the Page Objects Pattern:
107 ** [[Original article from Simon Steward>>https://code.google.com/p/selenium/wiki/PageObjects]]
108 ** [[Article from Martin Fowler>>http://martinfowler.com/bliki/PageObject.html]]
109 * Since functional tests take a long time to execute (XWiki to start, Browser to start, navigation, etc) it's important to write tests that execute as fast as possible. Here are some tips:
110 ** Write scenarios (i.e. write only a few test methods, even only one if you can so that you can have a single fixture) instead of a lot of individual tests as you'd do for unit tests.
111 ** Use ##getUtil()## to perform quick actions that are not part of what you wish to test (i.e that are part of the test fixture). For example to create a page you need in your fixture, write:(((
112 {{code language="java"}}
113 getUtil().createPage(...);
114 {{/code}}
115 )))instead of(((
116 {{code language="java"}}
117 WikiEditPage wep = new WikiEditPage();
118 wep.switchToEdit(SPACE_NAME, DOC_NAME);
119 wep.setTitle(DOC_TITLE);
120 wep.setContent(CONTENT);
121 ViewPage vp = wep.clickSaveAndView();
122 {{/code}}
123 )))
124 ** Your Maven module for test may depend on a specific profile (as a best practice, you can use ##integration-tests##). Doing this, it will be built only when specifically asked with ##-Pintegration-tests## (see below for an example of the ##pom.xml##)(((
125 {{code language="xml"}}
126 <profiles>
127 <profile>
128 <id>integration-tests</id>
129 <modules>
130 <module>application-task-test</module>
131 </modules>
132 </profile>
133 </profiles>
134 {{/code}}
135 )))
136 * {{warning}}Never, ever, wait on a timer!{{/warning}} Instead wait on elements.
137 * If you need to create, update or delete a page during your tests, use a space name specific to your test scenario with ##getTestClassName()##. For example:(((
138 {{code language="java"}}
139 getUtil().createPage(getTestClassName(), "Page", "Content", "Title");
140 {{/code}}
141 )))
142 * Tests must not depend on one another. In other words, it should be possible to execute tests in any order and running only one test class should work fine.
143 * Tests that need to change existing configuration (e.g. change the default language, set specific rights, etc) must put back the configuration as it was. This is true only in flavor tests or when several functional tests of different domains are executed one after another. However functional tests located in xwiki-platform specific modules are only running the tests for their module and thus it's not important, and saves times, if they don't clean up.
144 * Tests are allowed to create new documents and don't need to remove them at the end of the test.
145 * Stdout/stderr validation errors(((
146 We have a [[check>>Community.Building.WebHome||anchor="HAutomaticChecks"]] that verifies if functional tests output some invalid content to stdout/stderr. If your test contains such errors it'll fail and then you have 3 options:
147 * It's a normal and expected output from the test (for ex the test verifies an error condition and it's expected it will raise a stack trace in the console). In this case add an expectation in the test. For example:(((
148 {{code language='java'}}
149 public class TemplateTest extends AbstractTest
150 {
151 public void wrongTemplate()
152 {
153 ...
154 this.validateConsole.getLogCaptureConfiguration().registerExpected( "Possible break-in attempt!",
155 "Error getting resource [null]");
156 }
157 ...
158 {{/code}}
159 )))
160 * It's a non-expected error. You then have 2 sub-choices:
161 ** Fix the problem (the best and recommended solution!)
162 ** Increase the technical debt by adding an exclude. For example:(((
163 {{code language='java'}}
164 this.validateConsole.getLogCaptureConfiguration().registerExcludes(
165 "java.lang.IllegalStateException: Response is committed");
166 {{/code}}
167 )))
168 )))
169
170 Examples of functional tests:
171
172 * {{scm path="xwiki-platform-core/xwiki-platform-activeinstalls/xwiki-platform-activeinstalls-test"}}Active Installs Application Functional Tests{{/scm}}
173 * {{scm path="xwiki-platform-core/xwiki-platform-administration/xwiki-platform-administration-test"}}Administration Application Functional Tests{{/scm}}
174 * {{scm user="xwiki-contrib" project="application-faq"}}FAQ Application Functional Tests{{/scm}}
175 * {{scm path="xwiki-platform-core/xwiki-platform-ircbot/xwiki-platform-ircbot-test"}}IRCBot Application Functional Tests{{/scm}}
176 * {{scm path="xwiki-platform-core/xwiki-platform-linkchecker/xwiki-platform-linkchecker-test"}}Link Checker Application Functional Tests{{/scm}}
177 * {{scm path="xwiki-platform-core/xwiki-platform-panels/xwiki-platform-panels-test"}}Panels Application Functional Tests{{/scm}}
178
179 === Old Selenium1-based Framework ===
180
181 * We were using Selenium RC to perform functional tests for GUI. We had created some JUnit extension to easily write Selenium tests in Java.
182 ** Existing tests can be found in ##[[xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium>>https://github.com/xwiki/xwiki-enterprise/tree/master/xwiki-enterprise-test/xwiki-enterprise-test-selenium]]##
183 * To run these tests on your local machine go to ##xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium## and type ##mvn install##.
184 * To run a specific test, pass the ##pattern## property as in: ##mvn install -Dpattern=DeletePageTest## (this will run the ##DeletePageTest## - Note that you don't have to specify the extension). In addition if you wish to execute only a specific method from a Test Case class, you can pass the ##patternMethod## property as in: ##mvn install -Dpattern=DeletePageTest -DpatternMethod=testDeletePageCanDoRedirect##.
185 * To enable debug output from the selenium server start maven with the ##-Ddebug=true## switch and then all messages from the selenium server are saved to: ##xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium/target/selenium/server.log##.
186 * To debug a functional Selenium test in your favourite Java IDE go in ##xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium## and run maven with ##-Dmaven.surefire.debug##. Maven will wait till you connect your IDE to the running JVM on port 5005. You can put a breakpoint in your IDE and debug the test.
187
188 === Browser version ===
189
190 Currently we only run our functional tests on the ##Firefox## browser.
191 The browser you use for functional tests needs to match the selenium version, otherwise unpredictable results may occur.
192
193 * The current ##Selenium## version we are using is ##2.44.0##
194 ** (valid for 18.March.2015. Actual version used can be verified [[here>>https://github.com/xwiki/xwiki-commons/blob/master/pom.xml]] under the ##selenium.version## property)
195 * The ##Firefox## version we use in you continuous integration agents is ##32.0.1##.
196 ** (valid for 18.March.2015. Ask on the [[list>>dev:Community.MailingLists]] or on [[IRC>>dev:Community.IRC]] for the actual used version since it's not publicly verifiable)
197 ** To determine browser compatibility with the Selenium version used, scan [[Selenium's changelog>>https://code.google.com/p/selenium/source/browse/java/CHANGELOG]] and look for entries like "* Updating Native events to support Firefox 24, 31, 32 and 33". That shows the supported browser version for the particular Selenium version.
198
199 If you wish to run tests with the exact configuration as XWiki's [[Continous Integration server>>http://ci.xwiki.org]] uses, you need to install and use locally the same Firefox version. To do so, you have to:
200
201 1. [[Download>>https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/]] the corresponding Firefox release
202 1. Unzip it locally
203 1. Use the ##webdriver.firefox.bin## java system property to specify the location of your firefox version
204 11. Depending on how you are starting the functional tests, you`d have to either add the system property in your maven build (surefire plugin configuration) or in your IDE (run configuration)
205 11. Read [[Selenium's FirefoxDriver documentation>>https://code.google.com/p/selenium/wiki/FirefoxDriver]] for more information and options
206
207 === Adding a new maven dependency for minimal war ===
208
209 All our functional testing are linked to the build of a custom instance of XWiki based on the dependencies contained in the pom.xml of the test.
210 However, those dependencies are added to a minimal war that is built using dependencies located on different places dependending if it is a docker test, or an older functional test.
211
212 So when a new Maven dependency is added for all distributions, it should be declared in three places:
213 * in {{code}}xwiki-platform-minimaldependencies/pom.xml{{/code}}: this will allow the dependency to be retrieved for both docker tests, and for the standard distribution
214 * in {{code}}xwiki-tools-packager-plugin{{/code}}, in both {{code}}pom.xml{{/code}} and {{code}}PackageMojo.java{{/code}}: those are used for older functional tests.
215
216 = XHTML, CSS & WCAG Validations =
217
218 * We are using JUnit to validate that all XWiki pages produce valid XHTML.
219 ** Existing tests can be found in ##enterprise/distribution-test/misc-tests/##
220 * [[WCAG Testing Strategy>>WCAGTesting]]
221 * CSS validation must be verified manually using the [[W3C validator>>http://jigsaw.w3.org/css-validator/]]. See [[XHTML & CSS Coding Style>>XhtmlCssCodeStyle]].
222
223 = Performance Testing =
224
225 * These are memory leakage tests, load tests and speed of execution tests.
226 * They are performed manually and in an ad hoc fashion for now. They are executed for some stable versions and for all super stable versions.
227 * See [[Methodology and reports>>test:Performances.WebHome]].
228
229 See the [[Profiling topic>>Profiling]] for details on how to use a profiler for detecting performance issues.
230
231 = Manual testing =
232
233 {{info}}
234 Here's the spirit we'd like to have from the XWiki Manual Testing Team: [[The Black Team>>http://www.t3.org/tangledwebs/07/tw0706.html]].
235 {{/info}}
236
237 Besides automated testing, ensuring software quality requires that features be tested by actual users (see [[Manual testing>>http://en.wikipedia.org/wiki/Manual_testing]]). In order to manage manual testing, a [[test plan>>http://en.wikipedia.org/wiki/Test_plan]] is required.
238
239 * [[Manual test reports>>test:Main.WebHome]]
240 * [[Manual Test Strategy>>ManualTestStrategy]]
241 * [[Manual Test Reports>>xwiki:TestReports.WebHome]]
242
243 = Tools Reference =
244
245 We use the following tools in our automated tests:
246
247 * [[JUnit>>https://junit.org/junit5/]]: test framework
248 * [[Mockito>>https://site.mockito.org/]]: mocking environment of unit test to isolate it
249 * [[Hamcrest>>http://hamcrest.org/JavaHamcrest/]]: extra assertions beyond what JUnit provides
250 * [[GreenMail>>http://www.icegreen.com/greenmail/]]: for testing email sending
251 * [[WireMock>>http://wiremock.org/]]: for simulating HTTP connections to remote servers
252 * [[Selenium>>https://seleniumhq.github.io/docs/]]: for functional UI test to simulate a user interacting with XWiki
253 * [[TestContainers>>https://www.testcontainers.org/]]: Docker-based functional testing
254 * [[JMeter>>http://jmeter.apache.org/]]: for performance tests
255 * [[Dumbbench>>https://github.com/tsee/dumbbench]]: for manual performance tests
256
257 = Test Coverage =
258
259 See the [[TestCoverage]] page.
260
261 = Mutation Testing =
262
263 We use [[PIT/Descartes>>https://github.com/STAMP-project/pitest-descartes]].
264
265 You can generate a report for the current module by running:
266
267 {{code}}
268 mvn clean install -Pquality -Dxwiki.pitest.skip=false -Djacoco.skip=true
269 {{/code}}
270
271 This will generate 2 reports:
272
273 * The PIT one in the ##target/pit-reports/<timestamp>## directory
274 * The Descartes one, showing the Pseudo Tested and Partially Tested code, which are good indicators of what needs to be fixed, in the ##target/pit-reports/<timestamp>/issues## directory

Get Connected