Wiki source code of Docker-based Testing

Last modified by Vincent Massol on 2019/10/04 17:23
Hide line numbers
  1: {{box cssClass="floatinginfobox" title="**Contents**"}}
  2: {{toc/}}
  3: {{/box}}
  4: 
  5: = Usage =
  6: 
  7: * Based on [[TestContainers>>https://www.testcontainers.org/]] and uses Docker to execute the tests under various Databases, Servlet Engines and Browsers.
  8: ** [[Introductory talk about TestContainers and XWiki at Devoxx FR 2019>>https://youtu.be/0TvWv4L_IJM]] (In French)
  9: * The only requirements for running these tests is to have Docker installed locally and to have the user under which you run your IDE and the Maven build be able to use the docker executable.
 10: * Tests get injected ##XWikiWebDriver##, ##TestUtils##, ##TestInfo## and ##TestReference## instances as test method parameters.
 11: * Test results is recording in a FLV file in the ##target## directory.
 12: * A screenshot of the UI when the test is failing is also taken in the ##target## directory.
 13: * When test execute it's possible to connect to the running VNC server and see the UI by using a VNC client and connect to the VNC URL printed in the console, e.g. ##vnc:~/~/vnc:secret@localhost:32936##.
 14: ** Note that if you want to debug and interact with the XWiki UI you should use a ##servletEngine## other than the default Jetty Standalone one since otherwise the debug point in your IDE will prevent the XWiki code from executing and you'll see a spinning wheel until you step forward in your debugger. You can use for example ##servletEngine = ServletEngine.TOMCAT##.
 15: * The version for the platform dependencies is specified using the ##platform.version## property in the ##pom.xml## file executing the tests.
 16: * The test configuration options can be accessed from tests by specifying a method parameter of type ##TestConfiguration## (see examples below).
 17: ** You can get access to the IP address/port to use to connect to XWiki from the host machine: ##testConfiguration.getServletEngine().getIP()/getPort()##
 18: ** You can get access to the IP address to use to connect to the host machine from inside XWiki: ##testConfiguration.getServletEngine().getHostIP()##
 19: ** You can get access to the IP address/port used internally by XWiki (for example to generate URLs): ##testConfiguration.getServletEngine().getInternalIP()/getInternalPort()##
 20: 
 21: = Configurations options =
 22: 
 23: {{info}}
 24: Note:
 25: * When passing as Java annotation attributes, in functional tests, the format is ##@UITest(<Option Name> = <Value>)##
 26: * When passing the options as System Properties, the format is ##-Dxwiki.test.ui.<Option Name>=<Value>##
 27: ** Except for passing ##properties## where the format is ##-Dxwiki.test.ui.properties.<Property name>=<Property Value>##
 28: ** Except for ##extraJARs##, ##sshPorts## & ##forbiddenServletEngines## which cannot be set from System Properties.
 29: ** Except for passing ##databaseCommands## where the format is ##-Dxwiki.test.ui.database.commands.<Command name>=<Command Value>##
 30: * System properties have priority over Java annotation attributes
 31: * The ##xwiki.test.ui.dependencies## configuration value can only be passed as a system property.
 32: {{/info}}
 33: 
 34: (% style="overflow-x: auto;" %)
 35: (((
 36: |=Option Name|=Default Value|=Valid Values|=Description
 37: |##browser##|##firefox##|(((
 38: * ##firefox## (##Browser.FIREFOX##)
 39: * ##chrome## (##Browser.CHROME##)
 40: )))|The browser used in the tests. Note that the version of the browser used is controlled by the version of Selenium that is defined in the ##pom.xml## of ##xwiki-platform-test-docker##.
 41: |##database##|##hsqldb_embedded##|(((
 42: * ##mysql## (##Database.MYSQL##)
 43: * ##postgresql## (##Database.POSTGRESQL##)
 44: * ##hsqldb_embedded## (##Database.HSQLDB_EMBEDDED##)
 45: * ##oracle DB## (##Database.ORACLE##)
 46: )))|The database used in the tests. Available tags are those listed on the Dockerhub page for the database image used. Note: for Oracle, we only have Oracle 11g Release 2.
 47: |##servletEngine##|##jetty_standalone##|(((
 48: * ##tomcat## (##ServletEngine.TOMCAT##)
 49: * ##jetty## (##ServletEngine.JETTY##)
 50: * ##jetty_standalone## (##ServletEngine.JETTY_STANDALONE##)
 51: * ##external## (##ServletEngine.EXTERNAL##)
 52: )))|The Servlet Engine used in the tests. Use ##external## to use your own started and alreayd provisioned XWiki instance.
 53: |##verbose##|##false##|(((
 54: * ##true##
 55: * ##false##
 56: )))|When active, displays more logs in the console (especially container startup logs)
 57: |##debug##|##false##|(((
 58: * ##true##
 59: * ##false##
 60: )))|When active, displays debug logs in the console (trace levels for Test Containers). Also starts XWiki in debug mode when using the "Jetty Standalone" Servlet Engine (and starts it with suspend on, i.e. wait for the remote debugger to connect to it before progressing, to make it easy to debug XWiki's server side).
 61: |##saveDatabaseData##|##false##|(((
 62: * ##true##
 63: * ##false##
 64: )))|By default, database data is not saved between test executions. {{warning}}Note that if you decide to save database data, they'll be saved under a ##docker## user by Docker and you need your local user to be able to remove them if you want to clean your Maven ##target## directory.{{/warning}}
 65: |##offline##|##false##|(((
 66: * ##true##
 67: * ##false##
 68: )))|When offline, the custom XWiki WAR generation and the XWiki provisioning are done solely from your local Maven repository. Otherwise, when artifacts are not present locally or newer SNAPSHOT versions are available, they'll be fetched from Maven remote repositories. Moreover no docker pull check is performed in offline mode.
 69: |##databaseTag##|[[Latest supported version by XWiki>>Community.DatabaseSupportStrategy]]|Any Docker tag available from Dockerhub for the container image|Version of the database to use. Isn't supported for HSQLDB Embedded since it doesn't run in a Docker container.
 70: |##servletEngineTag##|[[Latest supported version by XWiki>>Community.ServletContainerSupportStrategy.WebHome]]|Any Docker tag available from Dockerhub for the container image|Version of the Servlet Engine to use. Isn't supported for Jetty Standalone since it doesn't run in a Docker container.
 71: |##jdbcDriverVersion##|Latest version validated by the XWiki dev team|Any version for the specified database that is available on Maven Central, e.g. for MySQL the groupId/artifactId is ##mysql##/##mysql-connector-java##.| 
 72: |##vnc##|##true##|(((
 73: * ##true##
 74: * ##false##
 75: )))|When active a VNC container is started to record a video of the tests and more generally to allow connecting to the UI running the tests. Useful when debugging.
 76: |##properties##|No default (empty)|Check the ##*.vm## files for the configuration files to see the list of possible values.|(% style="word-break: break-all;" %)Velocity properties that are applied when generating XWiki's configuration files: ##xwiki.cfg##, ##xwiki.properties## and ##hibernate.cfg.xml##. Example: Tell XWiki that minification is off: ##-Dxwiki.test.ui.properties.xwikiPropertiesAdditionalProperties=debug.minify=false## or ##@UITest(properties = { "xwikiPropertiesAdditionalProperties=debug.minify=false" })##. Note that the ##xwikiCfgPlugins## property will be merged with the default values unless you start the new property value with the ##^## character.
 77: |##extraJARs##|No default (empty)|Comma-separated list of ##<groupId>:<artifactId>##|(% style="word-break: break-all;" %)There are cases where the module being tested cannot be fully deployed as an extension into a running XWiki and it needs to deploy one or several JARs into ##WEB-INF/lib##. One such example is when an extension includes a Hibernate HBM file, as right now only HBM files located in JARs in ##WEB-INF/lib## are supported.
 78: |##sshPorts##|No default (empty)|List of integers|The list of ports that should be SSH-forwarded when connecting from a Docker container to the host (i.e. when using the ##host.testcontainers.internal##. This is in addition to port ##8080## which is always added. For example if you need XWiki to send a mail to a SMTP server running on port 3025 on the host, you should add port 3025 to the list.
 79: |##profiles##|No default (empty)|Comma-separated list of Maven profile ids|When specified the defined profiles are active when resolving dependencies from the current POM.
 80: |##forbiddenServletEngines##|No default (empty)|List of ##ServletEngine##s enums|List of Servlet Engines for which the tests will be skipped (usually because they'd fail on them).
 81: |##databaseProperties##|No default (empty)|See Docker documentation for the target image|List of database docker commands to use and that will override default commands (example of command ##character-set-server=utf8mb4## which will translate into ##~-~-character-set-server=utf8mb4## at runtime).
 82: |##dependencies##|No default (empty)|Comma-separated list of Maven coordinates|List aritfacts that will be provisioned in the running XWiki instance. When specified then current pom.xml dependencies are not used to find out what to provision. This can be used for example in cases when you don't want to use project dependencies in order to not draw older version of dependencies (such as when you need to run functional docker-based tests in a specific more recent vesion of XWiki than the one used to build the code). Accepted format is ##groupId:artifactId##, ##groupId:artifactId:version## and ##groupId:artifactId:type:version##. {{info}}Since 11.2RC1{{/info}}.
 83: 
 84: = Examples =
 85: 
 86: == Full Examples ==
 87: 
 88: * {{scm path="xwiki-platform-core/xwiki-platform-menu/xwiki-platform-menu-test/xwiki-platform-menu-test-docker/src/test/it/org/xwiki/menu/test/ui/MenuIT.java"}}MenuIT{{/scm}}
 89: * {{scm path="xwiki-platform-core/xwiki-platform-mail/xwiki-platform-mail-test/xwiki-platform-mail-test-docker/src/test/it/org/xwiki/mail/test/ui/MailIT.java"}}MailIT{{/scm}}
 90: 
 91: == Example 1: Basic test ==
 92: 
 93: {{code language="java"}}
 94: @UITest
 95: public class SeleniumTest
 96: {
 97:     @Test
 98:     public void test(XWikiWebDriver driver, TestUtils setup)
 99:     {
100:         driver.get("http://xwiki.org");
101:         assertThat(driver.getTitle(),
102:             containsString("XWiki - The Advanced Open Source Enterprise and Application Wiki"));
103:         driver.findElement(By.linkText("XWiki's concept")).click();
104:     }
105: }
106: {{/code}}
107: 
108: == Example 2: Choosing Container + DB + Browser and versions ==
109: 
110: {{code language="java"}}
111: @UITest(database = Database.MYSQL, databaseTag = "5", servletEngine = ServletEngine.TOMCAT, servletEngineTag = "8",
112:   browser = Browser.CHROME)
113: public class MenuIT
114: ...
115: {{/code}}
116: 
117: {{code language="java"}}
118: @UITest(database = Database.POSTGRESQL, databaseTag = "9", servletEngine = ServletEngine.JETTY,
119:   servletEngineTag = "9", browser = Browser.CHROME)
120: public class MenuIT
121: ...
122: {{/code}}
123: 
124: {{code language="java"}}
125: @UITest(database = Database.HSQLDB_EMBEDDED, servletEngine = ServletEngine.JETTY_STANDALONE,
126:   browser = Browser.FIREFOX)
127: public class MenuIT
128: ...
129: {{/code}}
130: 
131: == Example 3: Test using Greenmail + Hibernate HBM ==
132: 
133: {{code language="java"}}
134: @UITest(sshPorts = {
135:         3025
136:     },
137:     properties = {
138:         "xwikiDbHbmCommonExtraMappings=mailsender.hbm.xml",
139:         // Pages created in the tests need to have PR since we ask for PR to send mails so we need to exclude them from
140:         // the PR checker.
141:         "xwikiPropertiesAdditionalProperties=test.prchecker.excludePattern=.*:MailIT\\..*"
142:     },
143:     extraJARs = {
144:         "org.xwiki.platform:xwiki-platform-mail-send-storage"
145:     }
146: )
147: public class MailIT
148: ...
149: {{/code}}
150: 
151: == Example 4: Passing Test Configuration ==
152: 
153: {{code language="java"}}
154: @Test
155: public void verifyMail(TestUtils setup, XWikiWebDriver webDriver, TestConfiguration testConfiguration)
156:     throws Exception
157: {
158:     ....
159: }
160: {{/code}}
161: 
162: == Example 5: Specifying explictly dependencies ==
163: 
164: Example of deploying a ##1.10-SNAPSHOT## version of ##org.xwiki.contrib.latex:latex-export## (and its dependencies) into a ##11.2-SNAPSHOT## version of XWiki. In this example the POM of ##org.xwiki.contrib.latex:latex-export## depends on XWiki 10.2. Thus this example demonstrates running the tests in a more recent version of XWiki than the code was built on.
165: 
166: {{code language="xml"}}
167: ...
168: <parent>
169:   <groupId>org.xwiki.platform</groupId>
170:   <artifactId>xwiki-platform</artifactId>
171:   <version>11.2-SNAPSHOT</version>
172: </parent>
173: <groupId>org.xwiki.contrib.latex</groupId>
174: <artifactId>latex-test</artifactId>
175: <version>1.10-SNAPSHOT</version>
176: ...
177: <plugin>
178:   <groupId>org.apache.maven.plugins</groupId>
179:   <artifactId>maven-failsafe-plugin</artifactId>
180:   <configuration>
181:     <includes>
182:       <include>**/AllIT.java</include>
183:     </includes>
184:     <systemProperties>
185:       <xwiki.test.ui.dependencies>org.xwiki.contrib.latex:latex-export</xwiki.test.ui.dependencies>
186:     </systemProperties>
187:   </configuration>
188: </plugin>
189: {{/code}}
190: 
191: = Best Practices =
192: 
193: == Scenarios ==
194: 
195: Always use scenarios and use JUnit5's ##@Order(...)## annotation. For example:
196: 
197: {{code}}
198: @UITest
199: public class MenuIT
200: {
201:     @Test
202:     @Order(1)
203:     public void verifyMenuInApplicationsIndex(TestUtils setup)
204:     {
205:       ...
206:     }
207: 
208:     @Test
209:     @Order(2)
210:     public void verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
211:     {
212:       ...
213:     }
214: 
215:     @Test
216:     @Order(3)
217:     public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws Exception
218:     {
219:     ...
220:   }
221: }
222: {{/code}}
223: 
224: == Suites ==
225: 
226: Use JUnit5's ##@Nested## annotation. For example:
227: 
228: {{code}}
229: @UITest(...)
230: public class AllIT
231: {
232:     @Nested
233:     @DisplayName("Office Import Tests")
234:     class NestedOfficeImporterIT extends OfficeImporterIT
235:     {
236:     }
237: 
238:     @Nested
239:     @DisplayName("Office Export Tests")
240:     class NestedOfficeExportIT extends OfficeExportIT
241:     {
242:     }
243: }
244: {{/code}}
245: 
246: == Test Reference ==
247: 
248: A lot of tests need to create a wiki page for testing. Example to do this:
249: 
250: {{code language="java"}}
251: @Test
252: public void showAndHideEditComments(TestUtils setup, TestReference reference) throws Exception
253: {
254:         ViewPage vp = setup.gotoPage(reference);
255: ...
256: {{/code}}
257: 
258: More generic way (more verbose but can useful in some cases) using JUnit5's ##TestInfo## class:
259: 
260: {{code language="java"}}
261: public class MailIT
262: {
263:     private String testClassName;
264: 
265:     @BeforeEach
266:     public void setUp(TestInfo info)
267:     {
268:         this.testClassName = info.getTestClass().get().getSimpleName();
269:     }
270:   ...
271: }
272: {{/code}}
273: 
274: == Docker out of Docker ==
275: 
276: The [[XWiki's CI>>https://ci.xwiki.org]] is using a [[Jenkins Agent Docker image>>https://github.com/xwiki/xwiki-jenkins-slave]] to execute the job builds. Thus, the Docker-based functional tests spawn Docker containers (using TestContainers) that execute inside Docker. This is the pattern called DOOD (Docker out of Docker). See [[TC's best practice>>https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/]] about this.
277: 
278: There are some consequences when writing tests for XWiki:
279: * Never use volume mapping or it won't execute fine. This is because that would require the Jenkins Agent Docker container to have that volume mapping defined too and the mapping would need to use the exact host path and container path. So, in short, it won't work. Instead copy files that you need to have in the container. Adn copy file out if need be. Here's an example from the LaTeX extension on how to achieve this:(((
280: {{code language="java"}}
281: try (GenericContainer container = new GenericContainer("blang/latex:ubuntu")) {
282:     // Note: we copy files instead of mounting volumes so that this can work when using DOOD
283:     // (Docker out of Docker).
284:     MountableFile mountableDirectory = MountableFile.forHostPath("target/latex");
285:     container.withCopyFileToContainer(mountableDirectory, "/data");
286: 
287:     container.withCommand("/bin/sh", "-c", "sleep infinity");
288:     container.withLogConsumer(new Slf4jLogConsumer(LOGGER));
289:     container.start();
290:     Container.ExecResult result = container.execInContainer("pdflatex", "-shell-escape", "index.tex");
291:     String stdout = result.getStdout();
292:     LOGGER.info(stdout);
293:     assertTrue(stdout.contains("Output written on index.pdf"));
294: 
295:     container.copyFileFromContainer("/data/index.pdf", "target/latex/index.pdf");
296: }
297: {{/code}}
298: )))
299: 
300: == Stdout/stderr validation errors ==
301: 
302: We have a [[check>>Community.Building.WebHome||anchor="HAutomaticChecks"]] that verifies if functional tests output some invalid content to stdout/stderr. If your test contains such errors it'll fail and then you have 3 options:
303: * It's a normal and expected output from the test (for ex the test verifies an error condition and it's expected it will raise a stack trace in the console). In this case add an expectation in the test. For example:(((
304: {{code language='java'}}
305: public void wrongTemplate(LogCaptureConfiguration logCaptureConfiguration)
306: {
307:     ...
308:     logCaptureConfiguration.registerExpected( "Possible break-in attempt!",
309:         "Error getting resource [null]");
310: }
311: {{/code}}
312: )))
313: * It's a non-expected error. You then have 2 sub-choices:
314: ** Fix the problem (the best and recommended solution!)
315: ** Increase the technical debt by adding an exclude. For example:(((
316: {{code language='java'}}
317: logCaptureConfiguration.registerExcludes(
318:     "java.lang.IllegalStateException: Response is committed");
319: {{/code}}
320: )))
321: 
322: = Test Resources =
323: 
324: You might need test resources to be used in interaction with the wiki (e.g. to upload an attachment). Any test resources placed in ##src/test/resources## is made automatically available to the browser container by mounting the dedicated volume mapped to ##target/test-classes##. Then, in your test, you can use the dedicated method to get access to the files located in ##target/test-classes##.
325: 
326: For example:
327: 
328: {{code language="java"}}
329: String resourceDirPath = testConfiguration.getBrowser().getTestResourcesPath();
330: File myResourceFile = new File(resourceDirPath, filename);
331: {{/code}}
332: 
333: = Test Coverage =
334: 
335: Activate support for Clover by setting the ##xwiki.test.ui.profiles## property in the ##clover## profile in the ##pom.xml## file for the test.
336: 
337: Example:
338: 
339: {{code language="xml"}}
340:   <profiles>
341:     <profile>
342:       <id>clover</id>
343:       <!-- Add the Clover JAR to the WAR so that it's available at runtime when XWiki executes.
344:            It's needed because instrumented jars in the WAR will call Clover APIs at runtime when they execute. -->
345:       <dependencies>
346:         <dependency>
347:           <groupId>org.openclover</groupId>
348:           <artifactId>clover</artifactId>
349:         </dependency>
350:       </dependencies>
351:       <build>
352:         <plugins>
353:           <plugin>
354:             <groupId>org.apache.maven.plugins</groupId>
355:             <artifactId>maven-failsafe-plugin</artifactId>
356:             <configuration>
357:               <systemProperties combine.children="append">
358:                 <!-- Tell the Docker-based test to activate the Clover profile so that the Clover JAR is added to
359:                      WEB-INF/lib -->
360:                 <property>
361:                   <name>xwiki.test.ui.profiles</name>
362:                   <value>clover</value>
363:                 </property>
364:               </systemProperties>
365:             </configuration>
366:           </plugin>
367: ...
368: {{/code}}
369: 
370: = Existing XWiki instance =
371: 
372: If you want to avoid the Docker-based tests to start and stop XWiki every time you re-run the tests (for example when you're developping/debugging something), you can tell the Docker Test framework to not provision an XWiki instance and to use your XWiki instance instead.
373: 
374: This is achieved by specifying {{code}}@UITest(servletEngine = ServletEngine.EXTERNAL){{/code}} or passing the {{code}}xwiki.test.ui.servletEngine=external{{/code}} system property.
375: 
376: A common use case is to debug a failing test. Here's what you can do:
377: 
378: {{code language="none"}}
379: ## Run the test once to see if it fails and to generate the XWiki instance. This may take some time.
380: > mvn clean install -Dit.test=NavigationPanelIT
381: ## Open a new console, look for the generated XWiki instance inside the target folder and start it.
382: ../target/../jetty > start_xwiki.sh
383: ## Modify the test as needed then go back to the initial console and compile the tests.
384: ## Note that you may also need to build the page objects module if you change those too.
385: > mvn compiler:testCompile
386: ## Then run the test again to see the changes. This should be way faster than the initial test run.
387: > mvn surefire:test -Dxwiki.test.ui.servletEngine=external -Dtest=NavigationPanelIT
388: {{/code}}
389: 
390: = Architecture =
391: 
392: {{image reference="xwiki-testcontainers.png"/}}
393: //See [[Vincent's blog>>https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperimentations#HExperimentation5:inJavaTestsusingTestContainers]] for more details//
394: )))
395: 
396: = CI Jobs =
397: 
398: There are 3 CI jobs executing Docker-based functional tests:
399: 
400: * [[Job 1>>https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/]]: run all tests on latest versions of supported configs.
401: * [[Job 2>>https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/]]: run smoke tests (ie a few tests only) on all supported configs.
402: * [[Job 3>>https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/]]: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc) that we want to support in the future.
403: 
404: Configuration definitions and job sources are all {{scm project="xwiki-jenkins-pipeline" path="src/org/xwiki/jenkins/DockerTests.groovy"}}here{{/scm}}.
Wiki source code of Docker-based Testing

Quick Links

My Recent Modifications

About

About

Support

Platform

User Guide

Admin Guide

Developer Guide

Projects

XWiki

Extensions

Other

Contribute

Status

Practices

Under the Hood

Get Involved

Get Connected
