Testing

Last modified by Vincent Massol on 2014/10/01 11:16

There are various types of tests and everyone has its own terminology. Here are the XWiki terminology and best practices related to testing.

Unit Testing

A unit test only tests a single class in isolation from other classes. Since in the XWiki project we write code using Components, this means a unit test is testing a Component in isolation from other Components.

Java Unit Testing

  • These are tests performed in isolation using Mock Objects. More specifically we're using JUnit 4.x and Mockito (we were using JMock 2.x before and still have a lot of tests not converted to Mockito yet)
  • These tests must not interact with the environment (Database, Container, File System, etc) and do not need any setup to execute
  • Your Maven module must depend on the xwiki-commons-test-simple (for tests not testing Components) or xwiki-commons-test-component (for tests testing Components) modules.
    • If you're testing Components, use MockitoComponentMockingRule (its javadoc explains how to use it in details)
      • Example 1: Canonical use case
        public class DefaultDiffManagerTest
        {
           @Rule
           public final MockitoComponentMockingRule<DiffManager> mocker =
               new MockitoComponentMockingRule(DefaultDiffManager.class);
        ...
           @Test
           public void testDiffStringList() throws Exception
           {
               // Null

                DiffResult<String> result = this.mocker.getComponentUnderTest().diff(null, null, null);
        ...
      • Example 2: Example showing how to not mock one dependency being injected (ObservationManager is excluded in the example)
        @ComponentList({DefaultLoggerManager.class, DefaultObservationManager.class, LogbackEventGenerator.class})
        public class DefaultLoggerManagerTest
        {
           @Rule
           public final MockitoComponentMockingRule<DefaultLoggerManager> mocker =
               new MockitoComponentMockingRule<DefaultLoggerManager>(DefaultLoggerManager.class);
        ...
    • If you're testing non Components, just use pure JUnit and Mockito (you're on your own!)
  • Examples:

Some best practices:

  • Name the Test class with the name of the class under test suffixed with Test. For example the JUnit test class for XWikiMessageTool should be named XWikiMessageToolTest
  • Name the test methods with the method to test followed by a qualifier describing the test. For example importWithHeterogeneousEncodings().

Some tips:

  • When Mocking, to ignore all debug logging calls and to allow all calls to is*Enabled you could write:
    // Ignore all calls to debug() and enable all logs so that we can assert info(), warn() and error() calls.
    ignoring(any(Logger.class)).method("debug");
    allowing(any(Logger.class)).method("is.*Enabled"); will(returnValue(true));

JavaScript Unit Testing

  • These are tests that do not rely on the DOM, written as "behavioral specifications", using the Jasmine test framework.
  • In development mode, you can start the test runner process by running mvn jasmine:bdd in your command line. This will start a test runner at http://localhost:8234, that will run all tests in the src/test/javascript directory. Write your tests there and hit refresh to see the results directly in your browser.
  • For tests that need a DOM, see Functional Testing

Integration Testing

An integration test tests several classes together but without a running XWiki instance. For example if you have one Component which has dependencies on other Components and they are tested together this is called Integration testing.

Java Integration Testing

  • These tests are written using JUnit 4.x and Mockito (we were using JMock 2.x before and still have a lot of tests not converted to Mockito yet).
  • Your Maven module must depend on the xwiki-commons-test module.
  • Use the MockitoComponentMockingRule Rule (its javadoc explains how to use it in details).
  • Examples:
    • example2
    • Other example:
      @ComponentList({
          DefaultExtensionLicenseManager.class
      })
      public class DefaultExtensionLicenseManagerTest
      {
         @Rule
         public final ComponentManagerRule componentManager = new ComponentManagerRule();
      ...
         @Before
         public void setUp() throws Exception
         {
             this.licenseManager = this.componentManager.getInstance(ExtensionLicenseManager.class);
         }
      ...
    • Another example (using the new @BeforeComponent annotation):
      @ComponentList({
          DefaultExtensionManagerConfiguration.class,
          DefaultLoggerManager.class,
          DefaultObservationManager.class
      })
      public class DefaultExtensionManagerConfigurationTest
      {
         @Rule
         public final MockitoComponentManagerRule componentManager = new MockitoComponentManagerRule();

         private ExtensionManagerConfiguration configuration;

         private MemoryConfigurationSource source;

         @BeforeComponent
         public void registerComponents() throws Exception
         {
             // Register a Mocked Environment since we need to provide one.
             this.componentManager.registerMockComponent(Environment.class);

             // Register some in-memory Configuration Source for the test
             this.source = this.componentManager.registerMemoryConfigurationSource();
         }

         @Before
         public void setUp() throws Exception
         {
             this.configuration = this.componentManager.getInstance(ExtensionManagerConfiguration.class);
         }
      ...

      @BeforeComponent is used by ComponentManagerRule and methods annotated with it are called before other components are registered (i.e. before processing of @AllComponents and @ComponentList annotations).

      MockitoComponentManagerRule extends ComponentManagerRule and just adds some helper methods to register a mock component.

Some best practices:

Java Rendering Testing

We have a special framework for making it easy to write Rendering tests, see the Rendering Testing Framework

Functional Testing

A functional test requires a running XWiki instance.

GUI tests

We now have 2 frameworks for running GUI tests:

  • One based on Selenium2/Webdriver which is the framework to use when writing new functional UI tests.
  • One based on Selenium1 which is now deprecated and shouldn't be used. We encourage committers to port tests written for it to the Selenium2 framework. Especially when committers bring modification to the old tests we encourage them to rewrite the tests as new Selenium2 tests.

Selenium2-based Framework

Using:

  • To debug a test simply start XE somewhere and then debug your JUnit tests as a normal JUnit test in your IDE.
  • In order to debug more easily flickering tests you can simply add the @Intermittent annotation on your test method and it'll be executed 100 times in a row (you can also specify @Intermittent(repetition=N)to repeat it N times). This is achieved thanks to the Tempus Fugit framework that we've integrated.
  • To run a specific test, pass the pattern property (it's a regex on the test class name) as in: mvn install -Dpattern=TestClass#testName (this will run the testName test from TestClass)
  • To run the tests on your own running instance (instead of letting mvn close your instance and start a fresh one), use -Dxwiki.test.verifyRunningXWikiAtStart=true. It could be useful to verify that you have not broken the tests on your instance before committing your changes.
  • By default the Firefox browser will be used but if you wish to run with another browser just pass the browser parameter as in:
    • Firefox (default): -Dbrowser=*firefox
    • Internet Explorer: -Dbrowser=*iexplore
    • Chrome: -Dbrowser=*chrome
    • PhantomJS: -Dbrowser=*phantomjs

Best practices:

  • Tests are located in xwiki-platform inside the module that they are testing. Note that in the past we were putting functional tests in xwiki-enterprise/xwiki-enterprise-test/xwiki-enterprise-test-ui but we have now started to move them to xwiki-platform.
  • Tests should be written using the Page Objects Pattern.
  • Since functional tests take a long time to execute (XWiki to start, Browser to start, navigation, etc) it's important to write tests that execute as fast as possible. Here are some tips:
    • Write scenarios (i.e. write only a few test methods, even only one if you can so that you can have a single fixture) instead of a lot of individual tests as you'd do for unit tests.
    • Use getUtil() to perform quick actions that are not part of what you wish to test. For example to create a page you need in your fixture, write:
      getUtil().createPage(...);
      instead of
      WikiEditPage wep = new WikiEditPage();
      wep.switchToEdit(SPACE_NAME, DOC_NAME);
      wep.setTitle(DOC_TITLE);
      wep.setContent(CONTENT);
      ViewPage vp = wep.clickSaveAndView();
  • Never, ever, wait on a timer! Instead wait on elements.
  • Tests must not depend on one another. In other words, it should be possible to execute tests in any order and running only one test class should work fine.
  • Tests that need to change existing configuration (e.g. change the default language, set specific rights, etc) must put back the configuration as it was.
  • Tests are allowed to create new documents and don't need to remove them at the end of the test.

Examples of functional tests:

Old Selenium1-based Framework

  • We were using Selenium RC to perform functional tests for GUI. We had created some JUnit extension to easily write Selenium tests in Java.
  • To run these tests on your local machine go to xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium and type mvn install.
  • To run a specific test, pass the pattern property as in: mvn install -Dpattern=DeletePageTest (this will run the DeletePageTest - Note that you don't have to specify the extension). In addition if you wish to execute only a specific method from a Test Case class, you can pass the patternMethod property as in: mvn install -Dpattern=DeletePageTest -DpatternMethod=testDeletePageCanDoRedirect.
  • To enable debug output from the selenium server start maven with the -Ddebug=true switch and then all messages from the selenium server are saved to: xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium/target/selenium/server.log.
  • To debug a functional Selenium test in your favourite Java IDE go in xwiki-enteprise/xwiki-enterprise-test/xwiki-enterprise-test-selenium and run maven with -Dmaven.surefire.debug. Maven will wait till you connect your IDE to the running JVM on port 5005. You can put a breakpoint in your IDE and debug the test.

Browser version

Currently we only run our functional tests on the Firefox browser.
The browser you use for functional tests needs to match the selenium version, otherwise unpredictable results may occur.

  • The current Selenium version we are using is 2.40.0
    • (valid for 26.August.2014. Actual version used can be verified here under the selenium.version property)
  • The Firefox version we use in you continuous integration agents is 27.0.1.
    • (valid for 26.August.2014. Ask on the list or on IRC for the actual used version since it's not publicly verifiable) 

If you wish to run tests with the exact configuration as XWiki's Continous Integration server uses, you need to install and use locally the same Firefox version. To do so, you have to:

  1. Download the corresponding Firefox release
  2. Unzip it locally
  3. Use the webdriver.firefox.bin java system property to specify the location of your firefox version
    1. Depending on how you are starting the functional tests, you`d have to either add the system property in your maven build (surefire plugin configuration) or in your IDE (run configuration)
    2. Read Selenium's FirefoxDriver documentation for more information and options

XHTML, CSS & WCAG Validations

Performance Testing

  • These are memory leakage tests, load tests and speed of execution tests.
  • They are performed manually and in an ad hoc fashion for now.

See the Profiling topic for details on how to use a profiler for detecting performance issues.

Manual testing

Here's the spirit we'd like to have from the XWiki Manual Testing Team: The Black Team.

Besides automated testing, ensuring software quality requires that features be tested by actual users (see Manual testing). In order to manage manual testing, a test plan is required. 

Test Coverage

We now have a SonarQube instance showing Test coverage for both unit tests and integration tests. However it doesn't aggregate coverage data across top level modules (commons, rendering, platform, enterprise, etc).

We support both Jacoco and Clover to generate test coverage reports.

To generate test coverage reports make sure you can build your module and then pick one of the following strategies below depending on what you wish to generate.

After running Clover you'll have some instrumented JARs in your local repository so you should be careful not to use those later on for deploying for example. And if you copy them in an existing xwiki setup in WEB-INF/lib you'll get an error at runtime saying that the Clover JAR is missing...

Strategy 1

Using Jacoco

  • Go in the first top level module (e.g. xwiki-commons) and run: mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=false -Plegacy,integration-tests
  • Go in all the other top level modules you wish to add and run: mvn clean jacoco:prepare-agent install -Djacoco.destFile=/tmp/jacoco.exec -Djacoco.append=true -Plegacy,integration-tests
  • Then whenever you wish to generate the full test report, run:
    TODO
    Note: It doesn't seem to be possible to generate a Jacoco report from an existing jacoco.exec file using the Maven plugin

Using Clover

Go to the top level module containing children modules for which to generate a Clover report. Note that you won't be able to aggregate Clover data across different Maven runs with this strategy so you really need a single parent module.

  1. Run the following command:
    mvn clean clover2:setup install clover2:aggregate clover2:clover -Pclover,integration-tests,dev,jetty,hsqldb
    You might need to run the "install" goal instead of the "test" one if your local Maven repository doesn't already contain some test jars (apparently and for some reason Maven won't download them from the remote repository under some conditions).

Strategy 2

Using Jacoco

TODO

Using Clover

Use a single Clover database to which you add coverage information as you build modules one after another. This strategy is especially useful when you wish to manually run some modules and ensure that coverage data aggregate in a single place so that when you generate the report you have the result of all your runs.

  1. Instrument the source code with Clover for all modules that you want to include in the report, using:
    mvn clover2:setup install -Pclover,integration-tests,dev,jetty,hsqldb -Dmaven.clover.cloverDatabase=/path/to/user/home/.xwiki/clover/clover.db

    When tests are executed they'll generate coverage data in the specified Clover database. Since there's a single Clover there's no need to merge Clover databases as in strategy 1 above.

  2. To generate the Clover report, execute the following from any module:
    mvn clover2:clover -N -Dmaven.clover.cloverDatabase=/path/to/user/home/.xwiki/clover/clover.db
  3. Remember to clean your clover database when you're done.
If you don't wish failing tests to stop the generation of the coverage report, you should pass -Dmaven.test.failure.ignore=true on the command line.

Here are typical steps you'd follow to generate full TPC for XWiki:

  • Clean your local repo and remove any previous clover DBs:
    rm -R ~/.m2/repository/org/xwiki
    rm -R ~/.m2/repository/com/xpn
    rm -R ~/.xwiki/clover
  • Generate coverage data for XWiki Commons:
    cd xwiki-commons
    mvn clean -Pclover,integration-tests -o
    mvn clover2:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db  -Dmaven.test.failure.ignore=true -o
  • Generate Clover report just for Commons:
    mvn clover2:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -o
  • Generate coverage data for XWiki Rendering:
    cd xwiki-rendering
    mvn clean -Pclover,integration-tests -o
    mvn clover2:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db  -Dmaven.test.failure.ignore=true -o
  • Generate Clover report for Commons and Rendering:
    mvn clover2:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -o
  • Generate coverage data for XWiki Platform:
    cd xwiki-platform
    mvn clean -Pclover,integration-tests -o
    mvn clover2:setup install -Pclover,integration-tests -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db  -Dmaven.test.failure.ignore=true -o
  • Generate Clover report for Commons, Rendering and Platform:
    mvn clover2:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -o
  • Generate coverage data for XWiki Enterprise:
    cd xwiki-enterprise
    mvn clean -Pclover,integration-tests,jetty,hsqldb -o
    mvn clover2:setup install -Pclover,integration-tests,jetty,hsqldb -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db  -Dmaven.test.failure.ignore=true -o
  • Generate Clover report for Commons, Rendering, Platform and Enterprise:
    mvn clover2:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -o
  • Generate coverage data for XWiki Enterprise Manager:
    cd xwiki-manager
    mvn clean -Pclover,integration-tests,jetty,hsqldb -o
    mvn clover2:setup install -Pclover,integration-tests,jetty,hsqldb -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db  -Dmaven.test.failure.ignore=true -o
  • Generate full Clover report:
    mvn clover2:clover -N -Dmaven.clover.cloverDatabase=/Users/vmassol/.xwiki/clover/clover.db -o

Example Reports

Tags:
Created by VincentMassol on 2007/03/07 13:33
    

Get Connected