Manual Testing Strategy (in progress)

Version 35.1 by Ilie Andriuta on 2020/11/09 10:29

Manual Testing Strategy

The Quality Assurance process of XWiki is ensured by both manual testing and automated testing. Manual testing role is to help this process by performing tests that cannot be run automatically yet. There is an ongoing strategy to improve the overall testing process by migrating manual tests to automated ones for XWiki Platform (where possible).

Support Strategies

XWiki and recommended extensions should be tested on all supported browsers, databases and servlet containers and the result of each test should be recorded accordingly on test.xwiki.org.

When the testing environment is configured, should be taken into consideration the following support strategies:

- Browser Support Strategy
- Database Support Strategy
- Servlet Container Support Strategy
- Java Support Strategy
- LibreOffice Support Strategy

A good practice is testing with a different combination of supported browsers, databases and servlet containers in order to cover as many cases as possible.

Releases and timeboxing

XWiki versions are released according to the established release strategy.

Since testing objectives can vary depending on the type of the release tested and since is not possible to run all manual tests on every XWiki or recommended extension version (for time reasons), the testing sessions should be performed in timeboxes.

Release Candidate Releases

Usually there is one RC release before the final release. Testing on RC releases should be performed within 1 week, the main objective being to find most issues before the final release.

Stable (Major and Minor) Releases

The stable releases can introduce new features and non backward compatible changes from the user's perspective. The testing sessions for these releases should have a 2 weeks timebox.

LTS Releases

According to release strategy on every end of cycle, a Long Term Support version is released.

From testing point of view this is a special stable release and should be tested more in-depth. The timebox indicated for this releases is 3 weeks.

Bugfix releases

These releases provide only bug fixes (which may contain also security fixes). They should be tested within 1 week, in order to test the fixes and to discover eventual regressions which might have been introduced along.

NOTE:

Some general rules:

- The LTS version has higher testing priority than other versions;

- If a stable version is under testing and a bugfix version is released (mainly the case of LTS) the tests are continued on the bugfix version.

Process

  • When an XWiki version is released, its new features should be tested (if there are any) and manual tests created/updated accordingly on Test Wiki if the case.

For every feature a test scenario should be presented and asked the team if there are automated tests already created for the respective features to avoid duplicate testing.

  • JIRA tickets marked as fixed in the Release Notes are tested and results are recorded in the Test Report
    • If the ticket is fixed, then the resolution will be 'Pass';
    • If the ticket is not fixed, the resolution will be 'Failed' and the reason for which failed should be explained in footnotes (which will eventually also contain the (new) JIRA ticket created for it);
  • The Release Notes should be updated on the corresponding section (both for RCs/bugfixes and stable releases) with the testing environment used for the respective release and the link to the tests run on it;
  • In order to identify issues early, exploratory tests should be performed, especially for the components on which modifications are known to be made made: fixed bugs, new features added, UI changes;
  • Extensive regression tests should be also run following the established priority criteria.
    • On regression, an important part is represented by the upgrade tests (which should be performed from older instances of the last 2-3 cycles - preferably also from last XWiki LTS version).

Test planning and priorities

In order to systematize the process of manual testing and to make sure the important features are tested with increased priority, the tests are categorized in a list of testing priorities both for XWiki and for recommended extensions. These priorities were established (also) based on the feedback received from teams/community.

Following this, when new tests are created, they need to be included in the right corresponding category and if the case, should be gathered feedback from the team/community regarding the placement of a new test in a category.

NOTES:

  • In a test session, the tests should be run in order, from the highest priority to the lowest one: P1 > P2 > P3, both for XWiki and for recommended extensions.
  • Since is not possible to run all manual tests on every XWiki or recommended extension version (given the limited time), the tests which will be run should be chosen from each category in a reasonable manner in order to include some tests from each one (P1, P2 and P3), distributed in the allowed timebox.
  • There are also a set of tests which are indicated to be run at every release, see the proposal

The priorities to run the tests are the following:

For XWiki:

  • P1: Priority 1 (high priority)
  • P2: Priority 2 (medium priority)
  • P3: Priority 3 (low priority)

P1.Product:

P2.Product:

P3.Product:

For recommended extensions:

P1.Extensions:

P2.Extensions:

P3.Extensions:

Manual tests migration strategy

In order to reduce the number of manual tests, a strategy has been proposed to identify manual tests run for XWiki Platform for which there are already automated tests or to create tasks for automation of tests that don't exist yet.

Strategy description:

  • One manual test per week (number can be adjusted depending on the complexity of tests) is analyzed and discussed on #xwiki:matrix.xwiki.com. With the help of Developers team will be searched whether an automated test exists and has or not the same reproduction steps as the manual test;
  • If an automated test doesn’t exist or the scenario is different, will be decided if a JIRA ticket should be raised for it (to be automated) or should be dropped and marked as a deprecated manual test.

NOTES:

  • The JIRA ticket:
    • should be raised on XWIKI platform Project, as type 'Task', will have the component 'Development Issues Only' and the label set 'testneeded';
    • should contain the link to the existing manual test and details about browsers/ databases/ servlet containers needed to be run on (if the case);
    • should contain the link to the matrix discussions to be easier to understand what needs to be done or it's already done.
  • The manual test:
    • should be updated by editing it in Objects mode and adding an QA.AutomatedTestStatusClass object;
    • should then be updated accordingly with:
      - Status ('Automated'/ 'Not Automatable'/ 'Needs Automated Test')
      - JIRA link created for automating the test
      - List of automated GitHub test links for the automated tests (if they exist)
      - Notes (eventually here should be explained some details and provided the matrix links to discussions if possible)
    • should also be updated on the 'Automated' xproperty

XWiki versions

XWiki uses Hudson, a continuous build tool, to generate new releases out of code committed in the project's source repository on a regular basis. If a version needs to be tested prior to a release, the latest SNAPSHOT of that given version should be used. For example:

Right after a release, the latest released version of the product should be used. For example:

Creating JIRA issues

The XWiki project uses JIRA to report bugs and issues.. A good JIRA ticket holds all the information needed to reproduce the issue that was encountered or even to describe an improvement that should be made. This includes:

  • a full description of the environment used (OS, Browser, Database, Servlet Container, XWiki version)
  • a description of all the steps to perform in order to trigger the issue
  • if applicable - a screenshot of the interface showing the faulty behavior
  • if applicable - the file attachment needed to reproduce the issue
  • if applicable - the content or code snippet needed to reproduce the issue

Follow this template:

STEPS TO REPRODUCE:...

EXPECTED RESULTS:...

ACTUAL RESULTS:...

NOTE:

Please avoid the following when creating a new JIRA issue:

  • Do not create a new issue which duplicates an existing one
  • Do not create a new issue which isn't filled/written correctly

Using JIRA's built-in search or Google (with the site:jira.xwiki.org operator) are good ways to find existing issues and avoid duplicating them.

Tags:
   

Get Connected