Manual Testing Strategy

Manual testing involves having a tester play with the software through an user interface. Its aim is to emulate the end-user experience of using the software. Manual testing is therefore a key component of the Quality Assurance process.

Context

XWiki software's UI is mostly made of web interfaces. Some XWiki-developed applications (Word addon, XEclipse) do not rely on the browser as their main visualization tool but those applications have much fewer cross-platform display discrepancies issues than web interfaces.

Most XWiki developers are working on the Mac & Linux platforms, using Firefox or one of its variants as their main browser. This has 2 main consequences:

  • XWiki UI code is almost always optimized for Firefox first, before getting fixed for other browsers
  • IE testing is harder to do for developers since the IE browser doesn't work on their platforms of choice

Browsers

Making XWiki's UI work cross-browser is a challenging task. Getting it done right is critical to having XWiki widely used and adopted, especially in a corporate setting. Based on current browser usage statistics, Chrome, IE, Firefox and Edge are the most used desktop browsers around the world as of today. Within enterprises, IE browsers still have a lot of weight.

Since XWiki software is mostly tested in Firefox by developers, manual testing should focus on Internet Explorer (and eventually Edge). Because browsers like Chrome and Firefox have a much more rapid release cycle than the IE browsers (or Edge), the XWiki Community has agreed on the Supported Browsers strategy.

After each Release candidate, people who want to help with manually testing can follow the Manual Test Report Template. They should copy this template following the name convention scheme: ManualTestReport + <version of the release being tested>. For example, for XWiki 10.10 RC1, the page name should be ManualTestReportXWiki1010RC1.
The page should be copied in the same space and linked from the TestReport's WebHome.

Releases

XWiki releases new software very frequently. Testing objectives vary depending on the type of the current release.

Release Candidate Releases

Those releases are meant to make the software ready for production release and squash any remaining bugs. The aim of testing at this point is to ensure that the product delivers a consistent, bug-free user experience across the full line of supported browsers.

Major Releases

A major release has the same quality standards as a Release Candidate. Major releases are often followed by point releases that correct an additional batch of bugs. Testing practices for major releases are similar to those for Release Candidates.

Process

The testing process is the following:

  1. Make sure you're testing the right thing
  2. Make sure you're reporting issues at the right location

Find out which version to test

XWiki uses Hudson, a continuous build tool, to generate new releases out of code committed in the project's source repository on a regular basis. Prior to a release, the latest SNAPSHOT of a given product version should be used. For example:

Right after a release, the latest released version of the product should be used.For example:

Information about the state of the current release and upcoming new releases can be gained through the mailing lists.

Create JIRA issues

The XWiki project uses JIRA to report bugs and issues. The following are a few best practices when it comes to creating JIRA issues.

What not to do

Please avoid the following when creating a new JIRA issue:

  • Do not create a new issue which duplicates an existing one
  • Do not create a new issue which isn't filled/written correctly. For example, follow this template instead:

STEPS TO REPRODUCE:...

EXPECTED RESULTS:...

ACTUAL RESULTS:...

Using JIRA's built-in search or Google (with the site:jira.xwiki.org operator) are good ways to find existing issues and avoid duplicating them. Read on for a quick rundown of XWiki's JIRA tips.

What to do

A good JIRA bug report holds all the information needed to reproduce the issue that was encountered. This includes:

  • a full description of the environment used (OS, Browser, XWiki version)
  • a description of all the steps to perform in order to trigger the issue
  • a quick description of the intended behavior and how it was different from the encountered result
  • if applicable - a screenshot of the interface showing the faulty behavior
  • if applicable - the file attachment needed to reproduce the issue
  • if applicable - the content or code snippet needed to reproduce the issue

Besides these elements, XWiki's JIRA issues also have the following fields:

  • Project: XWiki's development work is divided into projects. On top of the core, each application and plugin has its own project. When filing a new issue, the tester should choose the relevant project. Although sometimes complex in the beginning, it comes with a little bit of practice. Here are a couple examples:
    • A bug met while adding a tag to a page should go under "XWiki Tag Application"
    • A bug met while using the WYSIWYG editor should go under "XWiki Core"
    • A bug met on the wiki homepage should go under "XWiki Platform"
  • Issue type: this one is pretty self-explanatory
  • Summary: a short self-explanatory description of the bug encountered
    • "Close button not displayed when adding a new user" is good
    • "I can't see the damn button on the close thing" isn't
  • Priority: can be left set to "Major" or "Minor" most of the time
  • Component: the part of the project affected by the issue. For instance, a Notifications bug would be filled under the "Notifications" component in the "XWiki Platform" project
  • Affects Version/s: the (first) version where this bug was discovered

The more fields filled, the more likely it is that an issue will be easy to understand and fix by a developer.

Tags:
Created by Guillaume Lerouge on 2009/08/13 14:13
   

Get Connected