WCAG Testing
Strategy
The XWiki WCAG strategy is twofolds:
- 15.2+ Using Axe Core to automatically detect accessibility violations against WCAG 2.1 (AA level). This should detect about half of the accessibility issues on tested web pages (see below to see what pages are tested).
- Manually doing WCAG tests in the same manner as we do manual UI tests. This is needed to fully pass the WCAG 2.1 AA specification.
In addition, developers can use WCAG validation tools in their browsers (such as Axe devTools, see at the end for other resources) to reproduce WCAG problems on their computers easily and make it simpler to visually see the problems.
See the XHTML & CSS Coding Style which lists some best practices to be applied by developers and that are not automatically tested.
Testing
We reuse the XWiki docker-based functional tests to perform WCAG validations. This is done automatically at the end of the constructor of each BasePage PO.
This does not cover all possible states of the interface. However it's enough to cover all the static interface (see the Future section below for improvement ideas).
The results of the validation on each page will not fail any test per-see, but it'll fail the test suite if there are WCAG violations. Reports can be found in target/wcag-reports/*.txt files.
There are two WCAG validation rule types:
- Failing rules: if a violation is found, the test suite fails (and thus the build). The violations are also output in a target/wcag-reports/wcagFails.txt file.
- Warning rules: if a violation is found, the test-suite does not fail but a warning is displayed. The violations are also output in a target/wcag-reports/wcagWarnings.txt file.
The strategy is to move all the rules to be failing rules. The idea is to do this once we fix all the violations for a given rule so that no more violations can be added in the future without breaking the build.
15.8+ In addition to violations, information about Incomplete tests (in need of additionnal manual testing to validate) are stored in the target/wcag-reports/wcagIncompletes.txt file. For every test suite in the build, aggregated statistics about the test results can now be found in target/wcag-reports/wcagOverview.txt.
Reading though the reports
Testing Sessions
Since WCAG tests are not fully automated (and not full automatable), we have manual testing sessions that we do from time to time:
- Session for XE 2.1
- Testing session by NGI Accessibility ()
- Status for WCAG 2.1 AA in 2023
History
- 15.5+ Improved log traces of WCAG validation
- 15.2+ XWiki uses Axe Core to validate accessibility
- <15.2 XWiki was using the Dutch Web Guidelines to validate accessibility
Dutch Web Guidelines
Test Exceptions
Hidden pages
Hidden pages are not meant to be viewed by the user, so in order to simplify the management of WCAG validation, we decided to skip them for now.
See the full discussion related to this decision here.
Some hidden pages containing specific objects are still viewable by the users: we use a whitelist to enforce the check on them.
The full list of exceptions is then available in the dedicated pom.xml.
Other Exceptions
Here are some exceptions that make the WCAG validators fail but that we consider valid (i.e. we consider the validators are overzealous in these cases).
- We are using the H1 element several times on pages, once for the page title and once for the header level 1 from the page's content. We consider that this doesn't violate the rule "Do not skip any heading elements in the document hierarchy." since we're not skipping any level. Fails with Dutch Guidelines validation tool
- "Use (at least) the meta element to specify the character set and place this element as high as possible in the head section of the markup.". The Content-Type META element is th first one in the HEAD element and cannot be placed higher. Fails with Dutch Guidelines validation tool
- The ColorThemes application is using inline styles to display a preview of the colors and options of themes, this is something we cannot easily avoid.
- The XWiki syntax allows using elements which are not allowed by accessibility guidelines. Sandbox.WebHome contains demonstrations which does not satisfy the following guidelines:
- "Use the headers and id attributes to associate table labels (th cells) with individual cells in complex tables"
- "Use the scope attribute to associate table labels (th cells) with columns or rows. "
- "The use of sub and sup tags is not allowed."
- The XWiki syntax allows using elements which are not allowed by accessibility guidelines. XWiki.XWikiSyntax contains demonstrations which does not satisfy the following guidelines:
- "Use the p (paragraph) element to indicate paragraphs. Do not use the br (linebreak) element to separate paragraphs."
- "Use ol (ordered list) and ul (unordered list) elements to indicate lists."
- "Use the headers and id attributes to associate table labels (th cells) with individual cells in complex tables"
- "Use the scope attribute to associate table labels (th cells) with columns or rows. "
- "The use of sub and sup tags is not allowed."
- "Links to e-mail addresses: the e-mail address to which the message is addressed must be visible in the link text.
- "CSS should be placed in linked files and not mixed with the HTML source code."
- The Panels.PanelWizard and XWiki.Treeview pages are relying on JavaScript libraries that needs some HTML elements to contain inline styles.{{/version}}
WCAG Resources
WCAG validators
The following WCAG testing tools are listed for reference:
Online
- TAW3 (Tutorial for the Web start version).
- Total Validator
- Wave (ability to paste HTML content or upload a local file)
Browser based
- Axe devTools
- Firefox Web Developer Tool tutorial for Accessibility (especially precise for contrast issues)
- WAVE extension
- SiteImprove Accessibility Checker
- AInspector
- IBM Equal Access Accessibility Checker (2 sets of rules checkable, that can be changed in parameters)
Manual testing checklist
