Reading through WCAG reports

Last modified by Vincent Massol on 2023/10/04 16:58

Table of content:

Accessibility statistics in a module

XWiki 15.8+ In every module containing integration tests, an artifact named wcagOverview.txt is generated. It contains all the information necessary about the overall accessibility of the module.

Accessibility statistics in the build

XWiki 15.8+ 

It's possible to concatenate the wcagOverview.txt files mentionned above to get a quick overview over the whole build. For example, the script thereunder merges all the files, and retrieves all the stats in them to give percentage of test status through the whole build:

cd $1
find -type f -name 'wcagOverview.txt' -printf "__________________\n%h\n__________________\n" -exec cat {} \;> allOverviews.txt;
failamount=$(cat allOverviews.txt |grep -Po '(?<=\[)(\d+)(?=\]\(.*\) were fails)' | paste -sd+ - | bc)
incompleteamount=$(cat allOverviews.txt |grep -Po '(?<=\[)(\d+)(?=\]\(.*\) were incomplete)' | paste -sd+ - | bc)
passamount=$(cat allOverviews.txt |grep -Po '(?<=\[)(\d+)(?=\]\(.*\) were passed)' | paste -sd+ - | bc)
totalamount=$((failamount + incompleteamount + passamount))

failpercent=$(printf %.2f\\n "$((10000 * $failamount/$totalamount  ))e-2")
incompletepercent=$(printf %.2f\\n "$((10000 * $incompleteamount/$totalamount  ))e-2")
passpercent=$(printf %.2f\\n "$((10000 * $passamount/$totalamount  ))e-2")

echo "Total tests in the build: $totalamount"
echo "FAILED $failamount($failpercent%)"
echo "INCOMPLETE $incompleteamount($incompletepercent%)"
echo "PASSED $passamount($passpercent%)"

It takes the folder containing the artifacts as its first and only parameter.

Overview of the main accessibility violations

Automated testing produces report when warnings or errors are encountered. Here is an example of the process to report WCAG violations from the notes gathered in the artifact.

Finding accessibility issues

Then, for each line on the report generated above, it's possible to continue with this process:

  • Make sure you're in the artifact folder and not above
  • Check for the location of the occurence in the folder. E.g. grep -r "Description: Ensures every form element has a label"
  • (if there's a lot of different results, I'd recommend to flatten the repository before getting to search, using a script similar to the one hereafter.)
    cd $1
    find . -name 'wcag*.txt' -print0 | while read -d '' file
     target="${file#./}" # remove './' at the beginning
     target="${target//\//_}" # replace '/' by '_' and add '.txt'
     mv "$file" "$target" # do the moving

From now on, a lot is just knowledge of the violations:

  • If a rule has been reported less than 10 times, it's probably a single issue that's found in a specific interaction of the wiki. All the reports are probably from the same module.
  • If a rule has between 10 and 100 reports, it's probably either: a few smaller issues, or one issue that is found in a part of the interface that can be on multiple pages (e.g. tables).
  • If a rule has more than 100 reports, it's probably one important violation that is found in most modules. It can hide violations of the same issue (but different causes) easily, so it's recommended to deal with it as fast as possible. Usually it's also an important roadblock to accessibility by itself, at best a very recurrent lack, and at worst a terrible annoyance to assistive tech users.

Understanding an issue to report it properly

Once the issue has been identified, there's a few fields particularly important to find it's scope and hopefully observe it in your own distribution:

  1. The URL of the page the test is held on, can give an easy clue if the page isn't generated during the test. => Search in browser
  2. Elsewise, the name of the test that triggered the validation with the violation. => Search in source of xwiki-platform
  3. The selector of the element concerned by the violation => Search in xwiki-platform sources or browser page found in 1.

Once those three searches have been made, it's only a matter of a bit of reading to understand where/when the violation happens.

To create the Jira issue:

  • Check that it hasn't been already reported by searching on a few appropriate keywords.
  • Don't forget to add the `wcag` label to the issue, and fill up the number/name of the broken rule in WCAG 2.1 Rule ids
  • Add a screenshot of the element concerned by the issue if possible, with the page and the DOM tree. This makes it faster to understand the issue.

Get Connected