The scripts below work on the reports archived by the CI. They use GNU tools (Linux).

Download the Reports

• Open the latest build of the XWiki Environment Tests job that ran the docker-latest type.
• Download all its artifacts, at https://ci.xwiki.org/job/XWiki%20Environment%20Tests/job/xwiki-platform/job/master/<build number>/artifact/.
• Extract them in a folder.

Statistics for the Whole Build

This script merges the wcagOverview-*.txt files and computes the percentage of each result over the whole build. It takes the artifact folder as its only parameter.

cd $1
find -type f -name 'wcagOverview*.txt' -printf "__________________\n%h\n__________________\n" -exec cat {} \;> allOverviews.txt;
failamount=$(cat allOverviews.txt |grep -Po '(?<=\[)(\d+)(?=\]\(.*\) were fails)' | paste -sd+ - | bc)
incompleteamount=$(cat allOverviews.txt |grep -Po '(?<=\[)(\d+)(?=\]\(.*\) were incomplete)' | paste -sd+ - | bc)
passamount=$(cat allOverviews.txt |grep -Po '(?<=\[)(\d+)(?=\]\(.*\) were passed)' | paste -sd+ - | bc)
totalamount=$((failamount + incompleteamount + passamount))

failpercent=$(printf %.2f\\n "$((10000 * $failamount/$totalamount  ))e-2")
incompletepercent=$(printf %.2f\\n "$((10000 * $incompleteamount/$totalamount  ))e-2")
passpercent=$(printf %.2f\\n "$((10000 * $passamount/$totalamount  ))e-2")

echo "Total tests in the build: $totalamount"
echo "FAILED $failamount($failpercent%)"
echo "INCOMPLETE $incompleteamount($incompletepercent%)"
echo "PASSED $passamount($passpercent%)"

Main Violations

This script counts the violations of each rule, in a resultDescriptions.txt file. Compare it with the previous results to spot improvements and regressions.

cd $1
(echo "$(grep --exclude="result*" --no-filename -R "Description: " | sort |uniq -c)"
echo "Total issues:" "$( grep --exclude="result*" -R "Description: " | wc -l)" )> resultDescriptions.txt;

The number of violations of a rule gives a hint about its cause:

• Less than 10: probably a single issue, in a specific interaction, and all the reports come from the same module.
• Between 10 and 100: a few smaller issues, or one issue in a part of the interface found on several pages, such as tables.
• More than 100: probably one important violation found in most modules. Fix it first: it can hide other violations of the same rule, and it's usually a real roadblock for assistive technology users.

Find a Violation

From the artifact folder, search for the rule description, for example:

grep -r "Description: Ensures every form element has a label"

When there are many results, flatten the folder first:

cd $1
find . -name 'wcag*.txt' -print0 | while read -d '' file
do
  target="${file#./}" # remove './' at the beginning
  target="${target//\//_}" # replace '/' by '_'
  mv "$file" "$target"
done

Then locate the violation with:

1. The URL of the page, if the page isn't generated by the test: open it in your browser.
2. Otherwise, the name of the test: search for it in the xwiki-platform sources.
3. The selector of the element: search for it in the xwiki-platform sources or in the page.

Once you understand the violation, report it.