XS Quality Report for 2020 - 2021

The quality evolution of the XWiki product is very important, therefore some of the most important aspects of XWiki 13.x Cycle (2021) compared to 12.x Cycle (2020) were emphasized in the following report.

In order to track and assess properly the evolution of XS quality, were chosen the following metrics:

- Evolution of Bugs - Created vs Resolved
- Evolution of Blockers - Created vs Resolved
- Evolution of Security issues - Created vs Resolved
- Performance stats comparing latest finished cycle with the previous cycle (latest report is available here)
- Evolution of Global TPC (Test Percentage Coverage) and the number of tests (analyzed from http://maven.xwiki.org/site/clover/)

For a general view of some of the above mentioned metrics, was created the XS System Quality Dashboard which provides useful information, along with XWiki Standard Bug Statistics Dashboard and with Recommended Extensions Bug Statistics Dashboard for Recommended Extensions.

In order to have the closest image about reality when analyzing the evolution of the metrics, it must be taken into consideration that during 13.x cycle, an important part of the work was done by XWiki core committers on extensions such as Change Request, Replication, Realtime and Numbered Content, which are very important because were developed many new features which bring increased value.

Evolution of Bugs

Regarding bug issues, can be seen in the following chart that the gap between Created vs Resolved issues widened during last two years, the reason being that an important part of the work was done on previously mentioned extensions (which is not less important than work done on XS) and were done fewer BugFixingDays because of the very limited time left for them.

(NOTE: JQL filter used category = 10000 AND issuetype = Bug ORDER BY created DESC, chart available on XS System Quality Dashboard)

Evolution of Blockers

As it can be seen in the following chart, blockers have a very tight gap between Created vs Resolved, meaning that they are usually fixed at a short time after they've been reported.

(NOTE: JQL filter used category = 10000 AND type = Bug AND priority = Blocker AND affectedVersion !~ "*-rc-1" ORDER BY created DESC, chart available on XS System Quality Dashboard)

Evolution of Security issues

As the following chart indicates, the gap between created vs resolved security issues maintains a rather steady evolution over the last two years (app. 2-3 issues difference). As per XWiki Security Policy, the High Severity (Blocker) issues (2 reported in the analyzed period) are fixed promptly (before next release).

(NOTE: JQL filter used category = 10000 AND labels = security ORDER BY created DESC, chart available on XS System Quality Dashboard)

Evolution of Performance

The full report that was created after running performance tests on XWiki 13.10.2 vs 12.10.11 is available here.

Can be noticed that XWiki 13.10.2 remained similar in performance compared to 12.10.11 in some areas like servlet startup, first access of pages, SOLR reindex (both full and sync with empty index) whereas in some parts 13.10.2 lost some performance compared to 12.10.11: e.g. reload of pages (empty and Main page - 23%, respectively 36% slower), SOLR sync at idle (30% slower), search operations (33% slower) and rendering of pages with many macros, this being the result of a regression which was already fixed.

Evolution of Global Test Percentage Coverage and number of tests

The Global Test Percentage Coverage (TPC) is an important metric that determines how much of the entire code is successfully validated under a test procedure, therefore providing an image on how comprehensively the software is verified.

As of February 2020, the Global TPC (analyzed from http://maven.xwiki.org/site/clover/) was 70.2873% (source: http://maven.xwiki.org/site/clover/20200208/XWikiReport-20200123-0128-20200208-0128.html), while as of February 2021 (last available report) was 68.6465% (source: http://maven.xwiki.org/site/clover/20210208/XWikiReport-20210128-0145-20210208-0128.html), registering a small decrease of app. 1.64%.

Regarding the number of tests evolution, as of February 2020 there were 11056 tests, while as of February 2021 the number increased with 267 tests, up to 11323 tests.

New features and improvements implemented in 13.x

Among the most important new features and improvements that were implemented in 13.x can be mentioned (a comprehensive list of changes is available here):

- Live Data Macro and improvements added to it (like in-line editing, look and feel)

- Extension Manager UI improvements (more advanced search features of the Extension Manager were hidden under a 'More' button by default and more explanation was added to the show information)

- Global Notifications Filters

- Distinguish between system filters and custom filters in notification settings

- Added a Translate Page Button

- Administrator can access user notification settings from its profile

Contributed Recommended Extensions

As it can be seen in the below chart, generally over the last 5 years the gap between bug created create vs resolved issues widened; however, in 2021 some of the issues were fixed, leading to a slightly reduction of the gap difference.

(NOTE: JQL filter used category = "XWiki Contributed Projects - Recommended" AND issuetype = Bug ORDER BY created DESC, chart available on Recommended Extensions Bug Statistics Dashboard)

Conclusion

Although an important part of the work during XWiki 13.x cycle was allocated for mentioned extensions by developing new features which in the end will bring increased value to the product (thus limiting the time to fix more reported issues), the 13.x cycle adds new implemented features and improvements which lead to a better usability and user experience.

However, on the other part, in certain cases some performance was lost compared to the previous cycle, but a part of these issues were already addressed in the recent releases.

On Contributed Extensions side, in 2021 were fixed some of the existing bugs (among which can be mentioned: Blog Application - 11 issues, OpenID Connect - 8 issues, Media Wiki Tools - 6 issues, JIRA Macro - 4 issues), which helped improving the quality of these extensions.

Recommendations

- Continue systematic work prioritization in order to address the most important issues first;

- Work on improvement of the performance in affected areas (like page reloading, SOLR sync, search operations);

- As much as possible, do more BugFixingDays for stabilizing the product.