Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 = Manual Testing Strategy =
6
7 The Quality Assurance process of XWiki is ensured by both manual testing and automated testing. Manual testing role is to help this process by performing tests that cannot be run automatically yet. There is an [[ongoing strategy>>https://forum.xwiki.org/t/manual-tests-vs-automated-tests-analysis/7535]] to improve the overall testing process by migrating manual tests to automated ones for XWiki Platform (where possible).
8
9 == Support Strategies ==
10
11 XWiki and recommended extensions should be tested on all supported browsers, databases and servlet containers and the result of each test should be recorded accordingly on [[test.xwiki.org>>test:Main.WebHome]].
12
13 When the testing environment is configured, should be taken into consideration the following support strategies:
14
15 - [[Browser Support Strategy>>BrowserSupportStrategy]]
16 - [[Database Support Strategy>>DatabaseSupportStrategy]]
17 - [[Servlet Container Support Strategy>>ServletContainerSupportStrategy]]
18 - [[Java Support Strategy>>Community.SupportStrategy.JavaSupportStrategy.WebHome]]
19 - [[LibreOffice Support Strategy>>Community.SupportStrategy.LibreOfficeSupportStrategy.WebHome]]
20
21 A good practice is testing with a different combination of supported browsers, databases and servlet containers in order to cover as many cases as possible.
22
23 == Releases and time-boxing ==
24
25 XWiki versions are released according to the established [[release strategy>>VersioningAndReleasePractices#HReleaseCyclesandReleaseStrategy]].
26
27 Since testing objectives can vary depending on the [[type of the release>>VersioningAndReleasePractices#HVersionTypes]] tested and since is not possible to run all manual tests on every XWiki or recommended extension version (for time reasons), the testing sessions should be performed in time-boxes.
28
29
30 Here are the release types and the indicated time-boxes for them:
31
32
33 ==== Release Candidate Releases ====
34
35 Usually there is one RC release before the final release. Testing on RC releases should be performed within **1 week/person**, the main objective being to find most issues before the final release.
36
37
38 ==== Stable (Major and Minor) Releases ====
39
40 The stable releases can introduce new features and non backward compatible changes from the user's perspective. The testing sessions for these releases should have a **2 weeks/person** time-box.
41
42
43 ==== LTS Releases ====
44
45 According to [[release strategy>>VersioningAndReleasePractices#HReleaseCyclesandReleaseStrategy]] on every end of cycle, a Long Term Support version is released.
46
47 From testing point of view this is a special stable release and should be tested more in-depth. The time-box indicated for this releases is **3 weeks/person**.
48
49
50 ==== Bugfix releases ====
51
52 These releases provide only bug fixes (which may contain also security fixes). They should be tested within **1 week/person**, in order to test the fixes and to discover eventual regressions which might have been introduced along.
53
54
55 **NOTE:**
56
57 **Some general rules:**
58
59 - The **LTS version has higher testing priority** than other versions;
60
61 - If a stable version is under testing and a bugfix version is released (mainly the case of LTS) the tests are continued on the bugfix version.
62
63
64 == Process ==
65
66 * When an XWiki version is [[released>>xwiki:Download.WebHome]], its new features should be tested and manual tests created/updated accordingly on [[Test Wiki>>test:QA.WebHome]] if the case.
67
68 * A [[test report>>xwiki:TestReports.WebHome]] is created, which should also contain information about the [[browsers>>BrowserSupportStrategy]], [[databases>>DatabaseSupportStrategy]] and [[servlet containers>>ServletContainerSupportStrategy]] used for testing the release;
69
70 * JIRA tickets marked as fixed in the [[Release Notes>>xwiki:ReleaseNotes.Data.WebHome]] are tested and results are recorded in the [[Test Report>>xwiki:TestReports.WebHome]]
71 ** If the ticket is fixed, then the resolution will be **'Pass'**;
72 ** If the ticket is not fixed, the resolution will be **'Failed'** and the reason for which failed should be explained in footnotes (which will eventually also contain the (new) [[JIRA ticket>>https://jira.xwiki.org/issues/]] created for it);
73
74 * The [[Release Notes>>xwiki:ReleaseNotes.Data.WebHome]] should be updated on the corresponding section (both for RCs/bugfixes and stable releases) with the testing environment used for the respective release and the link to the tests run on it;
75
76 * In order to identify issues early, **exploratory tests** should be performed, especially for the components on which modifications are known to be made made: fixed bugs, new features added, UI changes;
77
78 * **Extensive regression tests** should be also run following the [[established priority criteria>>ManualTestStrategy#HTestingpriorities]].
79 ** On regression, an important part is represented by the upgrade tests (which should be performed from older instances of the last 2-3 cycles - preferably also from last XWiki LTS version).
80
81 == Test planning and priorities ==
82
83 In order to systematize the process of manual testing and to make sure the important features are tested with increased priority, the tests are categorized in a list of testing priorities both for XWiki and for recommended extensions. These priorities were established (also) based on the feedback received from teams/community.
84
85 Following this, when new tests are created, they need to be included in the right corresponding category and if the case, should be gathered feedback from the team/community regarding the placement of a new test in a category.
86
87 **NOTES:**
88
89 * In a test session, the tests should be run in order, from the highest priority to the lowest one: P1 > P2 > P3, both for XWiki and for recommended extensions.
90
91 * Since is not possible to run all manual tests on every XWiki or recommended extension version (given the limited time), the tests which will be run should be chosen from each category in a reasonable manner in order to include some tests from each one (P1, P2 and P3), distributed in the allowed timebox.
92
93 * **There are also a set of tests which are indicated to be run at every release, see [[the proposal>>https://forum.xwiki.org/t/testing-priorities-proposal/7553]]**
94
95 The priorities to run the tests are the following:
96
97 === **For XWiki:** ===
98
99 * **P1**: Priority 1 (high priority)
100 * **P2**: Priority 2 (medium priority)
101 * **P3**: Priority 3 (low priority)
102
103 ==== **P1.Product:** ====
104
105 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Administration+Tests>>test:Administration Tests.WebHome]]
106 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Nested+Spaces/>>test:Nested Spaces.WebHome]]
107 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Security+Tests/>>test:Security Tests.WebHome]]
108 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Search+Tests/>>test:Search Tests.WebHome]]
109 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Upgrade+tests/>>test:Upgrade tests.WebHome]]
110 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Syntax/>>test:Syntax.WebHome]]
111 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/AppWithinMinutes+Tests/>>test:AppWithinMinutes Tests.WebHome]]
112 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/CKEditor/>>test:CKEditor.WebHome]]
113 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Wiki+Tests/>>test:Wiki Tests.WebHome]]
114 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Web+Tests/>>test:Web Tests.WebHome]]
115 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/User+Tests/>>test:User Tests.WebHome]]
116 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Performances/>>test:Performances.WebHome]]
117 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Notifications Tests/>>test:Notification Tests.WebHome]]
118 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/User Mentions Tests/ [Since XWiki 12.5 RC1]>>test:User Mentions Tests.WebHome]]
119 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Page Likes Tests/ [Since XWiki 12.7 RC1]>>test:Page Likes Tests.WebHome]]
120
121 ==== **P2.Product:** ====
122
123 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Panels+Tests/>>test:Panels Tests.WebHome]]
124 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Dashboard+Tests/>>test:Dashboard Tests.WebHome]]
125 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Color+Themes+Tests/>>test:Color Themes Tests.WebHome]]
126 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Export/>>test:Export.WebHome]]
127 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Macros/>>test:Macros.WebHome]]
128 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Index+Tests/>>test:Index Tests.WebHome]]
129 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Skin+Tests/>>test:Skin Tests.WebHome]]
130
131 ==== **P3.Product:** ====
132
133 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Activity+Stream+Tests/>>test:Activity Stream Tests.WebHome]]
134 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Annotations+Tests/>>test:Annotations Tests.WebHome]]
135 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Other+Tests/>>test:Other Tests.WebHome]]
136 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Tag+Tests/>>test:Tag Tests.WebHome]]
137 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Message+Stream+Tests/>>test:Message Stream Tests.WebHome]]
138
139 === **For recommended extensions:** ===
140
141
142 ==== **P1.Extensions:** ====
143
144 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Admin+Tools+Application/>>test:Admin Tools Application.WebHome]]
145 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Blog+Tests/>>test:Blog Tests.WebHome]]
146 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/FAQ+Application+Tests/>>test:FAQ Application Tests.WebHome]]
147 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Menu+Application/>>test:Menu Application.WebHome]]
148 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/JW%20Player%20Tests/>>test:JW Player Tests.WebHome]]
149
150 ==== **P2.Extensions:** ====
151
152 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Publication+Workflow+Tests/>>test:Publication Workflow Tests.WebHome]]
153 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Excel%20Export%20Tests/>>test:Excel Export Tests.WebHome]]
154 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Ratings%20Application%20Tests/>>test:Ratings Application Tests.WebHome]]
155 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Full%20Calendar%20Tests/>>test:Full Calendar Tests.WebHome]]
156
157 ==== **P3.Extensions:** ====
158
159 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Notes+Application+Tests/>>test:Notes Application Tests.WebHome]]
160 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Piwik%20Application%20Tests/>>test:Piwik Application Tests.WebHome]]
161 * [[http:~~/~~/test.xwiki.org/xwiki/bin/view/Map%20Macro%20Tests/>>test:Map Macro Tests.WebHome]]
162
163 == Manual tests migration strategy ==
164
165 In order to reduce the number of manual tests, a [[strategy>>https://forum.xwiki.org/t/manual-tests-vs-automated-tests-analysis/7535]] has been proposed to identify manual tests run for XWiki Platform for which there are already automated tests or to create tasks for automation of tests that don't exist yet.
166
167 **Strategy description:**
168
169 * One manual test per week (number can be adjusted depending on the complexity of tests) is analyzed and discussed on #xwiki:matrix.xwiki.com. With the help of Developers team will be searched whether an automated test exists and has or not the same reproduction steps as the manual test;
170 * If an automated test doesn’t exist or the scenario is different, will be decided if a JIRA ticket should be raised for it (to be automated) or should be dropped and marked as a deprecated manual test.
171
172 **NOTES:**
173
174 * **The JIRA ticket:**
175 ** should be raised on XWIKI platform Project, as type //'Task'//, will have the component //'Development Issues Only'// and the label set //'testneeded'//;
176 ** should contain the link to the existing manual test and details about browsers/ databases/ servlet containers needed to be run on (if the case);
177 ** should contain the link to the matrix discussions to be easier to understand what needs to be done or it's already done.
178
179 * **The manual test:**
180 ** should be updated by editing it in Objects mode and adding an **QA.AutomatedTestStatusClass** object;
181 ** should then be updated accordingly with:
182 - __Status__ ('Automated'/ 'Not Automatable'/ 'Needs Automated Test')
183 - __JIRA link__ created for automating the test
184 - __List of automated GitHub test links__ for the automated tests (if they exist)
185 - __Notes__ (eventually here should be explained some details and provided the matrix links to discussions if possible)
186 ** should also be updated on the //'Automated'// xproperty
187
188 == XWiki versions ==
189
190 XWiki uses Hudson, a continuous build tool, to generate new releases out of code committed in the project's source repository on a regular basis. If a version needs to be tested prior to a release, the latest SNAPSHOT of that given version should be used. For example:
191
192 * [[Jetty distribution>>http://maven.xwiki.org/snapshots/org/xwiki/platform/xwiki-platform-distribution-jetty-hsqldb/]]
193 * [[Jetty distribution with Standard Flavor preinstalled>>http://maven.xwiki.org/snapshots/org/xwiki/platform/xwiki-platform-distribution-flavor-jetty-hsqldb/]]
194 * [[WAR distribution>>http://maven.xwiki.org/snapshots/org/xwiki/platform/xwiki-platform-distribution-war/]]
195
196 Right after a release, the latest released version of the product should be used. For example:
197
198 * [[Jetty distribution>>http://nexus.xwiki.org/nexus/content/groups/public/org/xwiki/platform/xwiki-platform-distribution-jetty-hsqldb/]]
199 * [[Jetty distribution with Standard Flavor preinstalled>>http://nexus.xwiki.org/nexus/content/groups/public/org/xwiki/platform/xwiki-platform-distribution-flavor-jetty-hsqldb/]]
200 * [[WAR distribution>>http://nexus.xwiki.org/nexus/content/groups/public/org/xwiki/platform/xwiki-platform-distribution-war/]]
201
202 == Creating JIRA issues ==
203
204 [[The XWiki project uses JIRA to report bugs and issues.>>https://jira.xwiki.org/]]. A good JIRA ticket holds all the information needed to reproduce the issue that was encountered or even to describe an improvement that should be made. This includes:
205
206 * a full description of the environment used (OS, Browser, Database, Servlet Container, XWiki version)
207 * a description of all the steps to perform in order to trigger the issue
208 * if applicable - a screenshot of the interface showing the faulty behavior
209 * if applicable - the file attachment needed to reproduce the issue
210 * if applicable - the content or code snippet needed to reproduce the issue
211
212 **Follow this template:**
213
214 STEPS TO REPRODUCE:...
215
216 EXPECTED RESULTS:...
217
218 ACTUAL RESULTS:...
219
220 **NOTE:**
221
222 Please avoid the following when creating a new JIRA issue:
223
224 * Do not create a new issue which duplicates an existing one
225 * Do not create a new issue which isn't filled/written correctly
226
227 Using JIRA's built-in search or Google (with the ##site:jira.xwiki.org## operator) are good ways to find existing issues and avoid duplicating them.

Get Connected