Wiki source code of Building XWiki from sources

Version 113.2 by Vincent Massol on 2019/04/03 14:38

Show last authors
1 {{box cssClass="floatinginfobox" title="**Contents**"}}
2 {{toc/}}
3 {{/box}}
4
5 {{error}}
6 * Use Java 8 (Java 9+ is not supported yet)
7 * Make sure you install a Maven version >= 3.5.0. Note that our [[CI>>http://ci.xwiki.org]] builds XWiki using Maven 3.6.0 and thus this is our recommended version.
8 * Windows users have several choices for getting past the [[long paths issue>>https://github.com/msysgit/git/pull/122]] (255 char limitation on Windows) and building XWiki:
9 ** Use either [[GitHub Desktop>>https://desktop.github.com/]] or [[Git for Windows>>https://git-for-windows.github.io/]] and set ##core.longpaths## to ##True##: {{code}}git config --global core.longpaths true{{/code}}.
10 ** Use [[Cygwin>>https://www.cygwin.com/]] which works out of the box.
11 * If you're using openJDK, you must use a [[version equal or newer than 8u191-b12>>https://stackoverflow.com/questions/53010200/maven-surefire-could-not-find-forkedbooter-class/53016532#53016532]].
12 {{/error}}
13
14 = Understanding the directory structure =
15
16 Before you start it's good to have some minimal understanding of [[how Maven works>>http://maven.apache.org/users/index.html]]. You probably won't need it if everything goes fine but you'll need that knowledge if something breaks! ;-)
17
18 The first thing to understand is the [[directory structure we use>>SourceRepository]].
19
20 = Checking out the sources =
21
22 Use your favorite SCM client to [[check out the Project>>SourceRepository]] you wish to build.
23
24 = Installing Maven =
25
26 * Install [[Maven 3.5.4 or greater>>http://maven.apache.org/]]. Create an ##M2_HOME## environment variable pointing to where you have installed Maven. Add ##M2_HOME/bin## to your PATH environment variable.(((
27 {{info}}
28 For more information on how to install Maven or how to use it check this [[excellent Maven book>>http://www.sonatype.com/books/mvnref-book/reference/]].
29 {{/info}}
30 )))
31 * Make sure you give Maven's JVM enough memory. XWiki's build uses Aspects and GWT compilers which need lots of memory. A good value is to give the JVM 1024MB. To do that set an environment variable named ##MAVEN_OPTS##. For example on Unix add the following to your shell startup script (depending on your shell you might need to prefix the variable with ##export##):(((
32 {{code language="none"}}
33 MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m"
34 {{/code}}
35 )))
36 * Create a ##~~/.m2/settings.xml## file with the XWiki custom remote repository defined as shown below (if you're using a Repository Manager such as Nexus or Artifactory, you should add the XWiki Maven Remote repositories in there instead). If you are on Windows, create the directory ##.m2## in your home directory, e.g. ##C:\Documents and Settings\Administrator\.m2##. If you cannot do this from Windows Explorer, open a command line and use ##md .m2##.(((
37 {{info}}
38 You might be wondering why we don't push all XWiki artifacts to Maven Central and thus remove the need to declare the XWiki Maven Repositories.
39
40 Actually we do this for XWiki Commons and XWiki Rendering. Since these 2 projects contain libraries that are independent of the concept of wikis, this makes it very easy for everyone to use them in their own projects.
41
42 We haven't done so for XWiki Platform and XWiki Enterprise this far because Maven Central imposes a requirement that we cannot fulfil at the moment: all artifacts deployed there must not have dependencies on other artifacts that are not present on Maven Central. And we do have a lot of such 3rd party dependencies. Some of them do not even exist in any Maven Repository and we put them in our [[XWiki Externals Repository>>http://maven.xwiki.org/externals/]].
43
44 Note that we recommend using a Maven Repository Manager to make it easier to maintain your Maven Repositories setup.
45 {{/info}}
46
47 {{code language="xml"}}
48 <settings>
49 <profiles>
50 <profile>
51 <id>xwiki</id>
52 <repositories>
53 <repository>
54 <id>xwiki-snapshots</id>
55 <name>XWiki Nexus Snapshot Repository Proxy</name>
56 <url>http://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
57 <releases>
58 <enabled>false</enabled>
59 </releases>
60 <snapshots>
61 <enabled>true</enabled>
62 </snapshots>
63 </repository>
64 <repository>
65 <id>xwiki-releases</id>
66 <name>XWiki Nexus Releases Repository Proxy</name>
67 <url>http://nexus.xwiki.org/nexus/content/groups/public</url>
68 <releases>
69 <enabled>true</enabled>
70 </releases>
71 <snapshots>
72 <enabled>false</enabled>
73 </snapshots>
74 </repository>
75 </repositories>
76 <pluginRepositories>
77 <pluginRepository>
78 <id>xwiki-snapshots</id>
79 <name>XWiki Nexus Plugin Snapshot Repository Proxy</name>
80 <url>http://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
81 <releases>
82 <enabled>false</enabled>
83 </releases>
84 <snapshots>
85 <enabled>true</enabled>
86 </snapshots>
87 </pluginRepository>
88 <pluginRepository>
89 <id>xwiki-releases</id>
90 <name>XWiki Nexus Plugin Releases Repository Proxy</name>
91 <url>http://nexus.xwiki.org/nexus/content/groups/public</url>
92 <releases>
93 <enabled>true</enabled>
94 </releases>
95 <snapshots>
96 <enabled>false</enabled>
97 </snapshots>
98 </pluginRepository>
99 </pluginRepositories>
100 </profile>
101 </profiles>
102 <activeProfiles>
103 <activeProfile>xwiki</activeProfile>
104 </activeProfiles>
105 </settings>
106 {{/code}}
107
108 {{info}}
109 Depending on your geographical location, you should use closer Maven repositories to speed up your build process. For more information refer to the [[Maven Mirror Settings Guide>>http://maven.apache.org/guides/mini/guide-mirror-settings.html]].
110 {{/info}}
111 )))
112
113 = Building with Maven =
114
115 Now that you have installed Maven and checked out the [[Projects>>SourceRepository#HTopLevelProjects]] you wanted to work on, you'll want to build them or parts of them. A project is made of modules. To build a module in Maven just issue the command:
116
117 {{code}}
118 mvn clean install
119 {{/code}}
120
121 Thanks to XWiki's [[Continuous Integration setup>>DevelopmentPractices#HContinuousIntegration]] you can check out and build any single module (and its children) you wish without having to rebuild the other modules from source. Maven will pick the latest version of your module's dependencies from [[XWiki's Maven remote repositories>>http://maven.xwiki.org/]]. Of course if you have uncommitted changes in a dependent module, you'll want to build that module before. The build result is placed in two places:
122
123 * Your [[local maven repository>>http://maven.apache.org/guides/introduction/introduction-to-repositories.html]], for making the module available to other projects or modules using maven (even other XWiki modules take the needed libraries from this repository, and not directly from a "neighbor" directory)
124 * In a subdirectory of that module, called ##target##.
125
126 {{info}}
127 Some of the modules built by XWiki are very large (in the 40MB+ size) and sometimes you don't absolutely need the latest version to get downloaded as it'll take too long. In that case here's a tip: run Maven in 'no-snapshot-update' mode with ##mvn install -nsu##.
128 {{/info}}
129
130 {{warning}}
131 You might encounter a ##java.lang.OutOfMemoryError## error during ##aspectj:compile##. You must increase the maximum memory used by maven, by setting ##MAVEN_OPTS=-Xmx1024m## (or a larger amount of memory if needed).
132 {{/warning}}
133
134 == Using Profiles ==
135
136 Project builds define several [[Maven Profiles>>http://maven.apache.org/guides/introduction/introduction-to-profiles.html]]. We've standardized their names across Projects.
137
138 Here are the most common ones:
139
140 * ##legacy##: Includes the [[legacy modules>>DevelopmentPractices#HDeprecationRules]] in the build.
141 * ##integration-tests##: Executes integration and functional tests in the build.
142 ** ##docker##: Executes integration tests that are Docker-based. You'll need to have Docker installed for this to work.
143 * ##jetty##, ##glassfish##: Run the build for the specified container (##jetty## is the default when not specified).
144 * ##hsqldb## (Hypersonic SQL), ##mysql## (MySQL), ##pgsql## (PostgreSQL), ##derby## (Derby): Run the build for the specified database (##hsqldb## is the default when not specified).(((
145 {{warning}}
146 Hypersonic and Derby are embedded database so you won't need anything setup to execute the build for them. However for MySQL and PostgreSQL you'll need to have the database installed, set up and running before you can run the build with their profiles.
147 {{/warning}}
148 )))
149 * ##firefox## (Firefox), ##iexplore## (Internet Explorer), ##chrome## (Google Chrome), ##otherbrowser## (need to define ##browserPath## property to use that one): Run the functional tests on the specified browser (##firefox## is the default when not specified).
150 * ##dev##: only build wysiwyg for English/Firefox. Very useful to speed up the build of XWiki Platform
151 * ##distribution##: {{info}}Since 9.5RC1{{/info}} Also build distribution packages (the WAR, the flavor, the jetty/hsqldb package, etc)
152 ** ##flavor-integration-tests##: {{info}}Since 9.5RC1{{/info}} Execute integration and functional tests related to the the default flavor located in ##xwiki-platform##.
153
154 More "exotic" ones:
155
156 * ##release##: Used by the Maven Release plugin when releasing XWiki. It generates Javadoc and Source artifacts. It also checks that the Java version used is JDK6, that the Javadoc exists and it signs artifacts using GPG (this is a good practice and a [[requirement for being able to upload some of our artifacts to the Maven Central Repository>>http://maven.apache.org/guides/mini/guide-central-repository-upload.html]]).
157 * ##snapshotModules##: {{info}}Since 6.2M1{{/info}} To be used on development/snapshot builds regularly ran on the CI (so never on a release!). This way, the resulting XWiki build includes the XWiki snapshots Extension Repository Source preconfigured and anyone downloading it for local testing will not have to enable it by hand. Distribution Wizard and Extension Manager will thus install the latest published snapshot extension builds and not fail completely saying that it can not find snapshot versions.
158 * ##m2e##: For [[m2eclipse>>http://www.eclipse.org/m2e/]] users. It sets the Eclipse output directory to ##target-eclipse## (instead of ##target##) to prevent race conditions between Maven within Eclipse and Maven on the command line.
159 * ##unix##, ##mac##, ##windows##: These profiles are automatically activated depending on the OS the build is running on. These profiles are useful for the Installers and for functional tests to decide how to start XWiki.
160 * ##macprofiler##, ##winprofiler##: Start [[XWiki for Profiling>>Profiling]] (they require that you set a ##profilePath## property in your ##settings.xml## or on the command line)
161 * ##gwt-test-manual##: allow running GWT unit tests manually
162 * ##debug##: remove JS minification.
163 * ##standalone##: Builds the Rendering Standalone artifact. Note that this profile is activated automatically when performing a release.
164 * ##quality##: Runs quality checks that take a long time to execute (e.g. checks Jacoco TPC thresholds)
165 * ##clover##: A profile to use when [[generating Test Coverage reports>>Testing#HTestCoverage]] with [[Clover>>http://www.atlassian.com/software/clover/overview]]. This profile skips execution of Checkstyle, Backward Compatibility and Enforcer plugin checks to speed up the coverage generation and because of potential conflicts between tools. Note that even though we've started using Jacoco for our ##quality## profile we still support Clover. The Clover reports are much nicer than the Jacoco ones and it's thus still useful to be able to generate them.
166
167 For example, if you use to include legacy modules and run all integration and functional tests and use Jetty/HSQLDB, you would use:
168
169 {{code}}
170 mvn clean install -Plegacy,integration-tests,hsqldb,jetty
171 {{/code}}
172
173 == Relationship between XWiki Enterprise and XWiki Platform ==
174
175 XWiki Platform is a generic platform and API for building flexible extensible wikis and web applications. XWiki Enterprise is an implementation of the platform geared toward making a wiki site for teams to share information and synchronize their efforts although it can be and has been used for a wide variety of different use cases like this website.
176
177 What does this mean to the developer?
178 If you wish to make changes in the core of XWiki, you are probably going to want to work on XWiki Platform. You can download and compile XWiki Enterprise and while compiling, you will notice Maven downloads a big 80MB+ file, this is the platform. Maven is smart enough to know that you don't have a copy of the platform and will download it from the XWiki Maven repository. Once it has been downloaded the first time, it is stored on your hard drive and you won't have to download it again //until// changes are made in the platform.
179
180 If you want to work on the core or if you have limited bandwidth, then you are well advised to get the platform sources. Then you can use the SCM's update command to get the latest copy without downloading the entire platform all over again. If you download the platform and compile it then you can compile XWiki Enterprise, Maven will use your newly compiled copy of the platform rather than downloading it, Maven is smart.
181
182 = Automatic Checks =
183
184 The XWiki build performs some automated checks:
185
186 * Style checks with Checkstyle. To skip: ##-Dxwiki.checkstyle.skip=true##. We also have custom Checkstyle checks:
187 ** Verify that ##components.txt## contains all Components
188 ** Verify that ##@Unstable## annotation don't stay too long (and that a ##@since## annotation is present too)
189 ** Verify that Script Services are not located in the ##internal## package
190 ** Verify that ##@since## javadoc tags have the correct format (and that they don't contain several versions per tag). Example of correct format: ##@since 10.8RC1##.
191 * Verify header licenses
192 * Backward compatibility checks with [[revapi>>http://revapi.org/]]. To skip: ##-Dxwiki.revapi.skip=true##.
193 * Enforcer checks (To skip: ##-Dxwiki.enforcer.skip=true##):
194 ** Verify that all plugins have versions specified
195 ** Verify that we don't use Commons Lang < 3 (i.e. that ##commons-lang:commons-lang## artifact is forbidden)
196 ** Verify the correct JUnit artifact is used (##junit-dep## and not ##junit##)
197 ** Verify we don't use Commons Logging or Log4j (since we use SLF4J)
198 ** Verify that the Java version used to release XWiki is correct (e.g. Java 8 starting with XWiki 8.1, Java 7 for XWiki 6.0 and Java 6 before)
199 ** Verify that Javadoc exist (in the release profile, this is a Maven Central requirement)
200 ** In Commons: Verify that Commons modules don't have a dependency on Rendering and Platform modules
201 ** In Rendering: Verify that Rendering modules don't have a dependency on Platform modules
202 * Spoon checks (To skip: ##-Dxwiki.spoon.skip=true##):
203 ** Verify none of the listed methods are called in our Java code
204 * For XARs, verify format and content using the [[XAR Plugin>>XARPlugin]]
205 * Verify that Test Percentage Coverage don't go down
206 ** This is achieved using Jacoco and the check is done at each module level using a Maven property named ##xwiki.jacoco.instructionRatio##.
207 * We check automatically that we don't have wiki pages that require Programming Rights unduly. This is exercised during functional tests. For more details see [[Testing documentation>>Community.Testing||anchor="HFunctionalTesting"]].
208 * Verify that test quality doesn't go down (for ##xwiki-commons## and ##xwiki-rendering## ATM), using PIT/Descartes. Note that this takes time and thus is executed only in the ##quality## profile and only when the ##-Dxwiki.pitest.skip=false## system property/value is passed.
209 ** The check is done at each module level using a Maven property named ##xwiki.pitest.mutationThreshold##.
210 ** There are jobs on XWiki's CI that execute regularly to check this:
211 *** [[xwiki-commons pitest job >>http://ci.xwiki.org/job/xwiki-commons_pitest/]]
212 *** [[xwiki-rendering pitest job >>http://ci.xwiki.org/job/xwiki-rendering_pitest/]]
213 *** [[xwiki-platform pitest job >>http://ci.xwiki.org/job/xwiki-platform_pitest/]]
214 * Verify that JUnit tests don't output content to stdout or stderr. Tests are supposed to either configure the code to not output anything or if it's expected then they're supposed to [[catch the logs and assert them>>dev:Community.Testing.JavaUnitTesting.WebHome#HCapturingandAssertingLogs]]. Note that this check is also executing for JUnit5 tests executed in your IDE (if the ##pom.xml## file in the current directory doesn't have the ##xwiki.surefire.captureconsole.skip## Maven property set to true).
215
216 = Tips =
217
218 == Skipping Tests ==
219
220 While you should almost never do that, here's a tip for the rare occasions when you'll need to build something fast:
221
222 {{code}}
223 mvn install -Dtest=none -DfailIfNoTests=false
224 {{/code}}
225
226 Alternatives are ##mvn install -DskipTests=true## or ##mvn install -Dmaven.test.skip=true##. Refere to the Maven documentation to understand the differences.
227
228 = Troubleshooting =
229
230 == Dealing with Out-of-Memory Errors ==
231
232 If you get an ##OutOfMemoryError## then increase the maximum heap space by setting the ##MAVEN_OPTS## environment variable. For example for building XWiki Products - Curriki - Database one seems to need ##MAVEN_OPTS=-Xmx100m## or higher.
233
234 If you're getting a ##PermGen space## error you need to increase the JVM PERMGEN. For example:
235
236 {{code language="none"}}
237 MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=128m"
238 {{/code}}
239
240 == Error in Windows installer build: Windres error ==
241
242 Launch4J is using a binary called Windres which requires that the environment is NOT set to use UTF-8. If you get the following error check your LANG environment variable:
243
244 {{code language="none"}}
245 Embedded error: net.sf.launch4j.BuilderException: net.sf.launch4j.ExecException: Exec failed(1): /var/tmp/installers/windows/target/dependency/bin/windres --preprocessor=cat -J rc -O coff -F pe-i386 /tmp/launch4j10018rc /tmp/launch4j10019o
246 {{/code}}
247
248 == Building behind a proxy ==
249
250 If you are connecting to the Internet through a proxy then you need to modify your ##settings.xml## file so that Maven knows it:
251
252 {{code language="xml"}}
253 <settings>
254 <proxies>
255 <proxy>
256 <active>true</active>
257 <protocol>http</protocol>
258 <host>host</host>
259 <port>port</port>
260 <username>uname</username>
261 <password>password</password>
262 </proxy>
263 </proxies>
264 <!--other tags-->
265 </settings>
266 {{/code}}
267
268 == Errors about missing resources ==
269
270 Some people have reported problems when building XWiki for the first time, or the first time after a version update. The error message is loosely looking like:
271
272 {{code}}
273 [...]
274 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.4:process (xwiki-license-resources) on project [[some-xwiki-project-here]]: Resources archive cannot be found. Could not find artifact org.xwiki.commons:[[some-xwiki-commons-lib]]:jar:x.y.z-SNAPSHOT
275 [ERROR]
276 [ERROR] Try downloading the file manually from the project website.
277 [ERROR]
278 [ERROR] Then, install it using the command:
279 [ERROR] mvn install:install-file -DgroupId=org.xwiki.commons -DartifactId=[[some-xwiki-commons-lib]] -Dversion=x.y.z-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
280 [ERROR]
281 [ERROR] Alternatively, if you host your own repository you can deploy the file there:
282 [ERROR] mvn deploy:deploy-file -DgroupId=org.xwiki.commons -DartifactId=[[some-xwiki-commons-lib]] -Dversion=x.y.z-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
283 [...]
284 {{/code}}
285
286 This is probably a problem with the maven version; try updating to the recommended maven version and retry.
287 \\Alternatively just call the maven build a second (or possible even a third time); the problem usually goes away for consecutive builds.
288
289 If the problem does not go away, check if http://nexus.xwiki.org/ is running. If it is not, please wait a moment until it comes back. (As this is a rather critical part of the infrastructure, you can be pretty sure it gets fixed quickly.)
290
291 = Build Tools =
292
293 We have developed some build tools in the form of Maven plugins. They are located in the [[##commons/tools##>>https://github.com/xwiki/xwiki-commons/tree/master/xwiki-commons-tools]] and [[##platform/xwiki-tools/##>>https://github.com/xwiki/xwiki-platform/tree/master/xwiki-platform-tools]] directories. Here are descriptions of some of them:
294
295 * ##xwiki-packager-plugin##: Takes XWiki pages defined as XML files and located in ##src/main/resources/## and load them into a database (defined by ##src/main/packager/hibernate.cfg.xml##). An example usage is available [[here>>https://github.com/xwiki/xwiki-platform/tree/master/xwiki-platform-tools/xwiki-platform-tool-packager-plugin/src/it/basic-import-export]].
296 * ##[[xwiki-xar-plugin>>XARPlugin]]##: Takes XWiki pages defined as XML files and located in ##src/main/resources/## and generates a XAR (ZIP format) out of them.
297 * ##[[xwiki-enterprise-archetype>>XEArchetype]]##: A Maven Archetype to create a Maven project extending XWiki Enterprise. This is useful is you're planning to extend XWiki Enterprise and create a distribution for your product.
298 * ##[[xwiki-extension-plugin>>ExtensionPlugin]]##: Various extension oriented mojos for Maven build. Current mostly about generating extensions descriptors (.xed) in various situations (WAR dependencies, etc.).
299 * ##[[xwiki-commons-tool-webjar-handlers>>WebJarHandler]]##: A handler for Maven type ##webjar## to use to produce explicitly typed and standard webjar artifacts.
300 * ##[[xwiki-platform-tool-provision-plugin>>ProvisionPlugin]]##: Installs Extensions into a running XWiki instance
301 * ##[[xwiki-commons-tool-remote-resource-plugin>>XWikiRemoteResourcePlugin]]##: A more memory friendly alternative to standard Maven Remote Resource Plugin
302
303 Also, you can find some contrib tools:
304
305 * [[Off-line XWiki Repository Packager Maven Plugin>>https://github.com/xwiki-contrib/maven-plugin-offline-xwiki-repository-packager]]
306
307 = Building in Eclipse =
308
309 * [[BuildingInEclipse]]
310
311 = Building in IntelliJ IDEA =
312
313 * [[BuildingInIdea]]
314
315 = Creating a Build for your own Modules =
316
317 == Generating a XAR ==
318
319 Follow these steps:
320
321 * Put the XWiki XML files into the ##src/main/resources## directory
322 * Use the ##xar## packaging in the your ##pom.xml##.
323 * Make sure the XAR packaging extension is registered in Maven.
324
325 For example:
326
327 {{code language="xml"}}
328 <project...>
329 ...
330 <packaging>xar</packaging>
331 ...
332 <build>
333 <extensions>
334 <!-- Needed to add support for the "xar" packaging -->
335 <extension>
336 <groupId>org.xwiki.commons</groupId>
337 <artifactId>xwiki-commons-tool-xar-handlers</artifactId>
338 <version>(version to use)</version>
339 </extension>
340 </extensions>
341 </build>
342 ...
343 </project>
344 {{/code}}
345
346 Make sure you run ##mvn xar:format## to ensure your XWiki XML files are properly formatted, following [[XWiki XML files best practices>>XWikiXMLFilesCodeStyle]].
347
348 = Building a patched version =
349
350 If you want to patch a part of XWiki without compiling the entire software, you may take a look at the following [[blog article>>http://hole.tuziwo.info/patch-a-dependency-of-maven.html]]. It explains how to patch and compile only the JAR archive you want, without building all of the JAR archives of XWiki.

Get Connected