Building XWiki from sources

Version 106.1 by Thomas Mortagne on 2018/06/07 09:10

Make sure you install a Maven version >= 3.5.0. Note that our CI builds XWiki using Maven 3.5.2 and thus this is our recommended version.

Windows users have several choices for getting past the long paths issue (255 char limitation on Windows) and building XWiki:

Understanding the directory structure

Before you start it's good to have some minimal understanding of how Maven works. You probably won't need it if everything goes fine but you'll need that knowledge if something breaks! ;-)

The first thing to understand is the directory structure we use.

Checking out the sources

Use your favorite SCM client to check out the Project you wish to build. 

Installing Maven

  • Install Maven 3.5.2 or greater. Create an M2_HOME environment variable pointing to where you have installed Maven. Add M2_HOME/bin to your PATH environment variable.

    For more information on how to install Maven or how to use it check this excellent Maven book.

  • Make sure you give Maven's JVM enough memory. XWiki's build uses Aspects and GWT compilers which need lots of memory. A good value is to give the JVM 1024MB. To do that set an environment variable named MAVEN_OPTS. For example on Unix add the following to your shell startup script (depending on your shell you might need to prefix the variable with export):
    MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m"
  • Create a ~/.m2/settings.xml file with the XWiki custom remote repository defined as shown below (if you're using a Repository Manager such as Nexus or Artifactory, you should add the XWiki Maven Remote repositories in there instead). If you are on Windows, create the directory .m2 in your home directory, e.g. C:\Documents and Settings\Administrator\.m2. If you cannot do this from Windows Explorer, open a command line and use md .m2.

    You might be wondering why we don't push all XWiki artifacts to Maven Central and thus remove the need to declare the XWiki Maven Repositories.

    Actually we do this for XWiki Commons and XWiki Rendering. Since these 2 projects contain libraries that are independent of the concept of wikis, this makes it very easy for everyone to use them in their own projects.

    We haven't done so for XWiki Platform and XWiki Enterprise this far because Maven Central imposes a requirement that we cannot fulfil at the moment: all artifacts deployed there must not have dependencies on other artifacts that are not present on Maven Central. And we do have a lot of such 3rd party dependencies. Some of them do not even exist in any Maven Repository and we put them in our XWiki Externals Repository.

    Note that we recommend using a Maven Repository Manager to make it easier to maintain your Maven Repositories setup.

    <settings>
     <profiles>
       <profile>
         <id>xwiki</id>
         <repositories>
           <repository>
             <id>xwiki-snapshots</id>
             <name>XWiki Nexus Snapshot Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </repository>
           <repository>
             <id>xwiki-releases</id>
             <name>XWiki Nexus Releases Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </repository>
         </repositories>
         <pluginRepositories>
           <pluginRepository>
             <id>xwiki-snapshots</id>
             <name>XWiki Nexus Plugin Snapshot Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </pluginRepository>
           <pluginRepository>
             <id>xwiki-releases</id>
             <name>XWiki Nexus Plugin Releases Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </pluginRepository>
         </pluginRepositories>
       </profile>
     </profiles>
     <activeProfiles>
       <activeProfile>xwiki</activeProfile>
     </activeProfiles>
    </settings>

    Depending on your geographical location, you should use closer Maven repositories to speed up your build process. For more information refer to the Maven Mirror Settings Guide.

Building with Maven

Now that you have installed Maven and checked out the Projects you wanted to work on, you'll want to build them or parts of them. A project is made of modules. To build a module in Maven just issue the command:

mvn clean install

Thanks to XWiki's Continuous Integration setup you can check out and build any single module (and its children) you wish without having to rebuild the other modules from source. Maven will pick the latest version of your module's dependencies from XWiki's Maven remote repositories. Of course if you have uncommitted changes in a dependent module, you'll want to build that module before. The build result is placed in two places:

  • Your local maven repository, for making the module available to other projects or modules using maven (even other XWiki modules take the needed libraries from this repository, and not directly from a "neighbor" directory)
  • In a subdirectory of that module, called target.

Some of the modules built by XWiki are very large (in the 40MB+ size) and sometimes you don't absolutely need the latest version to get downloaded as it'll take too long. In that case here's a tip: run Maven in 'no-snapshot-update' mode with mvn install -nsu.

You might encounter a java.lang.OutOfMemoryError error during aspectj:compile. You must increase the maximum memory used by maven, by setting MAVEN_OPTS=-Xmx1024m (or a larger amount of memory if needed).

Using Profiles

Project builds define several Maven Profiles. We've standardized their names across Projects. 

Here are the most common ones:

  • legacy: Includes the legacy modules in the build.
  • integration-tests: Executes integration and functional tests in the build.
  • jetty, glassfish: Run the build for the specified container (jetty is the default when not specified).
  • hsqldb (Hypersonic SQL), mysql (MySQL), pgsql (PostgreSQL), derby (Derby): Run the build for the specified database (hsqldb is the default when not specified).

    Hypersonic and Derby are embedded database so you won't need anything setup to execute the build for them. However for MySQL and PostgreSQL you'll need to have the database installed, set up and running before you can run the build with their profiles.

  • firefox (Firefox), iexplore (Internet Explorer), chrome (Google Chrome), otherbrowser (need to define browserPath property to use that one): Run the functional tests on the specified browser (firefox is the default when not specified).
  • dev: only build wysiwyg for English/Firefox. Very useful to speed up the build of XWiki Platform
  • distribution: Since 9.5RC1 Also build distribution packages (the WAR, the flavor, the jetty/hsqldb package, etc)

More "exotic" ones:

  • release: Used by the Maven Release plugin when releasing XWiki. It generates Javadoc and Source artifacts. It also checks that the Java version used is JDK6, that the Javadoc exists and it signs artifacts using GPG (this is a good practice and a requirement for being able to upload some of our artifacts to the Maven Central Repository).
  • snapshotModules: Since 6.2M1 To be used on development/snapshot builds regularly ran on the CI (so never on a release!). This way, the resulting XWiki build includes the XWiki snapshots Extension Repository Source preconfigured and anyone downloading it for local testing will not have to enable it by hand. Distribution Wizard and Extension Manager will thus install the latest published snapshot extension builds and not fail completely saying that it can not find snapshot versions.
  • m2e: For m2eclipse users. It sets the Eclipse output directory to target-eclipse (instead of target) to prevent race conditions between Maven within Eclipse and Maven on the command line.
  • unix, mac, windows: These profiles are automatically activated depending on the OS the build is running on. These profiles are useful for the Installers and for functional tests to decide how to start XWiki.
  • macprofiler, winprofiler: Start XWiki for Profiling (they require that you set a profilePath property in your settings.xml or on the command line)
  • gwt-test-manual: allow running GWT unit tests manually
  • debug: remove JS minification.
  • standalone: Builds the Rendering Standalone artifact. Note that this profile is activated automatically when performing a release.
  • quality: Runs quality checks that take a long time to execute (e.g. checks Jacoco TPC thresholds)
  • clover: A profile to use when generating Test Coverage reports with Clover. This profile skips execution of Checkstyle, Backward Compatibility and Enforcer plugin checks to speed up the coverage generation and because of potential conflicts between tools. Note that even though we've started using Jacoco for our quality profile we still support Clover. The Clover reports are much nicer than the Jacoco ones and it's thus still useful to be able to generate them.
  • office-tests: Since 7.3M1 Executes integration and functional tests related to the office module (Office Importer Application, Office Macro, ...) in the build.
  • flavor-integration-tests: Since 9.5RC1 Execute integration and functional tests related to the the default flavor located in xwiki-platform

For example, if you use to include legacy modules and run all integration and functional tests and use Jetty/HSQLDB, you would use:

mvn clean install -Plegacy,integration-tests,hsqldb,jetty

Relationship between XWiki Enterprise and XWiki Platform

XWiki Platform is a generic platform and API for building flexible extensible wikis and web applications. XWiki Enterprise is an implementation of the platform geared toward making a wiki site for teams to share information and synchronize their efforts although it can be and has been used for a wide variety of different use cases like this website.

What does this mean to the developer?
If you wish to make changes in the core of XWiki, you are probably going to want to work on XWiki Platform. You can download and compile XWiki Enterprise and while compiling, you will notice Maven downloads a big 80MB+ file, this is the platform. Maven is smart enough to know that you don't have a copy of the platform and will download it from the XWiki Maven repository. Once it has been downloaded the first time, it is stored on your hard drive and you won't have to download it again until changes are made in the platform.

If you want to work on the core or if you have limited bandwidth, then you are well advised to get the platform sources. Then you can use the SCM's update command to get the latest copy without downloading the entire platform all over again. If you download the platform and compile it then you can compile XWiki Enterprise, Maven will use your newly compiled copy of the platform rather than downloading it, Maven is smart.

Automatic Checks

The XWiki build performs some automated checks:

  • Style checks with Checkstyle. To skip: -Dxwiki.checkstyle.skip=true. We also have custom Checkstyle checks:
    • Verify that components.txt contains all Components
    • Verify that @Unstable annotation don't stay too long (and that a @since annotation is present too)
    • Verify that Script Services are not located in the internal package
    • Verify that @since javadoc tags have the correct format (and that they don't contain several versions per tag)
  • Verify header licenses
  • Backward compatibility checks with revapi. To skip: -Dxwiki.revapi.skip=true.
  • Enforcer checks (To skip: -Dxwiki.enforcer.skip=true):
    • Verify that all plugins have versions specified
    • Verify that we don't use Commons Lang < 3 (i.e. that commons-lang:commons-lang artifact is forbidden)
    • Verify the correct JUnit artifact is used (junit-dep and not junit)
    • Verify we don't use Commons Logging or Log4j (since we use SLF4J)
    • Verify that the Java version used to release XWiki is correct (e.g. Java 8 starting with XWiki 8.1, Java 7 for XWiki 6.0 and Java 6 before)
    • Verify that Javadoc exist (in the release profile, this is a Maven Central requirement)
    • In Commons:  Verify that Commons modules don't have a dependency on Rendering and Platform modules
    • In Rendering: Verify that Rendering modules don't have a dependency on Platform modules
  • Spoon checks (To skip: -Dxwiki.spoon.skip=true):
    • Verify none of the listed methods are called in our Java code
  • For XARs, verify format and content using the XAR Plugin
  • Verify that Test Percentage Coverage don't go down
  • We check automatically that we don't have wiki pages that require Programming Rights unduly. This is exercised during functional tests. For more details see Testing documentation.
  • Verify that test quality doesn't go down (for xwiki-commons ATM), using PIT/Descartes Since 10.3RC1. Note that this takes time and thus is executed only in the quality profile and only when the -Dxwiki.pitest.skip=false system property/value is passed. There's a job on XWiki's CI that executes regularly to check this.

Tips

Skipping Tests

While you should almost never do that, here's a tip for the rare occasions when you'll need to build something fast:

mvn install -Dtest=none -DfailIfNoTests=false

Alternatives are mvn install -DskipTests=true or mvn install -Dmaven.test.skip=true. Refere to the Maven documentation to understand the differences.

Troubleshooting

Dealing with Out-of-Memory Errors

If you get an OutOfMemoryError then increase the maximum heap space by setting the MAVEN_OPTS environment variable. For example for building XWiki Products - Curriki - Database one seems to need MAVEN_OPTS=-Xmx100m or higher.

If you're getting a PermGen space error you need to increase the JVM PERMGEN. For example:

MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=128m"

Error in Windows installer build: Windres error

Launch4J is using a binary called Windres which requires that the environment is NOT set to use UTF-8. If you get the following error check your LANG environment variable:

Embedded error: net.sf.launch4j.BuilderException: net.sf.launch4j.ExecException: Exec failed(1): /var/tmp/installers/windows/target/dependency/bin/windres --preprocessor=cat -J rc -O coff -F pe-i386 /tmp/launch4j10018rc /tmp/launch4j10019o

Building behind a proxy

If you are connecting to the Internet through a proxy then you need to modify your settings.xml file so that Maven knows it:

<settings>
 <proxies>
   <proxy>
     <active>true</active>
     <protocol>http</protocol>
     <host>host</host>
     <port>port</port>
     <username>uname</username>
     <password>password</password>
   </proxy>
 </proxies>
 <!--other tags-->
</settings>

Errors about missing resources

Some people have reported problems when building XWiki for the first time, or the first time after a version update. The error message is loosely looking like:

[...]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.4:process (xwiki-license-resources) on project [[some-xwiki-project-here]]: Resources archive cannot be found. Could not find artifact org.xwiki.commons:[[some-xwiki-commons-lib]]:jar:x.y.z-SNAPSHOT
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.xwiki.commons -DartifactId=[[some-xwiki-commons-lib]] -Dversion=x.y.z-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.xwiki.commons -DartifactId=[[some-xwiki-commons-lib]] -Dversion=x.y.z-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[...]

This is probably a problem with the maven version; try updating to the recommended maven version and retry.

Alternatively just call the maven build a second (or possible even a third time); the problem usually goes away for consecutive builds.

If the problem does not go away, check if http://nexus.xwiki.org/ is running. If it is not, please wait a moment until it comes back. (As this is a rather critical part of the infrastructure, you can be pretty sure it gets fixed quickly.)

Build Tools

We have developed some build tools in the form of Maven plugins. They are located in the commons/tools and platform/xwiki-tools/ directories. Here are descriptions of some of them:

  • xwiki-packager-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and load them into a database (defined by src/main/packager/hibernate.cfg.xml). An example usage is available here.
  • xwiki-xar-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and generates a XAR (ZIP format) out of them.
  • xwiki-enterprise-archetype: A Maven Archetype to create a Maven project extending XWiki Enterprise. This is useful is you're planning to extend XWiki Enterprise and create a distribution for your product.
  • xwiki-extension-plugin: Various extension oriented mojos for Maven build. Current mostly about generating extensions descriptors (.xed) in various situations (WAR dependencies, etc.).
  • xwiki-commons-tool-webjar-handlers: A handler for Maven type webjar to use to produce explicitly typed and standard webjar artifacts.
  • xwiki-platform-tool-provision-plugin: Installs Extensions into a running XWiki instance
  • xwiki-commons-tool-remote-resource-plugin: A more memory friendly alternative to standard Maven Remote Resource Plugin

Also, you can find some contrib tools:

Building in Eclipse

Building in IntelliJ IDEA

Creating a Build for your own Modules

Generating a XAR

Follow these steps:

  • Put the XWiki XML files into the src/main/resources directory
  • Use the xar packaging in the your pom.xml.
  • Make sure the XAR packaging extension is registered in Maven.

For example:

<project...>
...
 <packaging>xar</packaging>
...
 <build>
   <extensions>
     <!-- Needed to add support for the "xar" packaging -->
     <extension>
       <groupId>org.xwiki.commons</groupId>
       <artifactId>xwiki-commons-tool-xar-handlers</artifactId>
       <version>(version to use)</version>
     </extension>
   </extensions>
 </build>
...
</project>

Make sure you run mvn xar:format to ensure your XWiki XML files are properly formatted, following XWiki XML files best practices.

Building a patched version

If you want to patch a part of XWiki without compiling the entire software, you may take a look at the following blog article.  It explains how to patch and compile only the JAR archive you want, without building all of the JAR archives of XWiki.

Tags:
   

Get Connected