Building XWiki from sources

Last modified by Lucas Charpentier (Sereza7) on 2024/01/19 11:27

Build Requirements

  • See Java support to find which version to build XWiki with
    • e.g. Use Java 8 for XWiki < 14.0, Java 11 for 14.0 =< XWiki < 16.0.0 and Java 17 for XWiki >= 16.0.0
  • Make sure you install a Maven version >= 3.6.3. Note that our CI builds XWiki using Maven 3.9.6 and thus this is our recommended version.
  • Windows users have several choices for getting past the long paths issue (255 char limitation on Windows) and building XWiki:
  • If you're using openJDK, you must use a version equal or newer than 8u191-b12.
  • To run functional tests you need to have Docker installed.

Understanding the directory structure

Before you start it's good to have some minimal understanding of how Maven works. You probably won't need it if everything goes fine but you'll need that knowledge if something breaks! ;-)

The first thing to understand is the directory structure we use.

Checking out the sources

Use your favorite SCM client to check out the Project you wish to build. 

Installing Maven

  • Install Maven 3.9.6 or greater. Create an M2_HOME environment variable pointing to where you have installed Maven. Add M2_HOME/bin to your PATH environment variable.

    For more information on how to install Maven or how to use it check this excellent Maven book.

  • Make sure you give Maven's JVM enough memory. XWiki's build uses Aspects and GWT compilers which need lots of memory. A good value is to give the JVM 1024MB. To do that set an environment variable named MAVEN_OPTS. For example on Unix add the following to your shell startup script (depending on your shell you might need to prefix the variable with export):
    MAVEN_OPTS="-Xmx1024m"

    Note that XWiki's tests are executed in a separate JVM which requires 1024MB of memory. Thus, in total, you need to have 2GB of free RAM to build XWiki.

  • Create a ~/.m2/settings.xml file with the XWiki custom remote repository defined as shown below (if you're using a Repository Manager such as Nexus or Artifactory, you should add the XWiki Maven Remote repositories in there instead). If you are on Windows, create the .m2 directory in your %USERPROFILE% directory, e.g. C:\Users\YourUserName\.m2. If you cannot do this from Windows Explorer, open a command line and execute command md .m2 from inside of the %USERPROFILE% directory.

    You might be wondering why we don't push all XWiki artifacts to Maven Central and thus remove the need to declare the XWiki Maven Repositories.

    Actually we do this for XWiki Commons and XWiki Rendering. Since these 2 projects contain libraries that are independent of the concept of wikis, this makes it very easy for everyone to use them in their own projects.

    We haven't done so for XWiki Platform this far because Maven Central imposes a requirement that we cannot fulfil at the moment: all artifacts deployed there must not have dependencies on other artifacts that are not present on Maven Central. And we do have a lot of such 3rd party dependencies. Some of them do not even exist in any Maven Repository and we put them in our XWiki Externals Repository.

    Note that we recommend using a Maven Repository Manager to make it easier to maintain your Maven Repositories setup.

    <settings>
     <profiles>
       <profile>
         <id>xwiki</id>
         <repositories>
           <repository>
             <id>xwiki-snapshots</id>
             <name>XWiki Nexus Snapshot Repository Proxy</name>
             <url>https://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </repository>
           <repository>
             <id>xwiki-releases</id>
             <name>XWiki Nexus Releases Repository Proxy</name>
             <url>https://nexus.xwiki.org/nexus/content/groups/public</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </repository>
         </repositories>
         <pluginRepositories>
           <pluginRepository>
             <id>xwiki-snapshots</id>
             <name>XWiki Nexus Plugin Snapshot Repository Proxy</name>
             <url>https://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </pluginRepository>
           <pluginRepository>
             <id>xwiki-releases</id>
             <name>XWiki Nexus Plugin Releases Repository Proxy</name>
             <url>https://nexus.xwiki.org/nexus/content/groups/public</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </pluginRepository>
         </pluginRepositories>
       </profile>
     </profiles>
     <activeProfiles>
       <activeProfile>xwiki</activeProfile>
     </activeProfiles>
    </settings>

    Depending on your geographical location, you should use closer Maven repositories to speed up your build process. For more information refer to the Maven Mirror Settings Guide.

Building with Maven

Now that you have installed Maven and checked out the Projects you wanted to work on, you'll want to build them or parts of them. A project is made of modules. To build a module in Maven just issue the command:

mvn clean install

Thanks to XWiki's Continuous Integration setup you can check out and build any single module (and its children) you wish without having to rebuild the other modules from source. Maven will pick the latest version of your module's dependencies from XWiki's Maven remote repositories. Of course if you have uncommitted changes in a dependent module, you'll want to build that module before. The build result is placed in two places:

  • Your local maven repository, for making the module available to other projects or modules using maven (even other XWiki modules take the needed libraries from this repository, and not directly from a "neighbor" directory)
  • In a subdirectory of that module, called target.

Some of the modules built by XWiki are very large (in the 40MB+ size) and sometimes you don't absolutely need the latest version to get downloaded as it'll take too long. In that case here's a tip: run Maven in 'no-snapshot-update' mode with mvn install -nsu.

You might encounter a java.lang.OutOfMemoryError error during aspectj:compile. You must increase the maximum memory used by maven, by setting MAVEN_OPTS=-Xmx1024m (or a larger amount of memory if needed).

Using Profiles

Project builds define several Maven Profiles. We've standardized their names across Projects. 

Here are the most common ones:

  • legacy: Includes the legacy modules in the build.
  • integration-tests: Executes integration and functional tests in the build.
    • docker: Executes integration tests that are Docker-based. You'll need to have Docker installed for this to work.
  • jetty, glassfish: Run the build for the specified container (jetty is the default when not specified).
  • hsqldb (Hypersonic SQL), mysql (MySQL), pgsql (PostgreSQL), derby (Derby): Run the build for the specified database (hsqldb is the default when not specified).

    Hypersonic and Derby are embedded database so you won't need anything setup to execute the build for them. However for MySQL and PostgreSQL you'll need to have the database installed, set up and running before you can run the build with their profiles.

  • distribution: Since 9.5RC1 Also build distribution packages (the WAR, the flavor, the jetty/hsqldb package, etc)
    • flavor-integration-tests: Since 9.5RC1 Execute integration and functional tests related to the the default flavor located in xwiki-platform.

More "exotic" ones:

  • release: Used by the Maven Release plugin when releasing XWiki. It generates Javadoc and Source artifacts. It also checks that the Java version used is JDK6, that the Javadoc exists and it signs artifacts using GPG (this is a good practice and a requirement for being able to upload some of our artifacts to the Maven Central Repository).
  • snapshot: Since 11.8RC1 To be used when building snapshots (so never on a release!). Enabling this profile does the following:
    • The resulting XWiki Standard packagings includes the XWiki snapshots Extension Repository Source preconfigured and anyone downloading it for local testing will not have to enable it by hand. Distribution Wizard and Extension Manager will thus install the latest published snapshot extension builds and not fail completely saying that it can not find snapshot versions. Note that before version 11.8RC1, the profile used to be named snapshotModules and has been renamed to be more generic.
    • Defines automatically the XWiki Maven Repository so that you don't have to modify your Maven settings.xml to have it there. This is useful for tools such as LGTM which need to have the XWiki Maven Remote repo configured. It'll also be useful for simpler developer onboarding when we enable this profile automatically when the version in the pom.xml ends with SNAPSHOT (when we find how to do it! emoticon_wink).
  • m2e: For m2eclipse users. It sets the Eclipse output directory to target-eclipse (instead of target) to prevent race conditions between Maven within Eclipse and Maven on the command line.
  • unix, mac, windows: These profiles are automatically activated depending on the OS the build is running on. These profiles are useful for the Installers and for functional tests to decide how to start XWiki.
  • macprofiler, winprofiler: Start XWiki for Profiling (they require that you set a profilePath property in your settings.xml or on the command line)
  • gwt-test-manual: allow running GWT unit tests manually
  • debug: remove JS minification.
  • standalone: Builds the Rendering Standalone artifact. Note that this profile is activated automatically when performing a release.
  • quality: Runs quality checks that take a long time to execute (e.g. checks Jacoco TPC thresholds)
  • clover: A profile to use when generating Test Coverage reports with Clover. This profile skips execution of Checkstyle, Backward Compatibility and Enforcer plugin checks to speed up the coverage generation and because of potential conflicts between tools. Note that even though we've started using Jacoco for our quality profile we still support Clover. The Clover reports are much nicer than the Jacoco ones and it's thus still useful to be able to generate them.

For example, if you use to include legacy modules and run all integration and functional tests and use Jetty/HSQLDB, you would use:

mvn clean install -Plegacy,integration-tests,hsqldb,jetty

Building with Docker

XWiki offers a Docker image to build XWiki, that is used by XWiki's CI. It can also be used to build XWiki locally very easily. If you want to use it, you only need to install Docker and you don't need to have Maven installed or any of the other needed tools need to run XWiki's build (like a given version of Firefox for some old functional tests).

Check how to build with the XWiki Docker build image.

Build Cache

The XWiki build is using the Gradle Maven Extension (along with the Custom Common User Data Maven Extension) to speed up the build by caching locally and remotely the Maven goal outputs.

The configuration is done in the .mvn/extensions.xml, .mvn/gradle-enterprise*.* files.

If you need to skip the build caches you can execute Maven with the following system property:

Automatic Checks

The XWiki build performs some automated checks:

  • Style checks with Checkstyle. To skip: -Dxwiki.checkstyle.skip=true. To run on the command line: mvn checkstyle:check@default -fae | grep ERROR. We have the following custom Checkstyle checks:
    • Verify that @Unstable annotation don't stay too long (and that a @since annotation is present too)
    • Verify that Script Services are not located in the internal package
    • Verify that @since javadoc tags have the correct format (and that they don't contain several versions per tag). Example of correct format: @since 10.8RC1.
  • Verify header licenses with the Mycila Maven license plugin.
  • Backward compatibility checks with revapi. To skip: -Dxwiki.revapi.skip=true.
  • Maven Enforcer checks (To skip: -Dxwiki.enforcer.skip=true):
    • Verify that all plugins have versions specified
    • Verify that we don't use Commons Lang < 3 (i.e. that commons-lang:commons-lang artifact is forbidden)
    • Verify the correct JUnit artifact is used (junit-dep and not junit)
    • Verify we don't use Commons Logging or Log4j (since we use SLF4J)
    • Verify that the Java version used to release XWiki is correct (e.g. Java 8 starting with XWiki 8.1, Java 7 for XWiki 6.0 and Java 6 before)
    • Verify that Javadoc exist (in the release profile, this is a Maven Central requirement)
    • In Commons:  Verify that Commons modules don't have a dependency on Rendering and Platform modules
    • In Rendering: Verify that Rendering modules don't have a dependency on Platform modules
  • Spoon checks (To skip: -Dxwiki.spoon.skip=true):
    • Verify none of the listed methods are called in our Java code
    • Verify that components.txt contains all Components
    • Verify that JUnit4 and JUnit5 APIs are not both used at the same time in a test
    • Verify that the @Inject annotation is done on the component role (fails if not done by error on the component implementation, unless it's using the roles attribute of the @Component annotation and it points to itsself). Since 12.4RC1

      It's possible to add exceptions on this check for some specific cases (e.g. injecting an external interface that we don't control). Exceptions are configured in the pom.xml with a configuration such as: 

          <build>
         <plugins>
           <plugin>
             <groupId>fr.inria.gforge.spoon</groupId>
             <artifactId>spoon-maven-plugin</artifactId>
             <executions>
               <execution>
                 <id>spoon-main</id>
                 <configuration>
                   <processorProperties combine.children="append">
                     <processorProperty>
                       <name>org.xwiki.tool.spoon.InjectAnnotationProcessor</name>
                       <properties>
                         <property>
                           <!-- Exclude javax.script.ScriptEngineFactory since it's not an XWiki interface and thus
                                 doesn't have a @Role annotation. there's no way for the Spoon checker to find
                                 out about this, since only the Component Manager knows what has been registered for it. -->

                           <name>excludedFieldTypes</name>
                           <value><![CDATA[
                              ["javax.script.ScriptEngineFactory"]
                            ]]>
      </value>
                         </property>
                       </properties>
                     </processorProperty>
                   </processorProperties>
                 </configuration>
               </execution>
             </executions>
           </plugin>
         </plugins>
       </build>
  • For XARs, verify format and content using the XAR Plugin
  • Verify that Test Percentage Coverage don't go down
    • This is achieved using Jacoco and the check is done at each module level using a Maven property named xwiki.jacoco.instructionRatio.
    • Note: We also perform a global coverage check but it's done only in the CI
  • We check automatically that we don't have wiki pages that require Programming Rights unduly. This is exercised during functional tests. For more details see Testing documentation.
  • Verify Test output
    • Verify that JUnit tests don't output content to stdout or stderr. Tests are supposed to either configure the code to not output anything or if it's expected then they're supposed to catch the logs and assert them. Note that this check is also executing for JUnit5 tests executed in your IDE (if the pom.xml file in the current directory doesn't have the xwiki.surefire.captureconsole.skip Maven property set to true).
    • Verify that functional tests don't contain errors or warnings when they execute. See Docker Tests (for JUnit5) and Selenium2 Tests (for JUnit4).
  • Verify code quality with SonarQube. This is achieved using SonarCloud.io which is configured so that any Blocker severity issue on new code (i.e. code that is less than 30 days old) fails the XWiki CI build.

Tips

Skipping Tests

While you should almost never do that, here's a tip for the rare occasions when you'll need to build something fast:

mvn install -Dtest=none -DfailIfNoTests=false

Alternatives are mvn install -DskipTests=true or mvn install -Dmaven.test.skip=true. Refere to the Maven documentation to understand the differences.

Troubleshooting

Dealing with Out-of-Memory Errors

If you get an OutOfMemoryError then increase the maximum heap space by setting the MAVEN_OPTS environment variable. For example:

MAVEN_OPTS="-Xmx1024m"

Error in Windows installer build: Windres error

Launch4J is using a binary called Windres which requires that the environment is NOT set to use UTF-8. If you get the following error check your LANG environment variable:

Embedded error: net.sf.launch4j.BuilderException: net.sf.launch4j.ExecException: Exec failed(1): /var/tmp/installers/windows/target/dependency/bin/windres --preprocessor=cat -J rc -O coff -F pe-i386 /tmp/launch4j10018rc /tmp/launch4j10019o

Building behind a proxy

If you are connecting to the Internet through a proxy then you need to modify your settings.xml file so that Maven knows it:

<settings>
 <proxies>
   <proxy>
     <active>true</active>
     <protocol>http</protocol>
     <host>host</host>
     <port>port</port>
     <username>uname</username>
     <password>password</password>
   </proxy>
 </proxies>
 <!--other tags-->
</settings>

Errors about missing resources

Some people have reported problems when building XWiki for the first time, or the first time after a version update. The error message is loosely looking like:

[...]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.4:process (xwiki-license-resources) on project [[some-xwiki-project-here]]: Resources archive cannot be found. Could not find artifact org.xwiki.commons:[[some-xwiki-commons-lib]]:jar:x.y.z-SNAPSHOT
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.xwiki.commons -DartifactId=[[some-xwiki-commons-lib]] -Dversion=x.y.z-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.xwiki.commons -DartifactId=[[some-xwiki-commons-lib]] -Dversion=x.y.z-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[...]

This is probably a problem with the maven version; try updating to the recommended maven version and retry.

Alternatively just call the maven build a second (or possible even a third time); the problem usually goes away for consecutive builds.

If the problem does not go away, check if https://nexus.xwiki.org/ is running. If it is not, please wait a moment until it comes back. (As this is a rather critical part of the infrastructure, you can be pretty sure it gets fixed quickly.)

Build Tools

We have developed some build tools in the form of Maven plugins. They are located in the xwiki-commons/xwiki-commons-tools and xwiki-platform/xwiki-platform-tools/ directories. Here are descriptions of some of them:

  • xwiki-packager-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and load them into a database (defined by src/main/packager/hibernate.cfg.xml). An example usage is available here.
  • xwiki-xar-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and generates a XAR (ZIP format) out of them.
  • xwiki-extension-plugin: Various extension oriented mojos for Maven build. Current mostly about generating extensions descriptors (.xed) in various situations (WAR dependencies, etc.).
  • xwiki-commons-tool-webjar-handlers: A handler for Maven type webjar to use to produce explicitly typed and standard webjar artifacts.
  • xwiki-platform-tool-provision-plugin: Installs Extensions into a running XWiki instance
  • xwiki-commons-tool-remote-resource-plugin: A more memory friendly alternative to standard Maven Remote Resource Plugin

Also, you can find some contrib tools:

Building in Eclipse

Building in IntelliJ IDEA

Creating a Build for your own Modules

Generating a XAR

Follow these steps:

  • Put the XWiki XML files into the src/main/resources directory
  • Use the xar packaging in the your pom.xml.
  • Make sure the XAR packaging extension is registered in Maven.

For example:

<project...>
...
 <packaging>xar</packaging>
...
 <build>
   <extensions>
     <!-- Needed to add support for the "xar" packaging -->
     <extension>
       <groupId>org.xwiki.commons</groupId>
       <artifactId>xwiki-commons-tool-xar-handlers</artifactId>
       <version>(version to use)</version>
     </extension>
   </extensions>
 </build>
...
</project>

Make sure you run mvn xar:format to ensure your XWiki XML files are properly formatted, following XWiki XML files best practices.

Building a patched version

If you want to patch a part of XWiki without compiling the entire software, you may take a look at the following blog article.  It explains how to patch and compile only the JAR archive you want, without building all of the JAR archives of XWiki.

Gradle Enterprise Cache

In order to speed up the XWiki Maven builds (either on the dev machines or on the CI), we use a Gradle Enterprise Cache.

This is the setup for XWiki:

  • In each Commons, Rendering, Platform source repos, inside .mvn there are 3 files:
    • extensions.xml: Declares the Gradle extension to Maven
    • gradle-enterprise.xml: Configuration for Gradle Enterprise
    • gradle-enterprise-custom-user-data.groovy: Custom configuration for Gradle Enterprise to set the Maven profiles as ge.xwiki.org tags
  • In the Jenkins Docker Cloud configuration, we bind a volume from the agent host to pass credential keys to authenticate the CI against ge.xwiki.org
  • On each agent host, there's a .m2/.gradle-enterprise/keys.properties file with an access key from the ci user on ge.xwiki.org (mounted as a volume for the Docker Cloud build image : type=bind,source=/home/hudsonagent/.m2/.gradle-enterprise/keys.properties,destination=/root/.m2/.gradle-enterprise/keys.properties)
  • In the xwikiBuild.groovy Jenkins pipeline, we pass a system property to disable caching fingerprints as it costs more to download them from the remote cache than to recompute them (it slows down the build otherwise)
Tags:
   

Get Connected