XWiki Development Zone » Building XWiki from sources

Building XWiki from sources

Last modified by Manuel Smeria on 2014/07/31 14:04

Make sure you install a Maven version >= 3.0 but < 3.1.1. It might build fine using earlier versions but we're not supporting them and all the XWiki committers and our CI server are running 3.0 and greater. We currently have an issue with Maven 3.1.1. The version that we currently recommend is 3.0.5.

Windows users have 2 choices for getting pass the long paths issue (255 char limitation on Windows) and building XWiki:

  • Use Git for Windows v1.9.0+ and set core.longpaths to True:
    git config --global core.longpaths true
  • Use Cygwin which works out of the box

Understanding the directory structure

Before you start it's good to have some minimal understanding of how Maven works. You probably won't need it if everything goes fine but you'll need that knowledge if something breaks! ;-)

The first thing to understand is the directory structure we use.

Checking out the sources

Use your favorite SCM client to check out the Project you wish to build. 

Installing Maven

  • Install Maven 3.0 or greater. Create an M2_HOME environment variable pointing to where you have installed Maven. Add M2_HOME/bin to your PATH environment variable.
    For more information on how to install Maven or how to use it check this excellent Maven book.
  • Make sure you give Maven's JVM enough memory. XWiki's build uses Aspects and GWT compilers which need lots of memory. A good value is to give the JVM 512MB. To do that set an environment variable named MAVEN_OPTS. For example on Unix add the following to your shell startup script (depending on your shell you might need to prefix the variable with export):
    MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
  • Create a ~/.m2/settings.xml file with the XWiki custom remote repository defined as shown below. If you are on Windows, create the directory .m2 in your home directory, e.g. C:\Documents and Settings\Administrator\.m2. If you cannot do this from Windows Explorer, open a command line and use md .m2.
    <settings>
     <profiles>
       <profile>
         <id>xwiki</id>
         <repositories>
           <repository>
             <id>xwiki-snapshots</id>
             <name>XWiki Nexus Snapshot Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </repository>
           <repository>
             <id>xwiki-releases</id>
             <name>XWiki Nexus Releases Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </repository>
         </repositories>
         <pluginRepositories>
           <pluginRepository>
             <id>xwiki-plugins-snapshots</id>
             <name>XWiki Nexus Plugin Snapshot Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public-snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </pluginRepository>
           <pluginRepository>
             <id>xwiki-plugins-releases</id>
             <name>XWiki Nexus Plugin Releases Repository Proxy</name>
             <url>http://nexus.xwiki.org/nexus/content/groups/public</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </pluginRepository>
         </pluginRepositories>
       </profile>
     </profiles>
     <activeProfiles>
       <activeProfile>xwiki</activeProfile>
     </activeProfiles>
    </settings>
    Depending on your geographical location, you should use closer Maven repositories to speed up your build process. For more information refer to the Maven Mirror Settings Guide.

Building with Maven

Now that you have installed Maven and checked out the Projects you wanted you'll want to build them. To build a project with Maven just issue the command:

mvn clean install

Each project is made of modules. Thanks to XWiki's Continuous Integration setup you can check out and build any single module (and its children) you wish without having to rebuild the other modules from source. Maven will pick the latest version of your module's dependencies from XWiki's Maven remote repositories. Of course if you have uncommitted changes in a dependent module, you'll want to build that module before. The build result is placed in two places:

  • Your local maven repository, for making the module available to other projects or modules using maven (even other XWiki modules take the needed libraries from this repository, and not directly from a "neighbor" directory)
  • In a subdirectory of that module, called target.
Some of the modules built by XWiki are very large (in the 40MB+ size) and sometimes you don't absolutely need the latest version to get downloaded as it'll take too long. In that case here's a tip: run Maven in 'no-snapshot-update' mode with mvn install -nsu.
You might encounter a java.lang.OutOfMemoryError error during aspectj:compile. You must increase the maximum memory used by maven, by setting MAVEN_OPTS=-Xmx512m (or a larger amount of memory if needed).

Below you'll find some common build tasks.

Using Profiles

Project builds define several Maven Profiles. We've standardized their names across Projects. 

Here are the most common ones:

  • legacy: Includes the legacy modules in the build.
  • integration-tests: Executes integration and functional tests in the build.
  • jetty, glassfish: Run the build for the specified container (jetty is the default when not specified).
  • hsqldb (Hypersonic SQL), mysql (MySQL), pgsql (PostgreSQL), derby (Derby): Run the build for the specified database (hsqldb is the default when not specified).
    Hypersonic and Derby are embedded database so you won't need anything setup to execute the build for them. However for MySQL and PostgreSQL you'll need to have the database installed, set up and running before you can run the build with their profiles.
  • firefox (Firefox), iexplore (Internet Explorer), chrome (Google Chrome), otherbrowser (need to define browserPath property to use that one): Run the functional tests on the specified browser (firefox is the default when not specified).
  • dev: only build wysiwyg for English/Firefox. Very useful to speed up the build of XWiki Platform

More "exotic" ones:

  • release: Used by the Maven Release plugin when releasing XWiki. It generates Javadoc and Source artifacts. It also checks that the Java version used is JDK6, that the Javadoc exists and it signs artifacts using GPG (this is a good practice and a requirement for being able to upload some of our artifacts to the Maven Central Repository).
  • m2e: For m2eclipse users. It sets the Eclipse output directory to target-eclipse (instead of target) to prevent race conditions between Maven within Eclipse and Maven on the command line.
  • unix, mac, windows: These profiles are automatically activated depending on the OS the build is running on. These profiles are useful for the Installers and for functional tests to decide how to start XWiki.
  • macprofiler, winprofiler: Start XWiki for Profiling (they require that you set a profilePath property in your settings.xml or on the command line)
  • gwt-test-manual: allow running GWT unit tests manually
  • debug: remove JS minification.
  • standalone: Builds the Rendering Standalone artifact. Note that this profile is activated automatically when performing a release.
  • quality: Runs quality checks that take a long time to execute (e.g. checks Jacoco TPC thresholds)
  • clover: A profile to use when generating Test Coverage reports with Clover. This profile skips execution of Checkstyle, CLIRR and Enforcer plugin checks to speed up the coverage generation and because of potential conflicts between tools. Note that even though we've started using Jacoco for our quality profile we still support Clover. The Clover reports are much nicer than the Jacoco ones and it's thus still useful to be able to generate them.

For example, if you use to include legacy modules and run all integration and functional tests and use Jetty/HSQLDB, you would use:

mvn clean install -Plegacy,integration-tests,hsqldb,jetty

Relationship between XWiki Enterprise and XWiki Platform

XWiki Platform is a generic platform and API for building flexible extensible wikis and web applications. XWiki Enterprise is an implementation of the platform geared toward making a wiki site for teams to share information and synchronize their efforts although it can be and has been used for a wide variety of different use cases like this website.

What does this mean to the developer?
If you wish to make changes in the core of XWiki, you are probably going to want to work on XWiki Platform. You can download and compile XWiki Enterprise and while compiling, you will notice Maven downloads a big 80MB+ file, this is the platform. Maven is smart enough to know that you don't have a copy of the platform and will download it from the XWiki Maven repository. Once it has been downloaded the first time, it is stored on your hard drive and you won't have to download it again until changes are made in the platform.

If you want to work on the core or if you have limited bandwidth, then you are well advised to get the platform sources. Then you can use the SCM's update command to get the latest copy without downloading the entire platform all over again. If you download the platform and compile it then you can compile XWiki Enterprise, Maven will use your newly compiled copy of the platform rather than downloading it, Maven is smart.

Automatic Checks

The XWiki build performs some automated checks:

  • Style checks with Checkstyle. To skip: -Dxwiki.checkstyle.skip=true.
  • Verify header licenses
  • Backward compatibility checks with CLIRR. To skip: -Dxwiki.clirr.skip=true.
  • Enforcer checks (To skip: -Dxwiki.enforcer.skip=true):
    • Verify that all plugins are versions specified
    • Verify the correct Junit artifact is used (junit-dep and not junit)
    • Verify we don't use Commons Logging or Log4j (since we use SLF4J)
    • Verify that the Java version is 7 starting with XWiki 6.0 and Java 6 before (in the release profile)
    • Verify that Javadoc exist (in the release profile, this is a Maven Central requirement)
    • In Commons:  Verify that Commons modules don't have a dependency on Rendering and Platform modules
    • In Rendering: Verify that Rendering modules don't have a dependency on Platform modules

Tips

Skipping Tests

While you should almost never do that, here's a tip for the rare occasions when you'll need to build something fast:

mvn install -Dtest=none -DfailIfNoTests=false

Alternatives are mvn install -DskipTests=true or mvn install -Dmaven.test.skip=true. Refere to the Maven documentation to understand the differences.

Troubleshooting

Dealing with Out-of-Memory Errors

If you get an OutOfMemoryError then increase the maximum heap space by setting the MAVEN_OPTS environment variable. For example for building XWiki Products - Curriki - Database one seems to need MAVEN_OPTS=-Xmx100m or higher.

If you're getting a PermGen space error you need to increase the JVM PERMGEN. For example:

MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=128m"

Error in Windows installer build: Windres error

Launch4J is using a binary called Windres which requires that the environment is NOT set to use UTF-8. If you get the following error check your LANG environment variable:

Embedded error: net.sf.launch4j.BuilderException: net.sf.launch4j.ExecException: Exec failed(1): /var/tmp/installers/windows/target/dependency/bin/windres --preprocessor=cat -J rc -O coff -F pe-i386 /tmp/launch4j10018rc /tmp/launch4j10019o

Building behind a proxy

If you are connecting to the Internet through a proxy then you need to modify your settings.xml file so that Maven knows it:

<settings>
 <proxies>
   <proxy>
     <active>true</active>
     <protocol>http</protocol>
     <host>host</host>
     <port>port</port>
     <username>uname</username>
     <password>password</password>
   </proxy>
 </proxies>
 <!--other tags-->
</settings>

Failed to execute CLIRR on xwiki-archetype-macro module

If you're using Maven 3.0.1 or 3.0.2 you might see the following build issue (reported in MNG-4973):

[ERROR] Failed to execute goal org.codehaus.mojo:clirr-maven-plugin:2.2.3:check (clirr-check) on project xwiki-archetype-macro: Error finding previous version: Failure to find org.xwiki.platform.tools:xwiki-archetype-macro:maven-archetype:1.1 in http://maven.xwiki.org/externals was cached in the local repository, resolution will not be reattempted until the update interval of xwiki-externals has elapsed or updates are forced
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.xwiki.platform.tools -DartifactId=xwiki-archetype-macro -Dversion=1.1 -Dpackaging=maven-archetype -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.xwiki.platform.tools -DartifactId=xwiki-archetype-macro -Dversion=1.1 -Dpackaging=maven-archetype -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]
[ERROR] org.xwiki.platform.tools:xwiki-archetype-macro:maven-archetype:1.1
...

It seems there's a bug in Maven 3.0.1. To work around it, use Maven 3.0. instead.

Build Tools

We have developed some build tools in the form of Maven plugins. They are located in the commons/tools and platform/xwiki-tools/ directories. Here are descriptions of some of them:

  • xwiki-packager-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and load them into a database (defined by src/main/packager/hibernate.cfg.xml). An example usage is available here.
  • xwiki-xar-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and generates a XAR (ZIP format) out of them.
  • xwiki-enterprise-archetype: A Maven Archetype to create a Maven project extending XWiki Enterprise. This is useful is you're planning to extend XWiki Enterprise and create a distribution for your product.

Building in Eclipse

Building in IntelliJ IDEA

Creating a Build for your own Modules

Generating a XAR

Follow these steps:

  • Put the XWiki XML files into the src/main/resources directory
  • Use the xar packaging in the your pom.xml.
  • Make sure the XAR packaging extension is registered in Maven.

For example:

<project...>
...
 <packaging>xar</packaging>
...
 <build>
   <extensions>
     <!-- Needed to add support for the "xar" packaging -->
     <extension>
       <groupId>org.xwiki.commons</groupId>
       <artifactId>xwiki-commons-tool-xar-handlers</artifactId>
       <version>(version to use)</version>
     </extension>
   </extensions>
 </build>
...
</project>

Make sure you run mvn xar:format to ensure your XWiki XML files are properly formatted, following XWiki XML files best practices.

Building a patched version

If you want to patch a part of XWiki without compiling the entire software, you may take a look at the following blog article.  It explains how to patch and compile only the JAR archive you want, without building all of the JAR archives of XWiki.

Tags:
Created by VincentMassol on 2007/11/26 17:16

Get Connected