Building XWiki from sources

Contents

Make sure you install Maven 2.2.0 or greater. It might build fine using earlier versions but we're not supporting them and all the XWiki committers and our CI server are running 2.2.0 and greater.

Note that we currently have a build issue with Maven 2.2.1 and JDK 1.6 which leads to the following build error:
org.apache.xerces.impl.dv.DVFactoryException: DTD factory class. It works fine with Maven 3.x and JDK 1.6 though.

Understanding the directory structure

Before you start it's good to have some minimal understanding of how Maven2 works. You probably won't need it if everything goes fine but you'll need that knowledge if something breaks! ;-)

The first thing to understand is the directory structure used.

Checking out the sources

Use your favorite Subversion client to check out the sources.

You can check out the whole source tree or only a single module you wish to build. Maven2 is powerful enough in that it'll always try to download the required dependencies from the remote repositories you have defined, so that you don't have to build the whole project from sources.

Installing Maven

  • Install Maven 2.1.0-M1 or greater. Create an M2_HOME environment variable pointing to where you have installed Maven. Add M2_HOME/bin to your PATH environment variable.
    For more information on how to install Maven or how to use it check this excellent Maven book.
  • Make sure you give Maven's JVM enough memory. XWiki's build uses Aspects and GWT compilers which need lots of memory. A good value is to give the JVM 600MB. To do that set an environment variable named MAVEN_OPTS. For example on Unix add the following to your shell startup script:
    MAVEN_OPTS=-Xmx500m -XX:MaxPermSize=256m
  • Create a ~/.m2/settings.xml file with the XWiki custom remote repository defined as shown below. If you are on Windows, create the directory .m2 in your home directory, e.g. C:\Documents and Settings\Administrator\.m2. If you cannot do this from Windows Explorer, open a commandline and use md .m2.
    <settings>
     <profiles>
       <profile>
         <id>xwiki</id>
         <repositories>
           <repository>
             <id>xwiki-externals</id>
             <name>XWiki Maven2 Remote Repository for Third Party Dependencies</name>
             <url>http://maven.xwiki.org/externals</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </repository>
           <repository>
             <id>xwiki-releases</id>
             <name>XWiki Maven2 Remote Repository for Releases</name>
             <url>http://maven.xwiki.org/releases</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </repository>
           <repository>
             <id>xwiki-snapshots</id>
             <name>XWiki Maven2 Remote Repository for Snapshots</name>
             <url>http://maven.xwiki.org/snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </repository>
         </repositories>
         <pluginRepositories>
           <pluginRepository>
             <id>xwiki-plugins-externals</id>
             <name>XWiki Maven2 Plugin Remote Repository for Third Party Plugins</name>
             <url>http://maven.xwiki.org/externals</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </pluginRepository>
           <pluginRepository>
             <id>xwiki-plugins-releases</id>
             <name>XWiki Maven2 Plugin Remote Repository for Releases</name>
             <url>http://maven.xwiki.org/releases</url>
             <releases>
               <enabled>true</enabled>
             </releases>
             <snapshots>
               <enabled>false</enabled>
             </snapshots>
           </pluginRepository>
           <pluginRepository>
             <id>xwiki-plugins-snapshots</id>
             <name>XWiki Maven2 Plugin Remote Repository for Snapshots</name>
             <url>http://maven.xwiki.org/snapshots</url>
             <releases>
               <enabled>false</enabled>
             </releases>
             <snapshots>
               <enabled>true</enabled>
             </snapshots>
           </pluginRepository>
         </pluginRepositories>
       </profile>
     </profiles>
     <activeProfiles>
       <activeProfile>xwiki</activeProfile>
     </activeProfiles>
    </settings>
    If you have a fresh maven setup, you can download the above xml and place it in your .m2 directory.
    Depending on your geographical location, you should use closer maven repositories to speed up your build process. For more information refer Maven Mirror Settings Guide.

Building with Maven

Projects are directories found under the XWiki SVN root. For example: platform, enterprise, manager, etc.

To build them, check them out and run mvn install.

Each project is made of modules. Thanks to XWiki's Continuous Integration setup you can check out and build any single module (and its children) you wish without having to rebuild the other modules from source. Maven will pick the latest version of your module's dependencies from XWiki's Maven remote repositories. Of course if you have uncommitted changes in a dependent module, you'll want to build that module before. The build result is placed in two places:

  • the local maven repository, for making the module available to other projects or modules using maven (even other XWiki modules take the needed libraries from this repository, and not directly from a "neighbor" directory)
  • and in a subdirectory of that module, called target.
Now some of the modules built by XWiki are very large (in the 40MB+ size) and sometimes you don't absolutely need the latest version to get downloaded as it'll take too long. In that case here's a tip: run Maven in 'no-snapshot-update' mode with mvn install -nsu.
You might encounter an java.lang.OutOfMemoryError during aspectj:compile. You must increase the maximum memory used by maven, by setting MAVEN_OPTS=-Xmx512m (or a different amount of memory).

You'll find below some common build tasks.

Building for a specific database

By default XWiki's Maven build will use Hypersonic SQL as the default database. If you wish to execute the build using a different database, you'll need to specify the correct database profile (-P<profile name>). Right now we support 4 profiles:

  • hsqldb: Hypersonic SQL. This is the default
  • mysql: MySQL
  • pgsql: PostGreSQL
  • derby: Apache Derby

For example to build a module for MySQL type mvn install -Pmysql in that module's directory.

Hypersonic and Derby are embedded database so you won't need anything setup to execute the build for them. However for MySQL and PostGreSQL you'll need to have the database installed, set up and running before you can run the build with their profiles. 

A simple way to do this is to define the necessary properties within {HOME}/.m2/settings.xml as explained above when configuring the maven mirrors. Check the property names required by looking at the corresponding profiles section in the platform's top level pom.xml file

Building on Windows

To build on Windows, you should specify the windows profile (-Pwindows).

Building XWiki Enterprise

Go in enterprise/ and type mvn install. It'll build it with a default config for HSQL DB. After the build is finished you'll find the standalone zip in enterprise/distribution/jetty/hsqldb/target/.

If you wish to build all the modules required for XWiki Enterprise and not the ones required by other products you can go at the top level and type mvn install"
The functional tests are not executed by default as the take a long time to execute and are meant mostly to be executed by the Continuous Integration tool. However if you wish to run them you can execute mvn install -Pxe,ci
The installer builds are not executed by default (they are executed on the Continuous Integration server though). If you wish to run them you can execute mvn install -Pxe,ci or better, go in xwiki-product-enterprise/installers and run mvn install.

The Relationship between XWiki Enterprise and XWiki Platform

XWiki Platform is a generic platform and API for building flexible extensible wikis and web applications. XWiki Enterprise is an implementation of the platform geared toward making a wiki site for teams to share information and synchronize their efforts although it can be and has been used for a wide variety of different use cases like this website.

What does this mean to the developer?
If you wish to make changes in the core of XWiki, you are probably going to want to work on XWiki Platform. You can download and compile XWiki Enterprise and while compiling, you will notice Maven downloads a big ~80MB file, this is the platform. Maven is smart enough to know that you don't have a copy of the platform and will download it from the XWiki Maven repository. Once it has been downloaded the first time, it is stored on your harddrive and you won't have to download it again until changes are made in the platform.

If you want to work on the core or if you have limited bandwidth, then you are well advised to download the platform. Then you can use the svn up command to get the latest copy without downloading the entire platform all over again. If you download the platform and compile it then you can compile XWiki Enterprise, Maven will use your newly compiled copy of the platform rather than downloading it, Maven is smart.

Building XWiki Platform

The platform is built much the same way as XWiki Enterprise but it is a large project and compiling can take a long time (over 1/2 hour.)
One of the reasons building the platform is slow is because the WYSIWYG module builds a version for each supported browser in each language. You can make it only build for English/Firefox by specifying -Pdev in the build command.
mvn install -Pdev
You can also build the platform faster if you skip the unit tests by building with the command:
mvn install -Dtest=none -DfailIfNoTests=false
Caution: If you have just made a patch, you should probably run the tests since breaking a test is embarrassing.
For added safety (especially when you have made changes which might affect the ability of other code to find the libraries it needs) you can compile with:
mvn clean install
and everything will be recompiled.
Press enter, go get a cup of coffee, and wait.

Executing the Automated Functional Tests for XWiki Enterprise

Go in enterprise/distribution-test and type mvn install. By default they are executed in Firefox. To run them in a different browser, use a profile. For example to run them in IE, type mvn install -Piexplore.

See the Testing page for more details.

Executing the Standard Web WAR quickly in development mode

Update 2010-03-07: The part below is a bit oldish and is probably not working any more. It needs someone to look at it. Please check the alternative below.

Go in platform/web/standard and type mvn install -Pjettyrun. This will execute it on Jetty with a HSQLDB database located in platform/web/standard/database (we need to improve the build to locate it elsewhere). So if you want to have any content displayed you'll need to put a HSQLDB there or simply import a XAR.

If you wish to use another database like MySQL, run mvn install -Pjettyrun,mysql. See the section above on building XWiki for a specific database for more information.

Once the build has started Jetty, open your browser and point it to http://localhost:8080/xwiki.

To debug in this mode, run MAVEN_OPTS='-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=5005,server=y,suspend=n' mvn install -Pjettyrun and attach you IDE to this process as in Debugging.

Note that this is a quick way of starting XWiki in development mode. However the real test is to start the XWiki Enterprise standalone distribution to ensure that everything works. This is achieved by running mvn install in enterprise/.

Here are some alternative strategies used by XWiki committers:

Building XEclipse

You'll need Eclipse 3.3.x (or above) and an associated PDE (Plugin Development Environment) in order to build XEclipse from source.

This section is about building XEclipse with maven. For instructions on how to build XEclipse from right within the Eclipse IDE, you may refer to Building in Eclipse

Building XEclipse as an Eclipse plugin

Change directory to plugins/org.xwiki.eclipse and run mvn install -DeclipseInstall=$ECLIPSE_DIR

Builing XEclipse as a standalone application

You will need the pde-maven-plugin.
Once this maven plugin is installed, please refer to plugins/org.xwiki.eclipse.rcp/README for further information about building instructions.

Building the WYSIWYG editor

Version 2.2.x and older

The WYSIWYG editor's code is split in three modules, all found in platform/web directory:

  • xwiki-gwt-dom (DOM API, Selection and Range API)
  • xwiki-gwt-user (Widgets including the RichTextArea)
  • wysiwyg (The editor code, including plugins)

The wysiwyg module depends on the xwiki-gwt-dom and xwiki-gwt-user modules. The xwiki-gwt-user module depends on xwiki-gwt-dom module. After you build a module you have to also build the modules that depend on it for your changes to take effect. To build one of the modules you just have to execute mvn clean install in the corresponding directory. xwiki-gwt-dom and xwiki-gwt-user modules generate jars (so no JavaScript). This jars are required at GWT compile time. The wysiwyg module generates a war that is merged with the war build from platform/web/standard. The WYSIWYG war contains mainly the GWT-compiled JavaScript code and the server side jar.

By default, when you build the wysiwyg module the GWT compiler generates compilation units (permutations) for each major browser and for each supported locales. Because of this the build takes a while and requires a lot of resources (both memory and processor). We cannot afford this during developemnt so we build the editor just for one browser and one locale at a time. We achive this by using the -Pdev maven profile and by specifying the browser and the locale in the src/main/resources/com/xpn/xwiki/wysiwyg/WysiwygDev.gwt.xml module descriptor that is used for developent. This module descriptor inherits Wysiwyg.gwt.xml which is used by default if the -Pdev profile is not activated. If you build the wysiwyg module using the -Pdev profile and you don't change the WysiwygDev.gwt.xml the GWT compiler generates only one permutation for Firefox + English locale.

The JavaScript code is obfuscated by default. Using the -Pdev profile the JavaScript code is kept detailed (for debugging) but its size (5MB at some point, for one permutation) makes it unusable in production. You can control the style of the generated JavaScript code using the gwt.style pom property.

To update the editor of an XWiki Enterprise instance you have to update the resources/js/xwiki/wysiwyg/xwe directory and the WEB-INF/lib/xwiki-web-wysiwyg.jar with the ones from the war generated by the wysiwyg module build. Note that the server side code of the WYSIWYG editor depends on the XWiki platform core and thus the xwiki-web-wysiwyg and xwiki-core jars should have the same version.

Version 2.3.x and newer

Starting with version 2.3 the wysiwyg module was split in two:

  • xwiki-gwt-wysiwyg-client
  • xwiki-gwt-wysiwyg-server

The client side module generates two jars. One is required at GWT compile time and the other, containing classes and interfaces shared between the client side and the server side code, is required at runtime. The server side module generates a war which contains mainly the GWT-compiled JavaScript code and two jars: the shared classes and the server side Java code.

As it used to happen with the wysiwyg module prior to version 2.3, when building the xwiki-gwt-wysiwyg-server module the GWT compiler generates by default all the permutations (browser x locale) and thus requires a lot of resources. You can use the -Pdev maven profile to generate just one permutation, specified in the WysiwygDev.gwt.xml module descriptor. Note that this file is now located in the xwiki-gwt-wysiwyg-client module. As a result, if you want to change the default development permutation (Firefox + English) you have to first build the client side module and then the server side module using the -Pdev profile.

Running unit tests in browser

Beginning with version 2.2 unit tests are run by default using HtmlUnit (GWT >=2.0) which tries to emulate the browser. Some of the JavaScript API is still not supported (e.g. Selection API is partially implemented) and thus we are forced to disable some of the tests in HtmlUnit mode. Fortunately we can run all the tests in a real browser using the  -Pmanual maven profile. When this profile is active the build stops at the integration-test lifecycle phase and waits for you to point your browser to a given URL. Tests can run in both Java and JavaScript code. You can tweak the test parameters by editing the pom.

Troubleshooting

Dealing with Out-of-Memory Errors

If you get an OutOfMemoryError then increase the maximum heap space by setting the MAVEN_OPTS environment variable. For example for building XWiki Products - Curriki - Database one seems to need MAVEN_OPTS=-Xmx100m or higher.

If you're getting a PermGen space error you need to increase the JVM PERMGEN. For example:
MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=128m"

Error in Windows installer build: Windres error

Launch4J is using a binary called Windres which requires that the environment is NOT set to use UTF-8. If you get the following error check your LANG environment variable:

Embedded error: net.sf.launch4j.BuilderException: net.sf.launch4j.ExecException: Exec failed(1): /var/tmp/installers/windows/target/dependency/bin/windres --preprocessor=cat -J rc -O coff -F pe-i386 /tmp/launch4j10018rc /tmp/launch4j10019o

Building behind a proxy

If you are connecting to Internet through a proxy then you need to modify your settings.xml file so that Maven knows it.

<settings>
 <proxies>
   <proxy>
     <active>true</active>
     <protocol>http</protocol>
     <host>host</host>
     <port>port</port>
     <username>uname</username>
     <password>password</password>
   </proxy>
 </proxies>
 <!--other tags-->
</settings>

Failure to build xwiki-rendering-macro-groovy module

If you're on Mac and want to run XWiki Enterprise on the JDK 1.5 but have JDK 1.6 installed then the Groovy macro in the new rendering won't build fine and you'll get some exceptions. This is because of a Mac problem: When Java 1.6.0 is installed in OSX 10.5 it overwrites /System/Library/Java/Extensions/AppleScriptEngine.jar with a 1.6 specific one. When the ScriptEngineManager gets initialized if finds it and tries to use it. However it only works on JDK 1.6. Moving both AppleScriptEngine.jar and libAppleScriptEngine.jnilib somewhere else fixes the problem.

Build Tools

We have developed some build tools in the form of Maven plugins. They are located in the platform/xwiki-tools/ directory. Here are descriptions of some of them:

  • xwiki-packager-plugin: Takes XWiki pages defined as XML files and located in src/main/resources/ and load them into a database (defined by src/main/packager/hibernate.cfg.xml). An example usage is available here.

Building in Eclipse

Building in IntelliJ IDEA

Tags:
Created by VincentMassol on 2007/11/26 17:16
Last modified by Eduard Moraru on 2010/06/15 14:33
This wiki is licensed under a Creative Commons license
2.3.28624