How to add metadata to your pages & search it with Lucene

Last modified by Timo Jay on 2020/01/28 14:36

How to add metadata to your pages & search it with Lucene

This tutorial was written by a XWiki user and has not been officially approved yet for accuracy by the XWiki development team. Thus there may be other ways to approach the problem and other ways of implementing it.

This tutorial is about :

  • adding metadata to documents in XWiki, using standard XWiki functionnality
  • add the possibility to search the metadata, using the Lucene plugin and customizing it

If you have any question or comments, please leave a comment on this page,<BR>
 so that the whole community can benefit from it.<BR>
 If you want faster answers, you can also email the author of this document,<BR> Jean-Vivien Maurice
[email protected]

Invalid macro parameters used for the [toc] macro. Cause: [Failed to validate bean: [must be greater than or equal to 1]]. Click on this message for details.

TODO : In java snippets, replace console prints by logging - I did not have time to learn how to do that

Tools

  • XWiki Enterprise - tested with version 1.2 Milestone 2
  • the Lucene plugin for XWiki - tested with the standard plugin as included in XWiki Enterprise (inside xwiki-core-1.2-milestone-2.jar)
  • a J2EE IDE (e.g. Eclise)

Knowledge required

  • the XWiki core API and classes
  • where and how the Lucene plugin indexes wiki pages in the Lucene sources
  • the syntax for Lucene search requests (this will be briefly explained in this tutorial)
  • Velocity (as the scripting language used to write content, in the XWiki platform)
  • HTML (duh)

Context for the example

Lucene is a search engine, whose classs are stored in the lucene-core-XXX.jar. Lucene structures its index in fields and documents, which would be the respective database equivalents of columns and rows (records). The Lucene plugin for XWiki has been written by Jens Krämer. It integrates Lucene technology into XWiki :

  • indexing of XWiki documents by the Lucene engine
  • automatic extraction of relevant fields from XWiki pages to the index : title, author, etc...
  • hook on events : at creation/updates of XWiki pages, the plugin starts rebuilding the index.
  • support for operations on Lucene in Velocity : index rebuild, index search, access to search results' entries, and access to the different fields in each entry.

    So for every document found in the wiki, the Lucene plugin will add one Lucene document into the Lucene index.

    We will assume we have a system (automated, or human) which adds one metadata object to pages in XWiki. We want to be able to add this metadata information to the Lucene index, and to allow the user, through the interface, to search documents not just by free text but also against a search criteria involving our fields. The metadata object will in fact be a regular XWiki object - the kind which you can attach to XWiki pages, not Java objects :-). An XWiki object is an instance of a XWiki class. A XWiki class has to be defined in its own specific XWiki page. For the XWiki class of our metadata objects, we will use the class MetadataObject, defined in the homonym page belonging to the space MetadataSpace. You can put your own metadata definition class wherever you want, of course. Conclusion : there will be one instance (XWiki object) of the XWiki class "MetadataSpace.MetadataObject" for each XWiki document. For the sake of this example, there will be two fields in the metadata object : "field1" and "field2". +1 for originality... The strings "field1" & "field2" are their respective Lucene names in the Lucene index, as well as the names of the corresponding properties in the XWiki objects.

Procedure - synopsis

At the Java plugin level

  • modify the Lucene plugin classes/sources (and replace the current Lucene-plugin classes by the modified version)

At the XWiki administration level

  • add metadata to your wiki pages
  • enable your customized Lucene plugin (cf plugin description, in References below)
  • create & display a panel to search using Lucene

At the XWiki page creation level

  • write the panel
  • create another page to handle the request & display the results of the search

Procedure - detailled

Customize the Lucene plugin

You need to check first : does your core xwiki jar already include the classes for Lucene ? For example, for XE 1.2 Milestone they would be located at

webapps\xwiki\WEB-INF\lib\xwiki-core-1.2-milestone-2.jar\com\xpn\xwiki\plugin\

If this is the case, you should delete them from the Jar. It will be more convenient to have them in a single, separate Jar, so that you know that it is a custom plugin.

Now check that you do not already have the Lucene Jar in your webapps/xwiki/WEB-INF/lib/ directory. You should remove it from the directory (it will be replaced by your custom version). Also make sure you have all the Jars required by Lucene (Lucene-core, etc... see the plugin page).

Then you need to find the sources for the plugin, and generate the jar from the sources. See references below to get the sources. If the link provided does not work, you can ask me the sources - provided the original author's copyrights still allow sourcecode distribution by third-parties.

You want to make sure the sources work before starting to modify them :-). For example, you can import the sources in a new Eclipse Java project. You will need to add the core xwiki Jar (mentionned above) to the BuildPath. You can also use the XWiki sources. For other JAR dependancies, see Jens Kramer's website. For information, here is the list of dependancies as specified in Eclipse, in the BuildPath of my project :

<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/J2SE-1.4"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/pdfbox-0.7.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/poi-3.0-FINAL.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/poi-scratchpad-3.0.1-FINAL.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/xpp3-1.1.3.4-RC8.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-beanutils-1.7.0.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-codec-1.3.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-collections-3.2.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-dbcp-1.2.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-digester-1.6.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-fileupload-1.1.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-httpclient-3.0.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-io-1.2.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-lang-2.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-logging-1.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-net-1.4.1.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-pool-1.2.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/commons-validator-1.1.4.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/log4j-1.2.13.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/lucene-core-2.0.0.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/xwiki-core-1.2-milestone-2.jar"/>
   <classpathentry kind="lib" path="C:/Program Files/XWiki Enterprise/webapps/xwiki/WEB-INF/lib/hsqldb-1.8.0.7.jar"/>

The Lucene plugin indexes documents using a class called IndexUpdater. This class implements the XWikiActionNotificationInterface and XWikiDocChangeNotificationInterface (XWiki hook mechanism - see the notification tutorial) in order to rebuild the whole index every time users add, modify or import documents in the Wiki. For each wiki page, the class calls its own method, IndexUpdater.addToIndex(IndexData data, XWikiDocument doc, XWikiContext context). There is one detail which concerns the Lucene plugin in general : every time you want to rebuild the index, the Lucene plugin creates a lock file to get exclusive access to its index directory. In some cases, for example if the building of the index fails with an exception, the lock file will not be deleted at the end of the building process. Then it will have to be deleted manually the next time you want to rebuild the index, or the next time the index tries to automatically rebuild after changes.

So... you need to modify the method used to extract all the fields to build a document added to the index. It is called IndexData.addDataToLuceneDocument(org.apache.lucene.document.Document luceneDoc, XWikiDocument doc, XWikiContext context).

From IndexUpdater.java :

private void addToIndex(IndexData data, XWikiDocument doc, XWikiContext context)
           throws IOException
        {
           if (LOG.isDebugEnabled()) {
               LOG.debug("addToIndex: " + data);
            }
           org.apache.lucene.document.Document luceneDoc = new org.apache.lucene.document.Document();
           
           //---------------------------------------------------
           //the following call is where all the fields are added to the Lucene document object luceneDoc
           data.addDataToLuceneDocument(luceneDoc, doc, context);
           //---------------------------------------------------
           
           Field fld = null;
           // collecting all the fields for using up in search
           for (Enumeration e = luceneDoc.fields(); e.hasMoreElements();) {
               fld = (Field) e.nextElement();
               if (!fields.contains(fld.name())) {
                   fields.add(fld.name());
                }
            }
           writer.addDocument(luceneDoc);
        }

For each field, the method IndexData.addDataToLuceneDocument uses a Lucene field name which is defined as a constant in IndexFields.java. For example, in the class IndexFields you will have :

public static final String            CUSTOM_FIELD1 = "CustomField1";
       public static final String            CUSTOM_FIELD2 = "CustomField2";

Some of the already existing fields could be of interest in search queries (because they can be searched by Lucene):

/**
     * Keyword field, holds the name of the virtual wiki a document belongs to
     */

    
public static final String DOCUMENT_WIKI = "wiki";

    
/**
     * Name of the document
     */

    
public static final String DOCUMENT_NAME = "name";

    
/**
     * Name of the document
     */

    
public static final String DOCUMENT_TITLE = "title";
   
    
/**
     * Name of the web the document belongs to
     */

    
public static final String DOCUMENT_WEB = "web";

    
/**
     * Language of the document
     */

    
public static final String DOCUMENT_LANGUAGE = "lang";
   
        
/**
     * Last modifier
     */

    
public static final String DOCUMENT_AUTHOR = "author";

    
/**
     * Creator of the document
     */

    
public static final String DOCUMENT_CREATOR = "creator";

Once you have defined your custom field(s)' names in IndexFields.java, you need to edit IndexData.java.

First, you have some declarations to do in the class IndexData itself :

//XWiki object, attached to each document, where you have attached metadata to your document
    private com.xpn.xwiki.objects.BaseObject     objectCustomMetaData;

    private String                        valueCustomField1 = null;
    private String                        valueCustomField2 = null;

For each Lucene document object you add to the Lucene index, you will use the class org.apache.lucene.document.Field. One instance of this class represents one specific field in one Lucene document entry in the index. You have to add code that will handle our custom fields, to the method IndexData.addDataToLuceneDocument in IndexData.java :

//doc is the XWiki page we are currently indexing.
       //doc is instance of the class com.xpn.xwiki.doc.XWikiDocument
       objectCustomMetaData = doc.getObject("MetadataSpace.MetadataObject");
       if(objectCustomMetaData != null) {
           //System.out.println("Metadata : " + objectCustomMetaData);
           //We use a separate method to extract each field - just to ensure more readable code.
           //context represents the current context, given as a parameter to the addDataToLuceneDocument method : our own methods need a reference to the current context in order to read the metadata object.
           //context is instance of com.xpn.xwiki.XWikiContext
           valueCustomField1 = extractCustomField1(objectCustomMetaData, context);
           valueCustomField2 = extractCustomField2(objectCustomMetaData, context);
           
           //luceneDoc is the Lucene document passed as parameter to addDataToLuceneDocument, used in the Lucene index to represent the XWiki page being indexed.
           //luceneDoc is instance of org.apache.lucene.document.Document
           if ((valueCustomField1 == null) || (valueCustomField1.equals(""))) valueCustomField1 = "Undefined";
           luceneDoc.add (new Field(IndexFields.CUSTOM_FIELD1, valueCustomField1,Field.Store.YES,Field.Index.TOKENIZED));
           if ((valueCustomField2 == null) || (valueCustomField2.equals(""))) valueCustomField2 = "Undefined";
           luceneDoc.add (new Field(IndexFields.CUSTOM_FIELD2, valueCustomField2,Field.Store.YES,Field.Index.TOKENIZED));
           //System.out.println(luceneDoc.toString());
        } else {
           //System.out.println("No metadata !");
        }

Our field extraction methods, still in the class IndexData :

public String extractCustomField1(com.xpn.xwiki.objects.BaseObject meta, XWikiContext context) {
       String field1 = meta.getStringValue("field1");
       if(field1 != null) {
           //System.out.println("- field1 : " + field1);
           return field1;
        } else return "";
    }
   
   public String extractCustomField2(com.xpn.xwiki.objects.BaseObject meta, XWikiContext context) {
       String field2 = meta.getStringValue("field2");
       if(field2 != null) {
           //System.out.println("- field2 : " + field2);
           return field2;
        } else return "";
    }

Add metadata to your wiki pages

The XWiki class

We will use the XWiki class editor, accessible through the following link (standard to all XWiki Enterprise installations) : XWiki.XWikiClasses. For class creation, see the FAQ tutorial. In our example, we will create a class called MetadataClass, in the space called MetadataSpace. The class will have two properties, respectively called "field1" & "field2", of type TextArea. Write the standard class sheet, i.e. the display() method called for all properties of the object.

Attaching the metadata to pages : user input through the class sheet

You will have to attach an object of the class MetadataSpace.MetadataClass to every document, and display the Class Sheet for MetadataClass, so that the user can edit the metadata. The easiest way to do this, in MetadataClassSheet, is to propose the addition of metadata via a link in the page :

#set($metadataClass = "MetadataSpace.MetadataClass")
#set($class = $doc.getObject("$metadataClass").xWikiClass)
#set($hasProps = false)

#foreach($prop in $class.properties)
#if($velocityCount == 1)
#set($hasProps = true)
<dl>
#end
<dt> ${prop.prettyName} </dt>
<dd>$doc.display($prop.getName())</dd>
#end
#if($hasProps)
</dl>
#end

#if($hasProp == false)
#set($xredirect = $request.getRequestURL())
#set($createUrl = $doc.getURL("objectadd", "classname=$metadataClass&xredirect=${xredirect}"))
<a href="$createUrl">Add metadata to this page</a>
#end

This way, once the user has saved her page, she can choose to attach metadata herself by clicking on the link "Add metadata to this page".

In all the pages you want to add metadata to, you will have to do an includeForm of the MetadataClassSheet page where you've put the code presented above.

#includeForm("MetadataSpace.MetadataClassSheet")

For example, if you create another class, e.g. let's say DocumentClass, for your applications, when you will write the class sheet DocumentClassSheet, you will add the line above to the source code of the sheet. This way all wiki pages created from the class DocumentClass will support metadata.

Attaching the metadata to pages : automated (Java plugin)

If the metadata can be extracted from the pages automatically, or if you want to fill it with default values, you can create a Java plugin to attach metadata to the pages in your wiki. You can also modify the code below to handle metadata from this plugin according to your needs.

A plugin is a set of 2 Java classes : the plugin class itself, and the Api class. See this tutorial.

The model class for metadata : MetadataModel.java

package com.xpn.xwiki.plugin.metadata;

public class MetadataModel {
   public String field1;
   public String field2;
   
   public MetadataModel(field1,field2) {
        this.field1 = field1;
        this.field2 = field2;
   }
}

The plugin class itself : MetadataPlugin.java

package com.xpn.xwiki.plugin.metadata;

import com.xpn.xwiki.XWikiContext;
import com.xpn.xwiki.api.Api;
import com.xpn.xwiki.doc.XWikiAttachment;
import com.xpn.xwiki.doc.XWikiDocument;
import com.xpn.xwiki.plugin.XWikiDefaultPlugin;
import com.xpn.xwiki.plugin.XWikiPluginInterface;

public class MetadataPlugin extends XWikiDefaultPlugin {

    private static Log LOG = LogFactory.getLog(HelloWorldPlugin.class);

    public MetadataPlugin(String name, String className, XWikiContext context) {
   super(name,className,context);
    init(context);
    }

    public String getName() {
       return "metadataplugin";
    }

    public Api getPluginApi(XWikiPluginInterface plugin, XWikiContext context) {
       return new MetadataPluginApi((MetadataPlugin) plugin, context);
    }

    public void flushCache() {
    }

    public void init(XWikiContext context) {
       super.init(context);
    }
}

The plugin API : MetadataPluginApi.java

package com.xpn.xwiki.plugin.metadata;

import com.xpn.xwiki.XWikiContext;
import com.xpn.xwiki.api.Api;
import java.util.List;

public class MetadataPluginApi extends Api {
    private MetadataPlugin plugin;

    public MetadataPluginApi(MetadataPlugin plugin, XWikiContext context) {
       super(context);
        setPlugin(plugin);
    }

    public MetadataPlugin getPlugin(){
       return (hasProgrammingRights() ? plugin : null);
       // Uncomment for allowing unrestricted access to the plugin
       // return plugin;
    }

    public void setPlugin(MetadataPlugin plugin) {
        this.plugin = plugin;
    }
   
    public MetadataModel createMetadata(String field1, String field2) {
       return new MetadataModel(field1,field2);
    }

    public String addMetadataToDocument(String space, String docName, MetadataModel meta) {
        XWikiDocument document = context.getWiki().getDocument(space + "." + docName, this.context);
       try {
            String className = "MetadataSpace.MetadataClass";
            com.xpn.xwiki.objects.BaseObject newChild = document.newObject(className, this.context);
            newChild.set("field1", meta.field1, this.context);
            newChild.set("field2", meta.field2, this.context);
            document.addObject(className, newChild);
           try {
                this.context.getWiki().saveDocument(document,this.context);
            } catch(XWikiException xe) {
                System.out.println("addMetadataToDocument : exception ! when saving " + document.getFullName());
                e.printStackTrace();
            }
        } catch (XWikiException e) {
           // TODO Auto-generated catch block
            System.out.println("addMetadataToDocument : exception !");
            e.printStackTrace();
        }
       return "Metadata added to " + space + "." + docName;
    }
   
    public String addMetadataToSpace(String space, MetadataModel meta) {
       if(space.equals("*")) {
            String returnString = "Metadata added to all pages of spaces :\n";
            List spaceList = this.context.getWiki().getSpaces();
            Iterator spaceIterator = spaceList.iterator();
           while(spaceIterator.hasNext()) {
                String spaceName = (String)spaceIterator.next();
                List docList = this.context.getWiki().getSpaceDocsName(space, this.context);
                Iterator docIterator=docList.iterator();
               while (docIterator.hasNext()){
                    String docName=(String)docIterator.next();
                    addMetadataToDocument(spaceName,docName,meta);
                }
                returnString += "- " + spaceName + "\n";
            }
           return returnString;
        } else {
            List docList = this.context.getWiki().getSpaceDocsName(space, this.context);
            Iterator docIterator=docList.iterator();
           while (docIterator.hasNext()){
                String docName=(String)docIterator.next();
                addMetadataToDocument(space,docName,meta);
            }
           return "Metadata added to all pages of space : " + space + ".";
        }
    }
}

To use your plugin, you could have a page with the following code :

#set($metadataplugin = $xwiki.getPlugin("metadataplugin"))
#if($metadataplugin)
   <form action="$request.getRequestURL()" method="post">
       <div>
           *field1* :
           <BR>
           <input type="text" name="field1" value="" />
           <BR>
           *field2* :
           <input type="text" name="field2" value="" />
           <BR>
           *Space (enter a * to treat all spaces)* :
           <input type="text" name="space" value="" />
           <input type="submit" value="Add metadata to space"/>
       </div>
   </form>
   <BR>
    #if($request.getParameter("space"))
        #set($meta = $metadataplugin.createMetadata($request.getParameter("field1"),$request.getParameter("field2")))
        $metadataplugin.addMetadataToSpace($request.getParameter("space"),$meta)
    #end
#end

The search panel

The search panel's code should be only one line : a template Velocity call to a script you have stored in the servlet container's filesystem, in your skin's directory : skins/yourskin. In a standard-installer setup on a windows machine :

C:\Program Files\XWiki Enterprise\webapps\xwiki\skins\yourskin\sources\metadatasearch\

This way you will not lose the development work done writing the search panel. The same applies to the paragraphs below, "The search result page" and "The search table". See the "template" macro reference.

#template("sources/metadatasearch/metadatasearchform.vm")

And in sources/metadatasearch/metadatasearchform.vm :

##replace the argument given to $xwiki.getPlugin(), with your Lucene-plugin name
##(i.e. the name returned by getName() in the plugin class)
#set($lucene = $xwiki.getPlugin("lucene"))
#if($lucene)
   <form action="/xwiki/bin/view/MetadataSpace/MetadataSearchPage" method="post">
       <div>
           *field1* :
           <BR>
           <input type="text" name="field1" value="" />
           <BR>
           *field2* :
           <input type="text" name="field2" value="" />
           <BR>
           *Free-text search* :
           <input type="text" name="textual" value="" />
           <input type="submit" value="Search"/>
       </div>
   </form>
#end

The search result page

The search form, in the search panel, was referring to the seach result page.

<form action="/xwiki/bin/view/MetadataSpace/MetadataSearchPage" method="post">

In MetadataSpace.MetadataSearchPage :

#template("sources/metadatasearch/metadatasearchpage.vm")

In sources/metadatasearch/metadatasearchpage.vm :

#set($field1 = $request.getParameter("field1"))
#set($field2 = $request.getParameter("field2"))
#set($textual = $request.getParameter("textual"))

#*
We check first that the user is not trying to access another page of a previous search
*#
#if($request.getParameter("query"))
    #set($query = $request.getParameter("query"))
#else
    #set($query ="")
#end
#*
Then we build the Lucene query string from the text inputs
*#
#if($query=="")
    #if($field1 !="")
        ##Adding a condition on the Lucene-index field "field1"
        #if($query !="")
            #set($query ="$query AND field1:$field1" )
        #else
            #set($query ="field1:$field1")
        #end
    #end
    #if($field2 !="")
        ##Adding a condition on the Lucene-index field "field2"
        #if($query !="")
            #set($query ="$query AND field2:$field2" )
        #else
            #set($query ="field2:$field2")
        #end
    #end
    #if($textual !="")
        ##Adding a condition on the Lucene-index field "ft"
        ##This field is already handled by the standard Lucene-plugin for XWiki
        ##The field contains an XWiki page's full text
        #if($query !="")
            #set($query ="$query AND ft:$textual" )
        #else
            #set($query ="ft:$textual")
        #end
    #end
#end

#set($itemsPerPage = "30")

1.1.1.1 Search results

#if($query != "")
   This is the query passed to the Lucene search engine : $query
   
    ##replace the argument given to $xwiki.getPlugin(), with your Lucene-plugin name
    ##(i.e. the name returned by getName() in the plugin class)
    #set($lucene = $xwiki.getPlugin("lucene"))
    #if($lucene)
       Using plugin : $lucene.getPlugin().getName()
       
                ## ---------------
                ## Lucene search
                ## ---------------              
        #set($wikinames = "xwiki")
        #set($languages = "default,en,de")
        #set($firstIndex = $request.getParameter("firstIndex"))
        #if(!$firstIndex)
            #set($firstIndex = "1")
        #end
        #set($searchresults = $lucene.getSearchResults($query, $wikinames, $languages, $xwiki))
        #set($results = $searchresults.getResults($firstIndex,$itemsPerPage))
       There are $searchresults.getHitcount() item(s) found.
        #if($searchresults.getHitcount()>0)
                        ## -----------------
                        ## Results numbers
                        ## -----------------
            #set($lastIndex=$searchresults.getEndIndex($firstIndex, $itemsPerPage))
            #if($searchresults.getHitcount()==1)
               One result:
            #else
               Results $firstIndex - $lastIndex of ${searchresults.getHitcount()}:
            #end
                        ## ---------------
                        ## Previous page
                        ## ---------------
            #if($searchresults.hasPrevious($firstIndex))
                #set($linkfirstIndex = $searchresults.getPreviousIndex($firstIndex,$itemsPerPage))
                #set($link = "${doc.name}?query=${query}&firstIndex=${linkfirstIndex}")
               <a href="$link"><img src="/xwiki/skins/yourskin/icons/search/previous.png" alt="previous" />previous page</a>
            #end        
                        ## -------------
                        ## Next page
                        ## -------------
            #if($searchresults.hasNext($firstIndex,$itemsPerPage))
                #set($linkfirstIndex = $searchresults.getNextIndex($firstIndex,$itemsPerPage))
                #set($link = "${doc.name}?query=${query}&firstIndex=${linkfirstIndex}")    
               <a href="$link"><img src="/xwiki/skins/yourskin/icons/search/next.png" alt="next" />next page</a>
            #end

                        ## -----------------
                        ## Display results
                        ## -----------------
                        #set ($list = $results)
                        #set ($isScored = true)
                        ## Two ways of doing it here :
                        ## either with includeInContext() or with template()
                       
                        ## #includeInContext("MetadataSpace.MetadataSearchResults")
                        ## if you have placed the search results Velocity script
                        ## in another XWiki page instead of storing it in a file.
                        ## Then you can pick up the $list variable from here,
                        ## inside the code of the included page MetadataSpace.MetadataSearchResults
                       
                        #template("sources/metadatasearch/metadatasearchresults.vm")
                #end
    #else
        #error("Lucene plugin not found. Make sure it's defined in your xwiki.cfg file.")
    #end
#end

##replace the argument given to $xwiki.getPlugin(), with your Lucene-plugin name
##(i.e. the name returned by getName() in the plugin class)
#set($lucene = $xwiki.getPlugin("lucene"))

#if($lucene)
#set($doRebuild = $request.getParameter("rebuild"))
#if($doRebuild)
  #if($doRebuild=="yes")
        #set($documentCount = $lucene.rebuildIndex($xwiki,$context))
        #if(${documentCount}>=0)
            #info("Started index rebuild with $documentCount documents.\\
                              Will take some time depending on the number of pages/attachments.")
        #else
            #error("Index rebuild failed.")
        #end
  #end
#else
    #if($xwiki.hasAdminRights())
        #info("[Rebuild the Lucene index>${doc.web}.${doc.name}?rebuild=yes]")
    #end
#end
#end

We have included a page which will display the table of search results. There will be extra columns for our metadata.

## Two ways of doing it here :
                        ## either with includeInContext() or with template()
                       
                        ## #includeInContext("YourWeb.MetadataSearchResults")
                        ## if you have placed the search results Velocity script
                        ## in another XWiki page instead of storing it in a file.
                        ## Then you can pick up the $list variable from here,
                        ## inside the code of the included page MetadataSpace.MetadataSearchResults
                       
                        #template("sources/metadatasearch/metadatasearchresults.vm")

The search table

We are now going to display a table listing search results, as in the normal Lucene scripts. Each line represents an entry pointing to a page in your wiki. The columns show data about your page. So we need to add columns to display our metada for each result entry.

It also means that we have to access the metadata corresponding to each search entry. In fact, a search entry stores the Lucene fields associated to a Lucene document. So we just ask every search entry its fields field1 and field2 in the Lucene index.

The search results, as returned by Lucene, are an instance of the class com.xpn.xwiki.plugin.lucene.SearchResults. This class is a special class handling the result entries for our search request. Each result entry is an instance of the class com.xpn.xwiki.plugin.lucene.SearchResult.

From the sources/metadatasearch/metadatasearchpage.vm script :

#set($searchresults = $lucene.getSearchResults($query, $wikinames, $languages, $xwiki))
## $searchresults is an instance of com.xpn.xwiki.plugin.lucene.SearchResults

#set($results = $searchresults.getResults($firstIndex,$itemsPerPage))
## $results is a classical container object, instance of java.util.List

There are $searchresults.getHitcount() item(s) found.

The List container object contains search entries - instances of the com.xpn.xwiki.plugin.lucene.SearchResult class.

You will have to modify the com.xpn.xwiki.plugin.lucene.SearchResult class.

In SearchResult.java you would add private properties to the class SearchResult :

private String field1;
private String field2;

You would also add the following methods to the same class :

public String getField1() {
       return field1;
}

public String getField2() {
       return field2;
}

These 2 methods will be called by the Velocity script in the search table.

Now the properties we added need to be set. This is why we also modify the SearchResult constructor by adding the following initializations :

//SearchResult(org.apache.lucene.document.Document doc, float score,
// com.xpn.xwiki.api.XWiki xwiki)
this.field1 = doc.get(IndexFields.CUSTOM_FIELD1);
this.field2 = doc.get(IndexFields.CUSTOM_FIELD2);

After that we will write the script that displays the search table. It is sources/metadatasearch/metadatasearchresults.vm :

#set($showdata = 0)
#set($formatDate = "yyyy MMMM dd, HH:mm")

## WARNING: Do not add any empty line inside the table element. This will potentially break
## the Javascript we're using for filtering/sorting columns. It might work in FF but will break
## in other browsers like IE. This is because empty lines add <p class="paragraph"></p> elements
## when rendered.

<table id="searchTableUnique" class="grid sortable doOddEven" style="font-size: 80%">
 <tr class="sortHeader">
   <th>Page</th>
   #*<th style="width:150px" class="selectFilter">Space</th>*#
   <th style="width:150px">Date</th>
   <th style="width:150px">Last Author</th>
   #if($isScored)
   <th style="width:50px">Score</th>
   #end
   <th style="width:70px">Field1</th>
   <th style="width:70px">Field2</th>
   #*
   #if($xwiki.hasAdminRights())
     <th style="width:210px" class="unsortable noFilter">Actions</th>
   #end
   *#
 </tr>
 #foreach ($item in $list)
   #set($oneNewComment = "no")
     #if ($xwiki.hasAccessLevel("view", $context.user, "${context.database}:${item}"))
       #if ($item.class == "class java.lang.String")
         #set($bentrydoc = $xwiki.getDocument($item))
       #elseif ($item.class == "class com.xpn.xwiki.plugin.lucene.SearchResult")          
         #set($bentrydoc = $item)
       #end
       #set($cclass = $xwiki.getDocument("XWiki.XWikiComments").getxWikiClass())
       #set($comment = $cclass.newObject())
       #if($xwiki.getWebPreferenceAsInt("commentsorder",1)==0)
         #set($comments = $bentrydoc.getComments())
       #else
         #set($comments = $bentrydoc.getComments(false))
       #end
       #set($creator = $xwiki.getUserName($bentrydoc.author))
       #set($ptitle = $bentrydoc.getDisplayTitle())
       <tr><td style="text-align:left">          
         ## LUCENE : entries are typed
         #if ($bentrydoc.type)
           #if ($bentrydoc.type == "attachment")
           <a href="${bentrydoc.url}" target="_blank"><img src="${xwiki.getSkinFile("icons/search/disk.png")}" ALT="download" /> ${bentrydoc.filename}</a>\\
        Attachment of
           #end
         #end
         #if($bentrydoc.M3Type == "Folder")
           #set($linkWithMenu = "<a href='/xwiki/bin/view/${bentrydoc.web}/${bentrydoc.name}' context='FolderSearchResult' menuRemove='/xwiki/bin/delete/${bentrydoc.web}/${bentrydoc.name}' menuRename='/xwiki/bin/view/${bentrydoc.web}/${bentrydoc.name}?xpage=rename'>${bentrydoc.title}</a>")
         #else
           #set($linkWithMenu = "<a href='/xwiki/bin/view/${bentrydoc.web}/${bentrydoc.name}' context='doc' menuEdit='/xwiki/bin/inline/${bentrydoc.web}/${bentrydoc.name}' menuRemove='/xwiki/bin/delete/${bentrydoc.web}/${bentrydoc.name}' menuRename='/xwiki/bin/view/${bentrydoc.web}/${bentrydoc.name}?xpage=rename'>${bentrydoc.title}</a>")
         #end
         #if($comments.size()>0)  
           #set($i = 0)  
           #set($cobj = $comments.get($i))  
           #set($comment = $bentrydoc.display("comment", "view", $cobj))  
           #set($date = $cobj.getXWikiObject().get("date").value)
           #if($date)
             #set($date2 = $!xwiki.formatDate($date,"yyyy MM dd HH:mm:ss")  )
           #end
           #if($bentrydoc)
             #set($date1 = $!xwiki.formatDate($!bentrydoc.date,"yyyy MM dd HH:mm:ss") )
           #end
           #if($date1.equals($date2) )
             ##[$ptitle>${bentrydoc.web}.$bentrydoc.name] <em>- 1 new comment</em>
             $linkWithMenu <em>- 1 new comment</em>
             #set($oneNewComment ="yes")
             #set($desc = $cobj.getXWikiObject().get("comment").value)
           #else
             ##[$bentrydoc.title>${bentrydoc.web}.$bentrydoc.name] #if ($ptitle != $bentrydoc.name) <em>- $ptitle</em>#end
             $linkWithMenu #if ($ptitle != $bentrydoc.name) <em>- $ptitle</em>#end
           #end
         #else  
           #set($comment = "")  
           $linkWithMenu
           ##[$bentrydoc.title>${bentrydoc.web}.$bentrydoc.name.replaceAll("@","%40")] #if ($ptitle != $bentrydoc.name) <em>- $ptitle</em>#end
         #end   
       </td>#*<td style="text-align:left">
          [$bentrydoc.web>${bentrydoc.web}.WebHome]
       </td>*#<td style="text-align:left">          
         $xwiki.formatDate($bentrydoc.date,"yyyy MMM dd") at $xwiki.formatDate($bentrydoc.date,"HH:mm")</td><td style="text-align:center">
         #if($oneNewComment =="yes")
           #set($creator = $xwiki.getUserName($cobj.author)   )
         #end
         #if ($creator == "XWikiGuest")
           Guest
         #else
           $creator
         #end
       </td>
       #if ($isScored)
       <td style="text-align:left">
         #set($resval=$bentrydoc.score*100)
         #set($starurl=$xwiki.getSkinFile("icons/search/star.png"))
         #set($star = "<img src='$starurl' alt='$resval' />")
         <span class="hidden">$bentrydoc.score</span>
         #*
         #if($resval>10) $star #end
         #if($resval>20) $star #end
         #if($resval>40) $star #end
         #if($resval>60) $star #end
         #if($resval>90) $star #end
         *#
         #set ($perc = $resval.toString())          
         ${perc.substring(0, $perc.indexOf("."))}%        
       </td>
       <td style="text-align:left">
           $bentrydoc.getField1().replace("Undefined","-")
       </td>
       <td style="text-align:left">
           $bentrydoc.getField2().replace("Undefined","-")
       </td>
         #set ($bentrydoc = $xwiki.getDocument("${bentrydoc.web}.${bentrydoc.name}"))
       #end
       #*
       #if($xwiki.hasAdminRights())
         <td>
           <a href="$xwiki.getURL("XWiki.CopyDocument", "view", "sourcedoc=${bentrydoc.fullName}")">Copy</a> - <a href="$bentrydoc.getURL("delete")">Delete</a> - <a href="$bentrydoc.getURL("view", "xpage=rename&amp;step=1")">Rename</a> - <a href="$bentrydoc.getURL("edit", "editor=rights")">Rights</a>
         </td>
       #end
       *#
     </tr>
   #end
 #end
</table>

Conclusion

Hopefully you have now learnt :

  • how to add support for metadata attached to wiki pages
  • how to automate adding of default metadata (or automatically computed metadata, since you can easily customize the meatadataplugin) for existing documents
  • how to integrate your metadata into Lucene
  • how to make the metadata "searchable" with Lucene

    The inspiration for this tutorial was a real world project, wherein one of the requirements was to have metadata that would be searchable through the web interface, and through Lucene behind the scenes. So the material presented here could be helpful to the whole XWiki community.

References

Tags:
   

Get Connected