New Rendering Architecture

Design: New Rendering Architecture

The need

• We need to refactor our parsing and rendering. Actually we don't have any parsing layer proper so it's extremely hard to perform manipulations on the wiki syntax. For example for the rename feature we have had to write an ad hoc Wiki link parser. We can imagine that in the future we'll need to perform more refactorings like moving to one syntax to another, replacing deprecated calls with new APIs, etc. Thus we need a proper and strong wiki parser.
• In term of rendering we need to plug-in several renderers (as we already do). However our API needs to be refactored and strengthened.
• We want to support several syntaxes
• Radeox is old and not maintained. We have copy-pasted a good deal of its code into XWiki.

Architecture Decisions

A1 - A XWiki API

We want to have a XWiki Rendering API so that all code inside XWiki that need some rendering uses this API. We shouldn't use an external API since:

• This is a core component that we want to fully control
• External APIs are hard to change to fit our needs
• External APIs change and we need some isolation from those changes
• New external APIs are created and we want it to be easy for us to use a new framework

This API will be a non-streaming API meaning that each document will be parsed and represented in-memory. Although this takes more memory and doesn't allow to have documents with 100s of MB of text, this is needed for several reasons:

• We want to cache the parsed documents for improved performances
• Macros need to execute in different orders. For example the TOC Macro must execute last so that it counts the sections (which could have been generated by other macros)
• Macros need to be able to modify existing elements anywhere in the document

A2 - Add a wiki syntax metadata to XWikiDocument

We need to be able to say, on a per document basis, in what syntax the document is written. This is needed for:

• Ability to migrate an existing wiki to a new syntax (say from the current syntax to the new syntax)
• Ability to offer different syntaxes to users (Confluence syntax, Mediawiki syntax, etc)

Notes:

• We should be able to give a default syntax used in a space or wiki for docs that don't have a syntax specified. And a default syntax to generate for the WYSIWYG editor or the wiki editor.

A3 - We store text format in the DB

We store the content of the document as seen by the user and thus in the syntax the document is written in. Reasons:

• Less change from what currently exists
• Easy to read DB format

A4 - New Macro Syntax and new Macro implementation

The new macro syntax is similar to the old one but with the important difference that they are always closed.

Examples:

{macroname:param1=value1|param2=value2/}

{macroname:param1=value1|param2=value2}
macro content
{/macro}

Notes:

• All current Radeox macros need to be reimplemented.
• Some current Velocity macros would also need to be reimplemented using the new Macro mechanism. Such as the TOC macro.
• Some macros need to execute after others. For example the TOC macro needs to be executed last so that all other macro that generate sections have already been executed when it executes. Thus in the implementation we need to specify an optional order. When not defined the macros execute as they are defined in the doc.
• Macros are passed a DOM in their execute methods but the DOM can also be serialized so that Macros can get access to the content as text should they require it.

A5 - Macros output Listener events

This is very important and allows the Macros to be portable across different renderers.

A6 - Introduce a Velocity macro

Replace the current Velocity pre-processing by a Velocity macro. This means writing any velocity script inside a {velocity} macro. For example:

{velocity}
<div>
  <h1>Hello $customer.Name!</h1>
  <table>
    #foreach( $mud in $mudsOnSpecial ) 
      #if ( $customer.hasPurchased($mud) ) 
        <tr>
          <td>
            * [link>$flogger.getPromo( $mud )]
          </td>
        </tr>
      #end
    #end
  </table>
</div>
{/velocity}

Reasons:

• This allows improved performance by caching all the non macro blocks (at the DOM level but also even after they've been rendered).
• This allows introducing other scripting languages such as Freemarker, JRuby, JPython, etc.
• This helps the WYSIWYG editor since it can provide a special way to edit Velocity macro content.

• This means allowing both HTML and Wiki syntax inside the Velocity macro as shown in the example. We will allow them by default and have a wiki=true/false and html=true/false optional parameters.
  • html=false means that the HTML will be escaped when rendered.
• The Velocity macro will work like this:
  • Run Velocity on the content. This generates some text containing XHTML and Wiki syntax.
  • Run the Parser on the generated text. This generates Listener events.
• We should have an optional parameter to decide if we should use a brand new Velocity context or reuse the global one.

A7 - Introduce a XHTML Macro

By default any XHTML entered as part of the text will be escaped when rendered. There are 2 reasons for this:

• This prevents introducing XHTML accidentally
• This allows us to know where the HTML blocks are and for example this will come very handy in the WYSIWYG editor so that portion of XHTML can be edited specifically
• Admins of a XWiki installation will be able to prevent using the XHTML macro for example (for security reasons) by disabling the macro.

{xhtml:wiki=true/false}

When wiki=true the content of the XHTML macro will be allowed to contain wiki syntax. If the user wants to also put velocity script then he must use the velocity Macro.

A8 - Introduce a Groovy Macro

To be consistent with other macros, introduce a {groovy} macro. It'll work similarly to the Velocity macro.

A9 - Nested Macro support

It'll be possible for macros to support other nested macros. This will be left to the Macro implementation. Nothing will prevent a Macro to call the Parser after a first pass is done. For example the Velocity macro will allow one level of nested macros.

A10 - New Include macro

Introduce new {include} macro to include other pages into the current one. The macro will act at the AST level (element blocks) and insert the AST from the included page into the current page.

The syntax will be something like:

{include:Space.Page|context=new/current}

• If context=current (the default when not specified) then the Space.Page's AST will first be merged with the calling page's AST and the page evaluated after.
• If context=new then the Space.Page will get parsed and have its macros applied and the resulting AST will be merged with the calling page's AST.

• Macros are executed at render time but get passed the DOM. Since the TOC macro will be executed last the include macro can contribute sections for example.

A11 - Caching Levels

There are 4 levels of caching:

• Level 1: Cache the DOM (element blocks) resulting from document parsing. Note that Macro block content are cached as text.
• Level 2: Cache non macro blocks after they are rendered (XHTML) since they are static.
• Level 3: Cache the fully rendered document. This is a time-cache (the cache is refreshed every N minutes). This is required for heavy sites (another possibility for heavy sites is to export the full wiki to HTML).
• Level 4: It's a variation of level 3. Cache a page (ie. document + skin)

A12 - No Interpreter needed

In the current implementation there are 2 kinds of renderers: renderers that do rendering (HTML, XML, etc) and renderers that simply evaluate text (Groovy, Velocity). In the new architecture the interpretation is left to the Velocity and Groovy components. Note that we already have the Velocity component and we simply need to add an evaluate method to it, and we need to create a similar Groovy component.

A13 - HTML Support

• HTML is escaped when not entered in a macro that supports HTML (like the XHTML macro or the Velocity Macro).
• Macros can allow HTML. For example here's how the Velocity macro should handle it:

1. 1. Execute Velocity first
   2. If html=true, use an XML Parser to parse the XML into an XML DOM.
      • Note that if we want to allow non XML before and after XML in the Velocity macro we'll need to strip it and pass it to the main wiki parser before we run the XML parser.
   3. If wiki=true then for each XML element's content run the wiki syntax parser. This generates Blocks. For each XML element, generate an XMLBlock too.
   4. Add begin/endXML listener events

A14 - Parsing is done at the word level

We had two options: at the sentence level or the word level. We have chosen at the word level for the following reasons:

• It's possible to easily write listener/renderers that act at the word level. For example, highlighting automatically words that have a wikipedia definition.
• We need an on special symbol event anyway sine we need (for example) to escape HTML that is not generated by a Macro. Thus sentences will be split anyway and it makes sense to push this further by having events for: words, space, special symbols, as it's done in WikiModel.

A15 - A new Wiki Syntax

The points above lead to defining a new wiki syntax which we call the XWiki Syntax 2.0. Of note:

• New syntax for bold which now uses two stars instead of one
• New syntax for links which now uses two brackets instead of one
• Support for new lines

Open questions

• Don't we need a {text} macro to have neither velocity neither HTML mneither wiki?
• Macro mapping to map velocity macros to Macros proper (so that Velocity macros can be used in the WYSIWYG editor for example)

Other ideas

• Have a special listener/renderer for the WYSIWYG editor that will extend the standard XHTML renderer and add special markup to help the editor.
• Have a "convert" button in the editors to convert a wiki syntax into another one. It should also support copy-pasting HTML or Word or Excel.
• Add the ability to export pages in other syntaxes (at the same level as PDF Export, XAR Export, etc), for easy migration to other wikis. We're that open! ;)

Migration path

Step 1: Velocity Component

• Improve the current xwiki-velocity component to add a evaluate() method. (done)
• Use this velocity component in core to replace current code (mostly done, we still have some init code in core but that's enough to progress)

• Modify all XWiki.renderTemplate*() methods to use the Velocity component's render method (and not the rendering component). This means we don't need to change any template at all.

Step 2: Groovy Component

• Create a new Groovy component similar to the Velocity one
• Migrate to the latest Groovy version (with Guillaume's help)
• Use this groovy component in core to replace current code

Step 3: Introduce new syntax metadata

• Add it in XWikiDocument + schema
• Create a migrator to migrate existing DBs, use "xwiki-1.0" for not defined syntax fields note: No migrator was required
• Add a xwiki.cfg setting to decide what the default syntax version is for new document
• Make newly created document use the configured syntax

Step 4: Extract current Rendering code into a component

• Create a new xwiki-rendering-compatibility component and move all the current rendering code in it
• Note: This component must use the same API as the new component
• Modify exiting code to use that new API (still pointing to the current code behind the API)

Step 5: New Rendering component

• Write + use the new rendering component
• Modify the calling code from core so that it decides which component to use based on the syntax metadata

• Modify XWikiDocument's getRenderedContent() to use the new rendering component to render a document
• Rewrite XWiki's getDisplayTitle(). We need to call the full Parser/Renderer (this will normally return the cached document DOM and not do a full parsing/rendering) and then find the first section block in the DOM. For now use an "if" depending on the syntax so that it works with xwiki-1.0 and xwiki-2.0 syntaxes.

Step 6: Write parser to support the xwiki-1.0 syntax

• Write parser to support the xwiki-1.0 syntax
• Remove xwiki-rendering-compatibility component

Implementation

The start of the implementation has started here: http://svn.xwiki.org/svnroot/sandbox/components/xwiki-rendering/

Possible frameworks to use

• WikiModel
• Doxia