More extension repositories

Version 5.1 by Thomas Mortagne on 2017/09/20 16:50

Mentor(s)
Student
Details

Right now XWiki Extension Manager support plain Maven and XWiki Extension (http://extensions.xwiki.org) repositories.

The idea is to add support for as much other kind of repositories as possible.

Here are some good examples:

  • Nexus
  • Bintray
  • Pypi

See http://dev.xwiki.org/xwiki/bin/view/Design/ExtensionManagerRepositories for more details.

Developer profile
  • Java
  • HTTP/REST
Active
No
Year

2017

Status

Successfully terminated

Progress

Google Submission


General


Introduction

So far it was possible inside XWiki to import extensions and artifacts from two types of repositories: XWiki Reposiory and Maven Repository. The aim of the "More extension repositories" project was to extend XWiki import capabilities and add integration with other types of remote repositories: Bintray, PyPi and NPM. The goal, in first place, was to enable download of artifacts and packages deployed on those remote repositories using Extension Manager, in the same way as from e.g. Maven Central. In second place, the goal was to adapt those packages and artifacts to make them usable in running XWiki: by scripts, or by other extensions.

XWiki architecture is strictly "extensions architecture". All the features of XWiki are supposed to be created and maintained as extensions, whilst the core of XWiki itself is responsible mainly for exposing easy and powerful API for extensions development. Thats why the product that I developed during GSoC was 3 separate extensions each per one type of repository: Bintray, PyPi and NPM.

What I have done:

More information on each extension in Details section.

Code

All code that I developed is in three github repositories (one per each extesion) created under Xwiki Contrib (https://github.com/xwiki-contrib) organization. I was the only code commiter to those repositories.
Links:

Note that the structure of README file is governed by the XWiki community rules. Documentation is provided on extensions.xwiki.org (links above).


Details


Architecture
XWiki has handy API for easy extension of Extension Module, especially:  ExtensionRepository, Extension, ExtensionFile, ExtensionRepositoryFactory. These interface classes are located in xwiki-commons core module. Adding support for new  repository came down to providing implementation of this interfaces in the form of  Components and packing them into JAR. To make extension initializable right after install - each extension had also special component implementing EventListener and Initializable interfaces. Because XWiki initializes each EventListener right after loading it to contexts that functionality was use to run initialization code of whole extension.


Bintray

About extension

Bintray is a platform for exposing built artifacts. It aggregates functionalities of different repositories, e.g.: maven, NPM, debian. It's possible to connect them separately in native way or together via unified Rest API - that provides e.g. searching functionality.
This extension is to enable connection of maven repositories on Bintray, particularly adding possibility of searching (which is not available in native maven connection). Typical use case is to connect JCenter - the biggest maven repository (bigger then Maven Central) and be able to import Java artifacts.

Current state

Fully developed extension with functionalities of search, download and installation of packages from maven repositories on Bintray.

Challenges and learnings

  • Discovering Component architecture of XWiki
  • Discovering Extension Modle API of XWiki
  • Discovering concept and architecture of Bintray (which is not just a simple single repository)
  • Discovering Bintray API

Possible improvements

  • Adding support for other types of repositories available on Bintray

PyPi

About extension

XWiki has possibility to script on its pages in Python. Under the hood it's using the Java Scripting API and Jython that takes care of running this script and returning result to page. Jython has implementation only for standard libraries. That's why this extension was created: to be able to import to XWiki other (non-standard) python packages that are hosted on PyPi

Current state

  • developed extension with functionalities of search, download and installation of .whl packages from PyPi repository
  • cpython packages like Numpy, Pandas, Scipy ect cannot be imported to XWiki due to limitations of Jython

Challenges and learnings

  • Discovering logic of Java Scripting API and particularly Jython
  • Discovering PyPi API
  • Because PyPi doesn't have searching facilities I needed to create mirroring and local search using Apache Lucene index
  • Discovering types of python packages, compatibilities issues between python version, python dependencies specification in different types of packages, format of version specification

Possible improvements

  • investigation of JyNI project which aim is to provide implementation of the most popular cpython packages for Jython and possibly integrate those packages to XWiki
  • add integration with egg packages

NPM

About extension

Inside XWiki JS libaries may be required for example in Java Script Skin Extensions. There's already mechanism in XWiki that solves the problem of providing JS packages using WebJars - it installs them and provides JS files for front pages: WebJars API.
More about WebJars.
However not all JS libs are already packed in webJars and available in Maven Central Repository. Thats why Extension Repository Connector - NPM was created. It enables XWiki user to download any JS library available in NPM registry and converting it internally to WebJar, made it available for import on front pages.

Current state

Fully developed extension with functionalities of search, download and installation of packages from NPM Registry.

Challenges and learnings

  • Discovering API of NPM registry
  • Discovering concept and structure of WebJar
  • Manipulating archives in different formats (TAR, GZ, ZIP) from Java whilst converting JS packages to WebJars

Possible improvements

No obvious improvements


Work progress report


Milestones

Milestone 1 [30 May - 26 June]: - Creating Extension Repository Connector - Bintray

  • Getting deeper familiarity with the architecture of Extension Module and it's API for new repository extensions
  • investigation of Bintray and Artifacotry architecture
  • investigation of Bintray integration API
  • development of BintrayExtensionRepository
  • deployment of an extension
  • documentation creation

Milestone 2 [27 June - 24 July]: - Creating Extension Repository Connector - PyPi

  • investigation of PyPi API
  • investigation of Java-Python integration (Jython) in context of XWiki system
  • investigation of Python package types, dependency system, version format 
  • development of PypiExtensionRepository
  • deployment of an extension 
  • documentation creation

Milestone 3 [24 July - 21 August]: - Creating Extension Repository Connector - NPM

  • investigation of NPM architecture
  • investigation of NPM Registry API
  • investigation of WebJar concept and package structure
  • development of NPMExtensionRepository
  • deployment of an extension
  • documentation creation

Deliverables

D1 [M1]: Extension Repository Connector - Bintray
D2 [M2]: Extension Repository Connector - PyPI
D3 [M3]: Extension Repository Connector - NPM

 

Tags:
   

Get Connected