More extension repositories
- Mentor(s)
- Student
- Details
Right now XWiki Extension Manager support plain Maven and XWiki Extension (http://extensions.xwiki.org) repositories.
The idea is to add support for as much other kind of repositories as possible.
Here are some good examples:
- Nexus
- Bintray
- Pypi
See http://dev.xwiki.org/xwiki/bin/view/Design/ExtensionManagerRepositories for more details.
- Developer profile
- Java
- HTTP/REST
- Active
- No
- Year
2017
- Status
Successfully terminated
- Progress
Google Submission
General
Introduction
So far it was possible inside XWiki to import extensions and artifacts from two types of repositories: XWiki Reposiory and Maven Repository. The aim of the "More extension repositories" project was to extend XWiki import capabilities and add integration with other types of remote repositories: Bintray, PyPi and NPM. The goal, in first place, was to enable download of artifacts and packages deployed on those remote repositories using Extension Manager, in the same way as from e.g. Maven Central. In second place, the goal was to adapt those packages and artifacts to make them usable in running XWiki: by scripts, or by other extensions.
XWiki architecture is strictly "extensions architecture". All the features of XWiki are supposed to be created and maintained as extensions, whilst the core of XWiki itself is responsible mainly for exposing easy and powerful API for extensions development. Thats why the product that I developed during GSoC was 3 separate extensions each per one type of repository: Bintray, PyPi and NPM.
What I have done:
- I created from scratch 3 independent XWiki extensions that connect publically available repositories (Bintray, PyPi, NPM Registry) and allow to:
- search in remote repository using Simple Search and Advanced Search in Extension Manger
- download artifacts and packages from repository
- install downloaded artifacts and packages inside running XWiki, making it available to use in by other components and scripts
- I released and deployed on each extension on XWiki nexus repository
- I created documentation on Extensions page:
- Bintray - http://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Repository%20Connector%20-%20Bintray/
- PyPi - http://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Repository%20Connector%20-%20Pypi/
- NPM - http://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Repository%20Connector%20-%20NPM/
More information on each extension in Details section.
Code
All code that I developed is in three github repositories (one per each extesion) created under Xwiki Contrib (https://github.com/xwiki-contrib) organization. I was the only code commiter to those repositories.
Links:- Bintray - https://github.com/xwiki-contrib/repository-bintray
- PyPi - https://github.com/xwiki-contrib/repository-pypi
- NPM - https://github.com/xwiki-contrib/repository-npm
Note that the structure of README file is governed by the XWiki community rules. Documentation is provided on extensions.xwiki.org (links above).
Details
Architecture
XWiki has handy API for easy extension of Extension Module, especially: ExtensionRepository, Extension, ExtensionFile, ExtensionRepositoryFactory. These interface classes are located in xwiki-commons core module. Adding support for new repository came down to providing implementation of this interfaces in the form of Components and packing them into JAR. To make extension initializable right after install - each extension had also special component implementing EventListener and Initializable interfaces. Because XWiki initializes each EventListener right after loading it to contexts that functionality was use to run initialization code of whole extension.Bintray
About extension
Bintray is a platform for exposing built artifacts. It aggregates functionalities of different repositories, e.g.: maven, NPM, debian. It's possible to connect them separately in native way or together via unified Rest API - that provides e.g. searching functionality.
This extension is to enable connection of maven repositories on Bintray, particularly adding possibility of searching (which is not available in native maven connection). Typical use case is to connect JCenter - the biggest maven repository (bigger then Maven Central) and be able to import Java artifacts.Current state
Fully developed extension with functionalities of search, download and installation of packages from maven repositories on Bintray.
Challenges and learnings
- Discovering Component architecture of XWiki
- Discovering Extension Modle API of XWiki
- Discovering concept and architecture of Bintray (which is not just a simple single repository)
- Discovering Bintray API
Possible improvements
- Adding support for other types of repositories available on Bintray
PyPi
About extension
XWiki has possibility to script on its pages in Python. Under the hood it's using the Java Scripting API and Jython that takes care of running this script and returning result to page. Jython has implementation only for standard libraries. That's why this extension was created: to be able to import to XWiki other (non-standard) python packages that are hosted on PyPi
Current state
- developed extension with functionalities of search, download and installation of .whl packages from PyPi repository
- cpython packages like Numpy, Pandas, Scipy ect cannot be imported to XWiki due to limitations of Jython
Challenges and learnings
- Discovering logic of Java Scripting API and particularly Jython
- Discovering PyPi API
- Because PyPi doesn't have searching facilities I needed to create mirroring and local search using Apache Lucene index
- Discovering types of python packages, compatibilities issues between python version, python dependencies specification in different types of packages, format of version specification
Possible improvements
- investigation of JyNI project which aim is to provide implementation of the most popular cpython packages for Jython and possibly integrate those packages to XWiki
- add integration with egg packages
NPM
About extension
Inside XWiki JS libaries may be required for example in Java Script Skin Extensions. There's already mechanism in XWiki that solves the problem of providing JS packages using WebJars - it installs them and provides JS files for front pages: WebJars API.
More about WebJars.
However not all JS libs are already packed in webJars and available in Maven Central Repository. Thats why Extension Repository Connector - NPM was created. It enables XWiki user to download any JS library available in NPM registry and converting it internally to WebJar, made it available for import on front pages.Current state
Fully developed extension with functionalities of search, download and installation of packages from NPM Registry.
Challenges and learnings
- Discovering API of NPM registry
- Discovering concept and structure of WebJar
- Manipulating archives in different formats (TAR, GZ, ZIP) from Java whilst converting JS packages to WebJars
Possible improvements
No obvious improvements
Work progress report
Milestones
Milestone 1 [30 May - 26 June]: - Creating Extension Repository Connector - Bintray
- Getting deeper familiarity with the architecture of Extension Module and it's API for new repository extensions
- investigation of Bintray and Artifacotry architecture
- investigation of Bintray integration API
- development of BintrayExtensionRepository
- deployment of an extension
- documentation creation
Milestone 2 [27 June - 24 July]: - Creating Extension Repository Connector - PyPi
- investigation of PyPi API
- investigation of Java-Python integration (Jython) in context of XWiki system
- investigation of Python package types, dependency system, version format
- development of PypiExtensionRepository
- deployment of an extension
- documentation creation
Milestone 3 [24 July - 21 August]: - Creating Extension Repository Connector - NPM
- investigation of NPM architecture
- investigation of NPM Registry API
- investigation of WebJar concept and package structure
- development of NPMExtensionRepository
- deployment of an extension
- documentation creation
Deliverables
D1 [M1]: Extension Repository Connector - Bintray
D2 [M2]: Extension Repository Connector - PyPI
D3 [M3]: Extension Repository Connector - NPM- I created from scratch 3 independent XWiki extensions that connect publically available repositories (Bintray, PyPi, NPM Registry) and allow to: