Last modified by Thomas Mortagne on 2022/02/25 09:25

<
From version < 2.1 >
edited by Thomas Mortagne
on 2017/05/05 11:24
To version < 4.1 >
edited by Krzysztof PÅ‚achno
on 2017/08/27 12:53
>
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -xwiki:XWiki.ThomasMortagne
1 +xwiki:XWiki.cristof_p
Content
... ... @@ -1,1 +1,0 @@
1 -{{include document="GoogleSummerOfCode.ProjectClassSheet"/}}
GoogleSummerOfCode.ProjectClass[0]
Progress
... ... @@ -1,0 +1,172 @@
1 += Google Submission =
2 +
3 +
4 +----
5 +
6 +== **General** ==
7 +
8 +----
9 +
10 +=== Introduction ===
11 +
12 +So far it was possible inside XWiki to import extensions and artifacts from two types of repositories: XWiki Reposiory and Maven Repository. The aim of the "More extension repositories" project was to extend XWiki import capabilities and add integration with other types of remote repositories: Bintray, PyPi and NPM. The goal, in first place, was to enable download of artifacts and packages deployed on those remote repositories using Extension Manager, in the same way as from e.g. Maven Central. In second place, the goal was to adapt those packages and artifacts to make them usable in running XWiki: by scripts, or by other extensions.
13 +
14 +XWiki architecture is strictly "extensions architecture". All the features of XWiki are supposed to be created and maintained as extensions, whilst the core of XWiki itself is responsible mainly for exposing easy and powerful API for extensions development. Thats why the product that I developed during GSoC was 3 separate extensions each per one type of repository: Bintray, PyPi and NPM.
15 +
16 +
17 +=== What I have done: ===
18 +
19 +* I created from scratch 3 independent XWiki extensions that connect publically available repositories (Bintray, PyPi, NPM Registry) and allow to:
20 +** search in remote repository using [[Simple Search>>http://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Manager%20Application#HSimpleSearch]] and [[Advanced Search>>http://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Manager%20Application#HAdvancedSearch]] in Extension Manger
21 +** download artifacts and packages from repository
22 +** install downloaded artifacts and packages inside running XWiki, making it available to use in by other components and scripts
23 +* I released and deployed on each extension on XWiki nexus repository
24 +* I created documentation on Extensions page:
25 +** Bintray - [[http:~~/~~/extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Repository%20Connector%20-%20Bintray/>>doc:extensions:Extension.Extension Repository Connector - Bintray.WebHome]]
26 +** PyPi - [[http:~~/~~/extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Repository%20Connector%20-%20Pypi/>>doc:extensions:Extension.Extension Repository Connector - Pypi.WebHome]]
27 +** NPM - [[http:~~/~~/extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Repository%20Connector%20-%20NPM/>>doc:extensions:Extension.Extension Repository Connector - NPM.WebHome]]
28 +
29 +More information on each extension in Details section.
30 +
31 +
32 +=== Code ===
33 +
34 +All code that I developed is in three github repositories (one per each extesion) created under Xwiki Contrib (https://github.com/xwiki-contrib) organization. I was the only code commiter to those repositories.
35 +Links:
36 +
37 +* Bintray - https://github.com/xwiki-contrib/repository-bintray
38 +* PyPi - https://github.com/xwiki-contrib/repository-pypi
39 +* NPM - https://github.com/xwiki-contrib/repository-npm
40 +
41 +Note that the structure of README file is governed by the XWiki community rules. Documentation is provided on extensions.xwiki.org (links above).
42 +
43 +
44 +
45 +----
46 +
47 +== **Details** ==
48 +
49 +----
50 +
51 +**Architecture**
52 +XWiki has handy API for easy extension of Extension Module, especially: ExtensionRepository, Extension, ExtensionFile, ExtensionRepositoryFactory. These interface classes are located in xwiki-commons core module. Adding support for new repository came down to providing implementation of this interfaces in the form of Components and packing them into JAR. To make extension initializable right after install - each extension had also special component implementing EventListener and Initializable interfaces. Because XWiki initializes each EventListener right after loading it to contexts that functionality was use to run initialization code of whole extension.
53 +
54 +
55 +----
56 +
57 +=== **Bintray** ===
58 +
59 +==== About extension ====
60 +
61 +[[Bintray>>https://bintray.com/]] is a platform for exposing built artifacts. It aggregates functionalities of different repositories, e.g.: maven, NPM, debian. It's possible to connect them separately in native way or together via unified Rest API - that provides e.g. searching functionality.
62 +This extension is to enable connection of maven repositories on Bintray, particularly adding possibility of searching (which is not available in native maven connection). Typical use case is to connect JCenter - the biggest maven repository (bigger then Maven Central) and be able to import Java artifacts.
63 +
64 +==== Current state ====
65 +
66 +Fully developed extension with functionalities of search, download and installation of packages from maven repositories on Bintray.
67 +
68 +==== Challenges and learnings ====
69 +
70 +* Discovering Component architecture of XWiki
71 +* Discovering Extension Modle API of XWiki
72 +* Discovering concept and architecture of Bintray (which is not just a simple single repository)
73 +* Discovering Bintray API
74 +
75 +==== Possible improvements ====
76 +
77 +* Adding support for other types of repositories available on Bintray
78 +
79 +----
80 +
81 +=== **PyPi** ===
82 +
83 +==== About extension ====
84 +
85 +XWiki has possibility to script on its pages in Python. Under the hood it's using the Java Scripting API and Jython that takes care of running this script and returning result to page. Jython has implementation only for standard libraries. That's why this extension was created: to be able to import to XWiki other (non-standard) python packages that are hosted on PyPi
86 +
87 +==== Current state ====
88 +
89 +* developed extension with functionalities of search, download and installation of .whl packages from PyPi repository
90 +* cpython packages like Numpy, Pandas, Scipy ect cannot be imported to XWiki due to limitations of Jython
91 +
92 +==== Challenges and learnings ====
93 +
94 +* Discovering logic of Java Scripting API and particularly Jython
95 +* Discovering PyPi API
96 +* Because PyPi doesn't have searching facilities I needed to create mirroring and local search using Apache Lucene index
97 +* Discovering types of python packages, compatibilities issues between python version, python dependencies specification in different types of packages, format of version specification
98 +
99 +==== Possible improvements ====
100 +
101 +* investigation of [[JyNI project>>http://jyni.org/]] which aim is to provide implementation of the most popular cpython packages for Jython and possibly integrate those packages to XWiki
102 +* add integration with egg packages
103 +
104 +----
105 +
106 +=== **NPM** ===
107 +
108 +==== About extension ====
109 +
110 +Inside XWiki JS libaries may be required for example in Java Script Skin Extensions. There's already mechanism in XWiki that solves the problem of providing JS packages using WebJars - it installs them and provides JS files for front pages: WebJars API.
111 +More about WebJars.
112 +However not all JS libs are already packed in webJars and available in Maven Central Repository. Thats why Extension Repository Connector - NPM was created. It enables XWiki user to download any JS library available in NPM registry and converting it internally to WebJar, made it available for import on front pages.
113 +
114 +==== Current state ====
115 +
116 +Fully developed extension with functionalities of search, download and installation of packages from NPM Registry.
117 +
118 +==== Challenges and learnings ====
119 +
120 +* Discovering API of NPM registry
121 +* Discovering concept and structure of WebJar
122 +* Manipulating archives in different formats (TAR, GZ, ZIP) from Java whilst converting JS packages to WebJars
123 +
124 +==== Possible improvements ====
125 +
126 +No obvious improvements
127 +
128 +
129 +
130 +----
131 +
132 +== **Work progress report** ==
133 +
134 +----
135 +
136 +=== Milestones ===
137 +
138 +**Milestone 1 [30 May - 26 June]:** - Creating Extension Repository Connector - Bintray
139 +
140 +* Getting deeper familiarity with the architecture of Extension Module and it's API for new repository extensions
141 +* investigation of Bintray and Artifacotry architecture
142 +* investigation of Bintray integration API
143 +* development of BintrayExtensionRepository
144 +* deployment of an extension
145 +* documentation creation
146 +
147 +**Milestone 2 [27 June - 24 July]:** - Creating Extension Repository Connector - PyPi
148 +
149 +* investigation of PyPi API
150 +* investigation of Java-Python integration (Jython) in context of XWiki system
151 +* investigation of Python package types, dependency system, version format
152 +* development of PypiExtensionRepository
153 +* deployment of an extension
154 +* documentation creation
155 +
156 +**Milestone 3 [24 July - 21 August]:** - Creating Extension Repository Connector - NPM
157 +
158 +* investigation of NPM architecture
159 +* investigation of NPM Registry API
160 +* investigation of WebJar concept and package structure
161 +* development of NPMExtensionRepository
162 +* deployment of an extension
163 +* documentation creation
164 +
165 +=== Deliverables ===
166 +
167 +D1 [M1]: [[Extension Repository Connector - Bintray>>doc:extensions:Extension.Extension Repository Connector - Bintray.WebHome]]
168 +D2 [M2]: [[Extension Repository Connector - PyPI>>doc:extensions:Extension.Extension Repository Connector - Pypi.WebHome]]
169 +D3 [M3]: [[Extension Repository Connector - NPM>>doc:extensions:Extension.Extension Repository Connector - NPM.WebHome]]
170 +
171 +
172 +

Get Connected