Changes for page Weblate Configuration

Last modified by Simon Urli on 2021/09/17 10:28

<
From version < 10.1 >
edited by Thomas Mortagne
on 2020/04/24 08:49
To version < 12.1 >
edited by Simon Urli
on 2020/12/14 14:50
>
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -xwiki:XWiki.ThomasMortagne
1 +xwiki:XWiki.surli
Content
... ... @@ -1,223 +1,9 @@
1 1  [[Weblate>>https://weblate.org/en/]] is an open source web-based translation tool written in Python. We use it as the implementation behind [[l10n.xwiki.org>>http://l10n.xwiki.org]].
2 +This page documents the operations contributors are able to perform with Weblate UI. For more information about maintaining Weblate, [[check this page>>doc:xwiki:XWiki.Maintenance.Weblate.WebHome]].
2 2  
3 3  {{toc/}}
4 4  
5 -= How to install =
6 6  
7 -== Prerequisite ==
7 += Add new component =
8 8  
9 -* Python (2.7 in our setup)
10 -* Python virtualenv
11 -* Dependencies [[listed here>>https://docs.weblate.org/en/latest/admin/install.html#python-dependencies]]
12 -* Postgresql
13 -* Memcached
14 -* [[hub>>https://hub.github.com/]]
15 -
16 -__Quick setup:__
17 -
18 -For debian users: {{box}}apt-get install memcached golang postgresql build-essential python virtualenv zlib1g-dev{{/box}}
19 -Python dependencies:
20 -
21 -{{code}}
22 -pip install --upgrade pip
23 -pip install -U setuptools
24 -pip install Babel certifi chardet defusedxml diff-match-patch Django django-appconf django-compressor django-crispy-forms djangorestframework filelock idna lxml oauthlib phply Pillow ply psycopg2 psycopg2-binary pydns PyJWT pyLibravatar python-bidi python-dateutil python-memcached python-openid pytz pyuca PyYAML rcssmin requests requests-oauthlib rjsmin setuptools siphashc six social-auth-app-django social-auth-core translate-toolkit ua-parser urllib3 user-agents Whoosh Weblate
25 -{{/code}}
26 -
27 -
28 -== Installation ==
29 -
30 -Most of the installation process can be found on the [[official documentation>>https://docs.weblate.org/en/latest/admin/install.html]]. We chose the [[virtualenv method>>https://docs.weblate.org/en/latest/admin/install.html#installing-in-virtualenv]] to install all the python dependencies.
31 -
32 -Let's call (% class="mark" %)##$VIRTUALENV##(%%) the path of the virtual environment (it could be {{box}}~~/weblate{{/box}} for instance).
33 -
34 -After the installation, you can find weblate's files and other installed modules at {{box}}$VIRTUALENV/local/lib/python2.7/site-packages/{{/box}} (your python version may vary).
35 -
36 -Note that if you use virtualenv, you'll have to use the {{box}}weblate{{/box}} command instead of {{box}}./manage.py{{/box}} that you may found on weblate's documentation.
37 -
38 -== Configuration ==
39 -
40 -Most of the configuration is made by editing the file {{box}}$VIRTUALENV/local/lib/python2.7/site-packages/weblate/settings.py{{/box}}. You'll find a sample configuration [[here>>https://docs.weblate.org/en/latest/admin/sample.html#sample-configurationhere]].
41 -
42 -Here is what you can add to allow authentication using [[xwiki.org>>http://www.xwiki.org/]] credentials and what you should add to include the required scripts:
43 -
44 -{{code language="python"}}
45 -...
46 -from social_core.backends.oauth import BaseOAuth2
47 -...
48 -AUTHENTICATION_BACKENDS = (
49 -    ...
50 -    'weblate.settings.XWikiOAuth2',
51 -)
52 -
53 -class XWikiOAuth2(BaseOAuth2):
54 -    name = 'XWiki'
55 -    ID_KEY = 'sub'
56 -    XWIKI_URL = 'https://www.xwiki.org/xwiki'
57 -    AUTHORIZATION_URL = '{0}/oidc/authorization'.format(XWIKI_URL)
58 -    ACCESS_TOKEN_URL = '{0}/oidc/token'.format(XWIKI_URL)
59 -    REDIRECT_STATE = False
60 -    ACCESS_TOKEN_METHOD = 'POST'
61 -    EXTRA_DATA = [
62 -        ('refresh_token', 'refresh_token', True),
63 -    ]
64 -
65 -    def get_user_details(self, response):
66 -        username = response.get('sub').split('.')[-1]
67 -        fullname, first_name, last_name = self.get_user_names(response.get('name'))
68 -        return {
69 -            'username': username,
70 -            'email': response.get('email'),
71 -            'fullname': fullname,
72 -            'first_name': first_name,
73 -            'last_name': last_name
74 -        }
75 -
76 -    def user_data(self, access_token, *args, **kwargs):
77 -        return self.get_json(
78 -            '{0}/oidc/userinfo'.format(self.XWIKI_URL),
79 -            params={'access_token': access_token}
80 -        )
81 -
82 -SOCIAL_AUTH_XWIKI_KEY = '[SOCIAL_AUTH_XWIKI_KEY]'
83 -...
84 -POST_UPDATE_SCRIPTS = (
85 -    '[xwiki-dev-tools]/weblate-scripts/post_update.sh',
86 -)
87 -POST_UPDATE_SCRIPTS = (
88 -    '[xwiki-dev-tools]/xwiki-dev-tools/weblate-scripts/post_update.sh',
89 -)
90 -POST_COMMIT_SCRIPTS = (
91 -    '[xwiki-dev-tools]/xwiki-dev-tools/weblate-scripts/post_commit.sh',
92 -)
93 -PRE_COMMIT_SCRIPTS = (
94 -    '[xwiki-dev-tools]/xwiki-dev-tools/weblate-scripts/pre_commit.sh',
95 -)
96 -{{/code}}
97 -
98 -You should replace this parameters:
99 -
100 -* ##[SOCIAL_AUTH_XWIKI_KEY]##: choose a unique string
101 -* ##[xwiki-dev-tools]##: local path of [[https:~~/~~/github.com/xwiki/xwiki-dev-tools>>https://github.com/xwiki/xwiki-dev-tools]]
102 -
103 -Generate a secret key by running {{box}}$VIRTUALENV/share/weblate/examples/generate-secret-key{{/box}}.
104 -
105 -In order for the settings to be found, you'll have to set the ##DJANGO_SETTINGS_MODULE## environment variable. You can do it by adding {{box}}export DJANGO_SETTINGS_MODULE=weblate.settings{{/box}} at the end of {{box}}$VIRTUALENV/bin/activate{{/box}} file.
106 -
107 -Other useful configurations can be found [[here>>https://docs.weblate.org/en/latest/admin/install.html#production-setup]].
108 -
109 -== Database configuration ==
110 -
111 -Postgresql can be configured by following [[this steps>>https://docs.weblate.org/en/latest/admin/install.html#postgresql]].
112 -
113 -== Github configuration ==
114 -
115 -You can configure Weblate to use a github account to make pull requests as [[described here>>https://docs.weblate.org/en/latest/vcs.html#hub-setup]] (##xwikiorgci## can be used).
116 -
117 -= Running Weblate =
118 -
119 -Make sure you have sourced the virtualenv activate script ({{box}}source $VIRTUALENV/bin/activate{{/box}}) and run {{box}}weblate runserver{{/box}}.
120 -
121 -= How to import translation files manually =
122 -
123 -Login with an admin account to your weblate instance. You can generate one by running {{box}}weblate createadmin{{/box}}.
124 -
125 -Go to the administration panel by clicking the [[image:image.png||height="16" width="17"]] icon or directly by going to ##/admin##.
126 -
127 -You'll need to create a project frst. We usually create one project for one git repository, for instance we have an ##XWiki Platform## project.
128 -
129 -== Create the main component ==
130 -
131 -You have to create the main component which will be the base of other components. A component is linked to a base translation file and all the related language files (e.g. ##ApplicationResources.properties## is the base translation file and ##ApplicationResources_fr.properties## is a related language file). This main component will be used to clone the git repository so that other components will use a link to this repository instead of making duplicates.
132 -
133 -Here is a configuration example of a main component:
134 -
135 -* Component name: **Help.SupportPanel.Content**
136 -* URL slug: **help-supportpanel-content**
137 -* Project: **XWiki Platform**
138 -* Version control system: **Github**
139 -* Source code repository: **https:~/~/github.com/xwiki/xwiki-platform**
140 -* Repository push URL: **https:~/~/github.com/xwiki/xwiki-platform**
141 -* File mask: **.translation/xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content_*.properties**
142 -* Monolingual base language file: **.translation/xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content_en.properties**
143 -* File format: **Java Properties (UTF-8)**
144 -* Additional commit files:
145 -** **xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content.xml**
146 -** **xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content.%(language)s.xml**
147 -* Post-update script: **{{scm path="xwiki-dev-tools/weblate-scripts/post_update.sh"}}post_update.sh{{/scm}}**
148 -* Pre-commit script: **{{scm path="xwiki-dev-tools/weblate-scripts/pre_commit.sh"}}pre_commit.sh{{/scm}}**
149 -* Post-commit script: **{{scm path="xwiki-dev-tools/weblate-scripts/post_commit.sh"}}post_commit.sh{{/scm}}**
150 -
151 -All files of this list as deployed at [[l10n.xwiki.org>>https://l10n.xwiki.org/]] are in the {{scm project="xwiki-dev-tools"}}xwiki-dev-tools repository{{/scm}}.
152 -
153 -After saving this component, Weblate will clone the git repository in {{box}}$VIRTUALENV/lib/python2.7/site-packages/data/vcs/xwiki-platform/help-supportpanel-content{{/box}}. An error should occur saying that the monolingual template file was not found. You need to run the (% class="mark" %)post_update.sh(%%) script a first time to fix this issue. For that, you can run {{box}}WL_PATH=$VIRTUALENV/local/lib/python2.7/site-packages/data/vcs/xwiki-platform/supportpanel xwiki-dev-tools/weblate-scripts/post_update.sh{{/box}}. See the [[readme file>>https://github.com/xwiki/xwiki-dev-tools/blob/master/weblate-scripts/README.md]] of the scripts repository to understand how it works.
154 -
155 -== Create other components ==
156 -
157 -Here is an example of another component based on the previous one:
158 -
159 -* Component name: **AnnotationCode.Translations**
160 -* URL slug: **annotationcode-translations**
161 -* Project: **XWiki Platform**
162 -* Version control system: **Github**
163 -* Source code repository: **weblate:~/~/xwiki-platform/help-supportpanel-content**
164 -* Repository push URL: **none**
165 -* File mask: **.translation/xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations_*.properties**
166 -* Monolingual base language file: **.translation/xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations_en.properties**
167 -* File format: **Java Properties (UTF-8)**
168 -* Additional commit files:
169 -** **xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations.xml**
170 -** **xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations.%(language)s.xml**
171 -* Post-update script: **none**
172 -* Pre-commit script: **pre_commit.sh**
173 -* Post-commit script: **post_commit.sh**
174 -
175 -= How to import translation files automatically =
176 -
177 -There are many components to be created, that's why the (% class="mark" %)generate_components.py(%%) script has been written to generate a json file with all the components to be created, based on some lists of translation files.
178 -The script can be found in the [[xwiki-dev-tools project>>https://github.com/xwiki/xwiki-dev-tools]]: {{box}}xwiki-dev-tools/weblate-scripts/generate_components.py{{/box}}
179 -A list of translation file has a name like this one `translation_list_xwiki-commons.txt`, where `xwiki-commons` is the name of the project, and looks like this:
180 -
181 -(% class="box" %)
182 -(((
183 -#Format: Name of the component; Relative path to the translation file; Repository url
184 - xwiki-commons-extension-api; xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/src/main/resources/ApplicationResources.properties; https:~/~/github.com/xwiki/xwiki-commons
185 - xwiki-commons-job; xwiki-commons-core/xwiki-commons-job/src/main/resources/ApplicationResources.properties; https:~/~/github.com/xwiki/xwiki-commons
186 -)))
187 -
188 -Running the script will generate the ##components_xwiki-commons.json## file. It can be imported by running {{box}}weblate import_json components_xwiki-commons.json ~-~-project xwiki-commons{{/box}}.
189 -
190 -== Importing new translation files (full procedure) ==
191 -
192 -If you just want to add new translation files to an existing project, you'll have to change two files and run two commands.
193 -Let's say you want to add a ##.properties## translation file from the ##xwiki-platform## project. In the [[xwiki-dev-tools project>>https://github.com/xwiki/xwiki-dev-tools]], you'll have to change the files ##xwiki-dev-tools/weblate-scripts/translation_list_xwiki-platform.txt## and ##xwiki-dev-tools/weblate-scripts/translation_properties.txt##.
194 -
195 -1. Add at the end of ##translation_list_xwiki-platform.txt## a line with the following format: {{box}}Name of the component; Relative path to the translation file; Repository url{{/box}}
196 -For instance: {{box}}Help.SupportPanel.Content; xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content.xml; https://github.com/xwiki/xwiki-platform{{/box}}
197 -1. Add at the end of ##translation_properties.txt## a line with the relative path to the translation files.
198 -For instance: {{box}}xwiki-platform-core/xwiki-platform-template/xwiki-platform-template-api/src/main/resources/ApplicationResources.properties{{/box}}
199 -1. Run the {{box}}generate_components.py{{/box}} script to generate ##components_xwiki-platform.json##
200 -1. Run the {{box}}call_update.py{{/box}} script with the full path of the weblate VCS, the name of the project, and the Git link of the project.
201 -For instance: {{code}}./call_updates.py /home/weblate/weblate/lib/python2.7/site-packages/data/vcs --project xwiki-platform --component https://github.com/xwiki/xwiki-platform{{/code}}
202 -1. Run the command {{code}}weblate import_json components_xwiki-platform.json --project xwiki-platform --ignore{{/code}} to finally import the new components (translation files).
203 -1. Run the commands to install the 3 different addons (pre-commit, post-commit and post-update) to the new component:
204 -
205 -* {{code}}weblate install_addon --addon xwiki.pre_commit xwiki-platform/jcaptcha-ui{{/code}}
206 -* {{code}}weblate install_addon --addon xwiki.post_commit xwiki-platform/jcaptcha-ui{{/code}}
207 -* {{code}}weblate install_addon --addon xwiki.post_update xwiki-platform/jcaptcha-ui{{/code}}
208 -
209 -{{warning}}
210 -Note that all commands starting with {{code}}weblate{{/code}} are done inside the weblate virtualenv. To enter in this virtualenv just run {{box}}source ~/weblate/bin/activate{{/box}}.
211 -{{/warning}}
212 -
213 -== Importing new translation files (simplified procedure) ==
214 -
215 -Instead of relying on all steps described above, it's heavily recommended to directly used the ##[[add_new_component.py>>https://github.com/xwiki/xwiki-dev-tools/blob/master/weblate-scripts/add_new_component.py]] script directly on Weblate machine which will automate most of the work.##
216 -
217 -= How to upgrade Weblate =
218 -
219 -The whole process can be found [[here>>https://docs.weblate.org/en/latest/admin/upgrade.html]].
220 -
221 -= Miscellaneous =
222 -
223 -If the offloading indexing (OFFLOAD_INDEXING parameter) is activated (which should be), you'll have to run this command frequently (it can be a cron job) to update the index: {{box}}weblate update_index{{/box}}.
9 +Each source translation file on a git repository is handled by a Weblate component. So whenever a committer adds a new source translation file, she should also creates the corresponding Weblate component.

Get Connected