Weblate is an open source web-based translation tool written in Python. We use it as the implementation behind l10n.xwiki.org.

As of 2018-05-08, l10n.xwiki.org is still implemented using an XWiki instance but work is under way to move it to Weblate and this should happen soon.

How to install

Prerequisite

  • Python (2.7 in our setup)
  • Python virtualenv
  • Dependencies listed here
  • Postgresql
  • Memcached
  • hub

Quick setup:

For debian users: apt-get install memcached golang postgresql build-essential python virtualenv zlib1g-dev
Python dependencies:

pip install --upgrade pip
pip install -U setuptools
pip install Babel certifi chardet defusedxml diff-match-patch Django django-appconf django-compressor django-crispy-forms djangorestframework filelock idna lxml oauthlib phply Pillow ply psycopg2 psycopg2-binary pydns PyJWT pyLibravatar python-bidi python-dateutil python-memcached python-openid pytz pyuca PyYAML rcssmin requests requests-oauthlib rjsmin setuptools siphashc six social-auth-app-django social-auth-core translate-toolkit ua-parser urllib3 user-agents Whoosh Weblate

Installation

Most of the installation process can be found on the official documentation. We chose the virtualenv method to install all the python dependencies.

Let's call $VIRTUALENV the path of the virtual environment (it could be ~/weblate for instance).

After the installation, you can find weblate's files and other installed modules at $VIRTUALENV/local/lib/python2.7/site-packages/ (your python version may vary).

Note that if you use virtualenv, you'll have to use the weblate command instead of ./manage.py that you may found on weblate's documentation.

Configuration

Most of the configuration is made by editing the file $VIRTUALENV/local/lib/python2.7/site-packages/weblate/settings.py. You'll find a sample configuration here.

Here is what you can add to allow authentication using xwiki.org credentials and what you should add to include the required scripts:

...
from social_core.backends.oauth import BaseOAuth2
...
AUTHENTICATION_BACKENDS = (
    ...
    'weblate.settings.XWikiOAuth2',
)

class XWikiOAuth2(BaseOAuth2):
    name = 'XWiki'
    ID_KEY = 'sub'
    XWIKI_URL = 'https://www.xwiki.org/xwiki'
    AUTHORIZATION_URL = '{0}/oidc/authorization'.format(XWIKI_URL)
    ACCESS_TOKEN_URL = '{0}/oidc/token'.format(XWIKI_URL)
    REDIRECT_STATE = False
    ACCESS_TOKEN_METHOD = 'POST'
    EXTRA_DATA = [
        ('refresh_token', 'refresh_token', True),
    ]

    def get_user_details(self, response):
        username = response.get('sub').split('.')[-1]
        fullname, first_name, last_name = self.get_user_names(response.get('name'))
        return {
            'username': username,
            'email': response.get('email'),
            'fullname': fullname,
            'first_name': first_name,
            'last_name': last_name
        }

    def user_data(self, access_token, *args, **kwargs):
        return self.get_json(
            '{0}/oidc/userinfo'.format(self.XWIKI_URL),
            params={'access_token': access_token}
        )

SOCIAL_AUTH_XWIKI_KEY = '[SOCIAL_AUTH_XWIKI_KEY]'
...
POST_UPDATE_SCRIPTS = (
    '[xwiki-dev-tools]/weblate-scripts/post_update.sh',
)
POST_UPDATE_SCRIPTS = (
    '[xwiki-dev-tools]/xwiki-dev-tools/weblate-scripts/post_update.sh',
)
POST_COMMIT_SCRIPTS = (
    '[xwiki-dev-tools]/xwiki-dev-tools/weblate-scripts/post_commit.sh',
)
PRE_COMMIT_SCRIPTS = (
    '[xwiki-dev-tools]/xwiki-dev-tools/weblate-scripts/pre_commit.sh',
)

You should replace this parameters:

Generate a secret key by running $VIRTUALENV/share/weblate/examples/generate-secret-key.

In order for the settings to be found, you'll have to set the DJANGO_SETTINGS_MODULE environment variable. You can do it by adding export DJANGO_SETTINGS_MODULE=weblate.settings at the end of $VIRTUALENV/bin/activate file.

Other useful configurations can be found here.

Database configuration

Postgresql can be configured by following this steps.

Github configuration

You can configure Weblate to use a github account to make pull requests as described here (xwikiorgci can be used).

Running Weblate

Make sure you have sourced the virtualenv activate script (source $VIRTUALENV/bin/activate) and run weblate runserver.

How to import translation files manually

Login with an admin account to your weblate instance. You can generate one by running weblate createadmin.

Go to the administration panel by clicking the image.png icon or directly by going to /admin.

You'll need to create a project frst. We usually create one project for one git repository, for instance we have an XWiki Platform project.

Create the main component

You have to create the main component which will be the base of other components. A component is linked to a base translation file and all the related language files (e.g. ApplicationResources.properties is the base translation file and ApplicationResources_fr.properties is a related language file). This main component will be used to clone the git repository so that other components will use a link to this repository instead of making duplicates.

Here is a configuration example of a main component:

  • Component name: Help.SupportPanel.Content
  • URL slug: help-supportpanel-content
  • Project: XWiki Platform
  • Version control system: Github
  • Source code repository: https://github.com/xwiki/xwiki-platform
  • Repository push URL: https://github.com/xwiki/xwiki-platform
  • File mask: .translation/xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content_*.properties
  • Monolingual base language file: .translation/xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content_en.properties
  • File format: Java Properties (UTF-8)
  • Additional commit files:
    • xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content.xml
    • xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content.%(language)s.xml
  • Post-update script: post_update.sh
  • Pre-commit script: pre_commit.sh
  • Post-commit script: post_commit.sh

After saving this component, Weblate will clone the git repository in $VIRTUALENV/lib/python2.7/site-packages/data/vcs/xwiki-platform/help-supportpanel-content. An error should occur saying that the monolingual template file was not found. You need to run the post_update.sh script a first time to fix this issue. For that, you can run WL_PATH=$VIRTUALENV/local/lib/python2.7/site-packages/data/vcs/xwiki-platform/supportpanel xwiki-dev-tools/weblate-scripts/post_update.sh. See the readme file of the scripts repository to understand how it works.

Create other components

Here is an example of another component based on the previous one:

  • Component name: AnnotationCode.Translations
  • URL slug: annotationcode-translations
  • Project: XWiki Platform
  • Version control system: Github
  • Source code repository: weblate://xwiki-platform/help-supportpanel-content
  • Repository push URL: none
  • File mask: .translation/xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations_*.properties
  • Monolingual base language file: .translation/xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations_en.properties
  • File format: Java Properties (UTF-8)
  • Additional commit files:
    • xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations.xml
    • xwiki-platform-core/xwiki-platform-annotations/xwiki-platform-annotation-ui/src/main/resources/AnnotationCode/Translations.%(language)s.xml
  • Post-update script: none
  • Pre-commit script: pre_commit.sh
  • Post-commit script: post_commit.sh

How to import translation files automatically

There are many components to be created, that's why the generate_components.py script has been written to generate a json file with all the components to be created, based on some lists of translation files.
The script can be found in the xwiki-dev-tools project: xwiki-dev-tools/weblate-scripts/generate_components.py
A list of translation file has a name like this one `translation_list_xwiki-commons.txt`, where `xwiki-commons` is the name of the project, and looks like this:

#Format: Name of the component; Relative path to the translation file; Repository url
 xwiki-commons-extension-api; xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-api/src/main/resources/ApplicationResources.properties; https://github.com/xwiki/xwiki-commons
 xwiki-commons-job; xwiki-commons-core/xwiki-commons-job/src/main/resources/ApplicationResources.properties; https://github.com/xwiki/xwiki-commons 

Running the script will generate the components_xwiki-commons.json file. It can be imported by running weblate import_json components_xwiki-commons.json --project xwiki-commons.

Importing new translation files

If you just want to add new translation files to an existing project, you'll have to change two files and run two commands.
Let's say you want to add a .properties translation file from the xwiki-platform project. In the xwiki-dev-tools project, you'll have to change the files xwiki-dev-tools/weblate-scripts/translation_list_xwiki-platform.txt and xwiki-dev-tools/weblate-scripts/translation_properties.txt.

  1. Add at the end of translation_list_xwiki-platform.txt a line with the following format: Name of the component; Relative path to the translation file; Repository url
    For instance: Help.SupportPanel.Content; xwiki-platform-core/xwiki-platform-help/xwiki-platform-help-ui/src/main/resources/Help/SupportPanel/Content.xml; https://github.com/xwiki/xwiki-platform
  2. Add at the end of translation_properties.txt a line with the relative path to the translation files.
    For instance: xwiki-platform-core/xwiki-platform-template/xwiki-platform-template-api/src/main/resources/ApplicationResources.properties
  3. Run the generate_components.py script to generate components_xwiki-platform.json
  4. Run the command weblate import_json components_xwiki-platform.json --project xwiki-platform --ignore to finally import the new components (translation files).

How to upgrade Weblate

The whole process can be found here.

Miscellaneous

If the offloading indexing (OFFLOAD_INDEXING parameter) is activated (which should be), you'll have to run this command frequently (it can be a cron job) to update the index: weblate update_index.

Tags:
Created by Adel Atallah on 2018/05/07 11:24
   

Get Connected