Internationalizing your Project

Introduction

You have a project that uses Obviel for its user interface, and now you want it to work in multiple languages. For this you need to adjust your application to support multiple languages, a process known as “internationalization”, or “i18n” for short (as “internationalization” has 18 letters between the ‘i’ and the ‘n’).

How do you go about this?

The idea is that you mark up all translatable texts in your project’s code (.js and .obvt files and .html files with embedded templates) in a special way. When your application runs, marked text is looked up in a translation source. If a translation is found for a marked up piece of text, it is used instead of the original text. By changing the locale you can change which translation sources are in use.

A special extraction tool can be used to extract the marked up pieces of text from .js, .html and .obvt files, so that they can be given to human translators for translation.

Now let’s go into some more detail. We’ll discuss the JavaScript i18n strategy in detail here. We recommend however that you use Obviel Template for your i18n purposes, as it’s often more maintainable to mark up templates instead of code. Read Obviel Template i18n for much more detail on how to mark up your .obvt files for i18n. Then come back here and look at how to work with Obviel Template in the context of extraction tools and obviel.i18n.

JavaScript i18n

Imagine we have this piece of JavaScript code we want to i18n:

alert("Hello world!");

This application has been written with one language in mind, in this case English. Now we want to modify the application so that if the application runs with another language (for instance French) we will see the popup in that language (for instance “Bonjour monde!”) instead of in English.

You can make this happen using obviel.i18n. This uses the gettext approach, which is very often used for i18n for a variety of projects and programming languages. obviel.i18n is built on top of jsgettext, so include jsgettext like this:

<script type="text/javascript" src="/path/to/Gettext.js"></script>

and you need to include obviel-i18n itself:

<script type="text/javascript" src="/path/to/obviel-i18n.js"></script>

We now need to make a translation source available for our Hello world! message. This is what that looks like:

var fr_FR = obviel.i18n.translationSource(
  {'Hello world!': 'Bonjour monde!'});
obviel.i18n.registerTranslation('fr_FR', fr_FR);

Here we create a translation source from a simple JavaScript object using translationSource, and then register this source using registerTranslation. Later we’ll see how to load translations from a URL source.

Yu may have a module where you define views that use templates with
translatable content in them.

Next we will create a function _ that can be used to mark up text to make it translatable:

var _ = obviel.i18n.translate();

The idea is that we use obviel.i18n.translate() once on top of each of the modules we want to i18n.

Now let’s return to our JavaScript code to i18n:

alert("Hello world!");

We adjust it to use our _ function:

alert(_("Hello world!"));

When you run your application now, you’ll still see Hello world!. Without a specific locale set the texts embedded in the code itself will be used. But earlier we registered the locale fr_FR, and we can alter the locale to that:

obviel.i18n.setLocale('fr_FR');

Now when we run the program we’ll see Bonjour monde!.

Here is a demo that has put all this together in a slightly more involved way.

Variables in JavaScript

Obviel, both in JavaScript as well as templates, supports a way to interpolate variables into translations. This could for instance be the English source of the translation:

Hello {who}!

The translator now needs to take care to include this variable in the translation as well:

Bonjour {who}!

A translator could decide it makes more sense to move the variable to another place in the text:

{who}, bonjour!

What is retrieved as the translation will still contain the variable {who}. We need to make sure it gets filled in.

To do so you can use a special variables() formatting function that Obviel i18n makes available:

obviel.i18n.variables(_("Hello {who}!"), { who: "Bob" });

Note that for variables to work you do need to include obviel-template.js on the page as well.

Pluralization

Obviel i18n, thanks to jsgettext, also supports plural forms of a piece of text.

First let’s introduce a program that contains some plural forms:

alert(count + " cows");

Given a value for count of 2, this will show 2 cows, and if you supply 10 you’ll get 10 cows. So far so good. But what now when we only have 1 cow? We’d see 1 cows, which is incorrect. We want to see 1 cow. The plural form for 1 in English is different than that for any other number.

Let’s consider another language, Dutch, which has the same pluralization rule, but different pluralization. We’ll set up a translation:

var nl_NL = obviel.i18n.translationSource(
  {'1 cow': ['{count} cows', '1 koe', '{count} koeien']});
obviel.i18n.registerTranslation('nl_NL', nl_NL);

Here we see that 1 cow should be translated as 1 koe, and {count} cows, where count is any other number than 1, should be translated as {count} koeien. The exact syntax of how we specify this directly in the translation source is not important, as later on we’ll discuss tools to generate translation source JSON automatically.

To enable pluralization for JavaScript code you write the following somewhere on top of the module:

var ngettext = obviel.i18n.pluralize();

Naming this function ngettext is important, as it enables the extraction tools to extract plural information automatically.

We will now modify our original program as follows:

alert(variables(ngettext('1 cow', '{count} cows', count), {count: count}));

If we keep the locale to the original, we’ll see, for count 1, 1 cow, and for count 3 for instance we’d see 3 cows. This is because without a specific locale, the texts embedded in the source will be used. ngettext will return its first argument if the count variable (the third argument) is 1, and otherwise will return the second argument. We then use variables to interpolate count variable in the result.

Now we’ll change the locale to Dutch:

obviel.i18n.setLocale('nl_NL');

You’ll now see 1 koe and 3 koeien instead (for count 1 and 3), as we expected.

Different pluralization rules

Some languages have simpler or more complicated pluralization rules. Some languages for instance have no separate singular and plural forms. Other languages instead have 3 or more plural forms and different rules about when they are used.

Let’s examine Polish for instance. In Polish there are 3 plural forms, the singular form, a second form used for numbers ending with the digits 2, 3 or 4, and a third form for numbers (besides 1) that end with 5 to 9 or 1.

Let’s consider the following program:

alert(variables(ngettext('1 file', '{count} files', count)), {count: count});

Let’s consider count [1, 2, 4, 5, 21, 22, 25] and look at the output for English:

1 file
2 files
4 files
5 files
21 files
22 files
25 files

With Polish, we want the output to be like this:

1 plik
2 pliki
4 pliki
5 pliko'w
21 pliko'w
22 pliki
25 pliko'w

We can specify the Polish rule when registering the source in the Plural-Forms metadata:

var pl_PL = i18n.translationSource({
    '': {
        'Plural-Forms': 'nplurals=3; plural=(n==1 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2)'
    },
    '1 file.':
    ['{count} file.',
     '1 plik.',
     '{count} pliki.',
     "{count} pliko'w."]});

If we register this as the translation source for pl_PL and set the locale to that, we’ll get the output as expected.

That’s a scary Plural-Forms entry! Luckily there is tool support for these. When you maintain the translations in .po files like we detail below, the msginit tool can automatically generate these for you based on the locale. There is of course the possibility that it doesn’t know the plural forms for the locale you specify to msginit. It will generate:

"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;\n"

In that case you will have to put in an appropriate rule yourself. For more information about plural form rules, see this page with rules for plural forms for a whole range of languages, and this launchpad page which has textual descriptions of plural forms for many languages.

Domains

What if you have an application that has translatable content, and you want to use a library that has translatable content of its own? Obviel Forms is an example of such a library, as error messages displayed in the form need to be translatable.

In such a case we want to maintain the application translations and the library translations separately from each other. The translations for the library after all would typically be maintained by the library maintainers, not by the application maintainers.

You can take care of this by using a separate translation domain for each library and application. By default, obviel.i18n uses the default domain, but let’s change this now. Let’s alter our basic “Hello world” application so that it uses a separate domain:

var fr_FR = obviel.i18n.translationSource(
  {'Hello world!': 'Bonjour monde!'});
obviel.i18n.registerTranslation('fr_FR', fr_FR, 'hello_world_app');

var _ = obviel.i18n.translate('hello_world_app');

alert(_("Hello world!"));

As you can see, we now pass an extra argument 'hello_world_app' when registering the translation as well as when we create the translation function _. This string indicates the domain. You can specify a domain for registered translations. You can also specify that domain for your module, indicating that it will look for translations in that domain. When you use _, the domain will be implicitly used for lookups.

If you define views they will pick up the domain implicitly as well. Any Obviel Template associated with that view will use that domain to look up translation. You can also explicitly pass a domain property to the view when you define it.

If you use obviel.i18n.pluralize in a module you’ll have to specify the domain there as well:

var ngettext = obviel.i18n.pluralize('hello_world_app');

Obviel Template pluralization will pick up the domain from the obviel.i18n.translate declaration already, so if you only use Obviel Template pluralization in a module, and don’t need to use ngettext there is no need to make the obviel.i18n.pluralize call.

Extracting translatable content

We now know how to mark up our JavaScript code so it can be translated. We need a tool that can extract all pieces of marked up text automatically so that we can give it all to a translator.

We recommend the use of the Babel i18n tool with code that uses Obviel. While Babel was written to support i18n of Python applications, it also supports JavaScript applications.

If you’re on Linux, Babel may be available in your Linux distribution. You can also install it manually using one of the Python tools used for this (pip, easy_install, or buildout). After installation, a pybabel commandline tool will be available.

To make Babel work with .js files we first need to configure it. Create a .cfg file with the following content:

[javascript: **.js]
extract_messages = _

Now you can use it on your project directory:

$ pybabel extract -F myconfig.cfg project_directory > myproject.pot

You will now have a myproject.pot (PO template) file that will be the template for all the actual translation files for your application, the .po files. For example, if you had applied this to the simple demo described above, you would get a .pot file like this (skipping some metadata boilerplate):

msgid "Hello world!"
msgstr ""

In the comment line it points out where in the code the text to translate was found; this is sometimes useful context for translators.

For the plural forms example with files you’d see this in your .pot file:

msgid "1 file"
msgid_plural "{count} files"
msgstr[0] ""
msgstr[1] ""

Extracting from .obvt files

We’ve demonstrated the extraction procedure for a .js file, but what about Obviel Template .obvt files?

We provide a plugin to Babel called babel-obviel that knows how to extract translatable text from .obvt files. You install it using the standard Python installation tools. Make sure it is installed in the same Python as Babel is, so that Babel can pick up on this plugin.

Imagine you have this template, marked up for translation:

<p data-trans="">Hello world!</p>

We need to teach pybabel extract about .obvt files in the .cfg file we created before:

[javascript: **.js]
extract_messages = _

[obvt: **.obvt]

When you now run pybabel extract on your project, it will extract translatable texts not only from .js files but from .obvt files as well.

Read Obviel Template i18n for much more detail on how to mark up your .obvt files for i18n.

Translating Obviel templates in .html files

It is possible to embed an Obviel template in a script tag in a HTML file. You can tell pybabel to extract from these too, by modifying the .cfg file to read like this:

[javascript: **.js]
extract_messages = _

[obvt: **.obvt]

[obvt_html: **.html]

Creating .po files

For each language that you want to support in your project, you now need to create a .po file from the project’s .pot file. You can do this with the GNU gettext tool suite, using the msginit tool (details about msginit):

$ msginit -l fr_FR -i myproject.pot -o myproject-fr_FR.po

This command says we want to create a specific .po file for the French language locale (fr_FR). This is what the generated .po file looks like:

#: src/demo/i18n-js-demo.js:15
msgid "Hello world!"
msgstr ""

This is in fact very similar to our .pot file!

We’ve ignored metadata here, but in the metadata you’ll see the Plural-Forms section for that locale if msginit knows about it. See the Different pluralization rules section for more information on this.

We can now edit the file to add the French translations:

#: src/demo/i18n-js-demo.js:15
msgid "Hello world!"
msgstr "Bonjour monde!"

In practice, you would likely give this file to someone else: the person translating the application to French. They can then edit it directly to add or update the translations, or use some GUI tool that can work with .po files. When they are done, they would give the file back to you.

Updating .po files

When your project evolves, you will likely add new texts to translate, or change or remove existing ones. You can use the babel extract tool as usual to extract a new .pot file containing all the current translation texts. You can then use the GNU gettext msgmerge tool (details) to update your existing .po files so that the new translatable texts become available to your translators:

$ msgmerge -U myproject-fr_FR.po myproject.pot

Creating .json files from the .po files

We now have a .po file with the right translations. To make it work with Obviel i18n, we need to convert it to a JSON file that can be used as a translation source. You can do this using the pojson tool (installable using one of the Python methods such as pip, easy_install or buildout).

Using pojson you can convert the .po file to a simple .json file:

$ bin/pojson convert myproject-fr_FR.po > myproject-fr_FR.json

Put that .json file somewhere where it can be found. Normally that would be in the same directory as the .js files.

Loading up the translation JSON

Now we need to specify what translations are available. We do this in a JSON file which we give the .i18n extension by extension. This .i18n file is normally put in the same directory as the .js files as well, and therefore also in the same directory as the translation JSON files.

Let’s imagine we have an English translation that simply uses the source language, and a French translation that does have a JSON file available. The .i18n file would look like this:

{
   "default": [
      {
        "locale": "en_US",
        "url": null
      },
      {
        "locale": "fr_FR",
        "url": "myproject-fr_Fr.json"
   ]
}

We’ve registered the information for the default domain in this example, but normally you’d register it for your application’s domain or library’s domain.

Now we need to load this .i18n information. We do this using a <link> tag in the web page that needs the translations:

<link href="path/to/some.i18n" ref="i18n" type="application/json" />

You can include as many such link tags as you like, typically one per library you have installed that uses obviel.i18n.

Now you need to load the link tags. You do this when the document is ready, so like this:

$(document).ready(function() {
   obviel.i18n.load().done(function() {
      obviel.i18n.setLocale('fr_FR').done(function() {
          ... start application ...
      });
   });
});

Let’s examine this structure. When the document is ready, we load the i18n information. When the i18n information is known, we set the locale. This in turn will then look up the i18n information for the supplied locale. Once the locale is set, we can start up the rest of the application, for instance by rendering the view for the application object.