Command localization in Ubiquity 0.5

A few important design decisions have already been made in implementing command localization in Ubiquity 0.5. The first was the choice of the gettext po (portable object) file format. The po format is a de facto industry standard with many tools built for the format and this design choice hopefully lower the bar for prospective localizers.

Second, in order to simplify the matching of localizations and commands, we require that each command feed have one localization po file, rather than splitting the localizations of different commands across multiple po files.

Finding the appropriate localization po file for a JavaScript command feed is very simple with bundled commands as the files are managed in our main repository so we know exactly where to find them. We just take the command feed’s filename sans extension—the feed key—and look for localizations/lang/feed key.po, where lang is the active language code. This gives us a simple one to one relationship between the JavaScript command feed source and the appropriate localization.

Background

The ability of users to easily write their own Ubiquity commands has always been a huge strength of the Ubiquity platform. Users can also “subscribe” to commands written by other users on other servers. In this case, local copies of those command sources are made.

The herd was developed as a dynamic aggregator of community Ubiquity commands. The herd keeps its own copy of each command source. The herd groups mirrors of command feeds on multiple servers together as well, giving each unique command feed a unique ID.

Localizing distributed resources

This distributed nature of the command feeds is exactly what complicates the localization of community commands. The original sources themselves are distributed, and we also want a community of localizers to be able to localize the commands—i.e., for the localizations to be (in some sense) distributed. Here I will present three options which I believe could be the basis for a winning solution, in order of more distributed to less distributed.

All three of these options have the property that localizations do not need to be “approved” or “managed” by the command author. For example, a possible standard where URL’s of localizations must be in the command feed is not considered. I believe this is a crucial property of any approach we decide on.¹

Option 1: a completely distributed option

In this option, localizers simply put the po files on their own servers (or some code snippet site such as github) and the user must then subscribe to those localizations much as they subscribe to commands now.

Option 2: registration and discovery through the herd

In this option, localizers put po files up on their own servers and then register that po file’s URL with the herd. The herd keeps track of each command feed’s localizations in different languages.

Option 3: localizations on the herd

In this option, po localizations are uploaded right onto the herd. The herd is the centralized repository of all localizations for each command feed.

Users first

In coming up with a criteria for judging different models of community command localization, I think it is helpful to think of the end-user experience. Right now to subscribe to a new command a user must find the command (perhaps via the herd) and click on the subscribe button, then in most cases confirm that they are aware of the possible dangers and confirm subscription.² What work is required for a user to subscribe to a new command and get it in their language?

Under option one, the user would somehow have to find the localization scattered someplace on the internet of their own accord, and then install that localization by themself. In my mind, this is clearly a no go. With options two or three, however, when a user subscribes to a command feed Ubiquity can check with the herd to see whether there are any localizations available. The localizations could be offered to the user or the localization for the currently active language could be automatically installed. There are, on the other hand, disadvantages to requiring a centralized authority, exemplified by the fact that the current iteration of the herd itself has often been down.

Summary and a call for comments

As laid out in this visual guide, I personally have a couple main criteria which I believe we should follow:

1. localization independence: Command authors should not have to manage their commands’ localizations. (See footnote 1.)
2. friendly discovery and subscription: Users should not have to go out and find localizations by themselves. Localizations should be offered to the user.

With these criteria, I’m pretty sure the only logical conclusion is that we need some level of centralization, pointing to options two and three. If you think of another option which satisfies these criteria, or disagree with the criteria above, I would love to know. In addition, have you encountered similar problems of localizing distributed resources elsewhere? What worked there?

If this problem has been solved before, there is no need to reinvent the wheel. As far as I can tell, however, this could be a particularly hairy problem.

Related posts:

1. Ubiquity Localization Update
2. Localizing Commands for Ubiquity 0.5
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

It’s not too late to add localizations for other languages to 0.5, though. Localizations help make Ubiquity more “natural” for more users, offering a new level of ease and familiarity to the already powerful Ubiquity. We have a new tutorial to help you localize commands.

To help encourage command localization, we now have gettext-style po template files for all the bundled command feeds in the hg repository. You can find these files in the ubiquity/localization/templates directory of the repository, or on our online hg repository.

If you complete some localizations (even incomplete) for your language and would like to submit them into the repository, for the time being, you can post them on this trac ticket.¹

I’ll be looking forward to seeing your localizations! If you have any questions, feel free to ask on the ubiquity-i18n Google group or on irc.mozilla.org#ubiquity. ^^

Related posts:

1. Localizing Ubiquity: commands and nountypes
2. Ubiquity Localization Update
3. Big Issues and Small Issues with Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.

• Goals for 0.5
  • Parser 2 (on by default)
  • underlying support for localization of commands
  • localization of standard feed commands for a few languages
  • Parser 2 language files for those same languages
• Nongoals for 0.5
  • distribution/sharing of localizations
  • localization of nountypes

The overall goal for this release of Ubiquity is to come up with a format and standard for localization. Localizations in Ubiquity 0.5 will only apply to commands bundled with Ubiquity, and the localization files themselves will be distributed with Ubiquity. In a future release we will tackle the problem of localizations for commands in the wild and truly croud-source¹ this process.

Localization Architecture

The localization of Ubiquity commands will use a gettext-style approach where localization files list key-value pairs for different properties and messages of the commands. For Ubiquity 0.5, where we only deal with the standard command feeds bundled Ubiquity, we can simply place all the localization files in ubiquity/standard-feeds/localization. Localization files are organized by source feed, with one localization file per source feed, per language.

The localizable components of commands will include the names, contributors, and help properties, as well as any localizable strings in the command’s preview() and execute() methods. To make strings localizable in preview() and execute(), they must be wrapped in the localize function, _().²

Other localizable components, like names, contributors, and help will not need to be wrapped in the _() function. In addition, as the localization files can only hold values of strings, for values such as names and contributors, the delimiter | can be used to delimit multiple values.

zoom.names=ズーム|ズームして|ズームする|ズームしろ

The Localization Experience

One tool we have planned to help kickstart the localization process is a tool that will automatically create a template of strings that need localization in a user’s commands. I took a first stab at this tool today. Clicking on the “get localization template” link next to each feed in the Ubiquity command list will give you a template which you can then copy into a text file:

Additionally, instructions will later be added to this page to specify how and where to save localizations to test them or perhaps we can add a button that will automatically save it in the right location.

Open Questions

Localization file formats

There are two kinds of file formats for localizations we are considering: .properties and .po, the native gettext format. As an example, here is the same key-value pair in the two formats:

.properties:

# This is a comment
welcomeMessage=Hello, world!

.po:

#. This is a comment (the . is actually optional)
msgid "welcomeMessage"
msgstr "Hello, world!"

The advantage of .properties over .po is that Mozilla natively supports this format with an XUL/XPCOM interface called stringbundle and it is what is used to localize JavaScript in Firefox itself. We actually already have the _() localization function working with the properties file format, following gomita’s great instructions (Japanese) on how to load properties files in using Mozilla’s native stringbundle tools via JavaScript.

The advantage of .po over .properties is that it is the de-facto standard in localization, particularly in the UNIX world. Lots of great tools have been written for it. The adoption of .po could make Ubiquity localization more accessible for more people. Another advantage is that .po files can have keys with spaces, as I note below.

If we do opt to work with .po files, the two libraries I see out in the wild for dealing with .po files are gettext-js (MIT) and jsgettext (LGPL). While I haven’t looked at the libraries in depth yet, so far jsgettext seems to be the winner, as some sections of gettext-js require the use of the prototype.js library.

A “key” question

In either file format, we need a unique way to refer to each localizable string—a key format. As each localization file refers to a command feed, the first collision we must avoid is the command name. With this in mind, we can come up with some trivial keys for the localizable properties: (here, consider the command hello)

• hello.names
• hello.contributors
• hello.help

However, we run into difficulty when we try to come up with keys for the arbitrary text in previews and executes. For example, for a message like “Hello world!” in the preview, we could simply make the key hello.preview.Hello world! but this may be unruly and be prone to typos. In addition, in .properties files keys cannot have certain characters in them, like spaces, so we would have to make the key something like hello.preview.Hello_world! or, stripping symbols and standardizing case, hello.preview.HELLO_WORLD.

Keys could also get very long with this type of key format, although here again .po files may have an advantage as they can stay relatively more legible even with long keys. One option to deal with this would be to optionally supply a key argument to _() so that it is used instead of the automatic key. For example, suppose the hello command’s preview() included this code:

_('This is a really long greeting message. Hello there!','longmessage')

then a localizer would only have to refer to hello.preview.longmessage, not hello.preview.THIS_IS_A_REALLY_LONG_GREETING_MESSAGE_HELLO_THERE.

satyr points out that some commands use another function to incorporate similar actions and messages in both preview() and execute(). In this case, he argues, it wouldn’t make sense to have to keep both localizations (hello.preview.… and hello.execute.…). He suggests that optional keys (mentioned above) could be used without the preview. or execute. infixes, as in hello.longmessage. By taking out the preview and execute namespacing in the localization keys, though, it becomes the command author’s responsibility to not accidentally use strings named “names”, “help”, etc. that will have unintended consequences.

Conclusion

I hope that this blog post gives people an idea of the progress we’ve made in the localization area and gets people thinking about the challenges we still face. We’d love to get your feedback on the localization format and process in Ubiquity, as well as the open problems of the file format and keys.

Related posts:

1. Big Issues and Small Issues with Parser 2
2. Localizing Ubiquity: commands and nountypes
3. Ubiquity Commands by The Numbers

Related posts brought to you by Yet Another Related Posts Plugin.

Today I’ll examine highlight some disparate typological data to help us understand these questions: where’s the verb? and what does the verb look like? There are broadly three different verb forms taken in commands in different languages:¹

1. the infinitive,
2. subjunctive mood, or
3. a special verb form such as imperative, participial, or conjunctive (such as Japanese て form)

Let’s give an example of each:

Infinitive (English):²

1

Hit me!

Subjunctive mood (Modern Greek): “Eat it all!”

2
3

Na   to fas olo!
SUBJ it eat all

Imperative form (French): “Eat it!”

4
5

Mange   -le!
eat.IMP it

It’s important to note that some languages have multiple forms available for the same command. For example:

Dutch: three ways to say “watch out!” with the same verb

1. Infinitive: Oppassen!
2. Imperative: Pas op!
3. Participial: Opgepast!

Similarly, I received a great comment by PhiliKON on German and associated data by Robert Kaiser on my blog post yesterday:

German: “search hello with google”

1. Infinitive: hello mit google suchen
2. Imperative: suche hello mit google

In addition, German and Dutch are interesting as they are verb second (V2) languages, so the verb may surface at the beginning or the end of the sentence, depending on the form.

The new parser design (which you can demo) assumes for simplicity that the verb should be found at the beginning or the end of the input, which is consistent with the data I’ve seen (modulo clitics). Multiple verb forms could be accounted for by supporting “synonyms” of the verbs.

What are the different ways verbs are expressed in commands in your language? Is the verb always found at the beginning or the end of the sentence? Is it ever somewhere in the middle?

Related posts:

1. Ubiquity i18n: questions to ask
2. Ubiquity Parser: The Next Generation Demo
3. Contribute: how your language identifies its arguments

Related posts brought to you by Yet Another Related Posts Plugin.

As a linguist, studying a language involves looking at instances of that language as data. To this end, we as Ubiquity internationalizers need to get at some examples of target utterances. Here’s an example survey which could be a good starting point for native speakers who want to contribute information on their language, based on Blair’s list of common Ubiquity verbs.

A survey for Ubiquity localization

Instructions

How would you express the following commands in your language? The words in CAPITAL LETTERS do not need to be translated. Feel free to give multiple possible answers for each command.

Try to express the same command rather than forcing a “literal translation”; for example, if there’s no “map” verb in your language, you could translate example (8) as lookup a map of PLACE. Please keep in mind that the addressee is a computer.

Basic word order / argument structure

1. search HELLO
2. search HELLO with google
3. translate HELLO from English to French
4. lookup the weather for PLACE
5. shop for SHOES with Amazon
6. email HELLO to Bill
7. email HELLO to ADDRESS
8. map PLACE
9. find HELLO
10. tab to HELLO or switch to HELLO tab

…

Pronominal/deictic arguments (aka “magic words”)

1. search this with google
2. translate this to French
3. bookmark this tab

…

How this data is used

Responses to these surveys would be used to identify certain salient features of the language, such as how the language codes for its arguments (for example using adpositions, case marking, or word order), whether the commands tend to be verb-inital or -final. Individual case markings, for example, can be identified by comparing minimal pairs—for example, by comparing item (1) and (2), we can learn how google in an instrumental role is marked, or by comparing example (2) and the “magic word” example (1), we can identify the appropriate “magic word” and determine whether the language uses any clitics or not.

Data collection

In the future we ideally could build a web-based system to collect these “utterances.” We could also use such a system to automatically test our parsers in different languages against the sentences in the command-bank, or ultimately even generate parser parameters based on those sentences. That would essentially reduce the parser-construction process to a more run-of-the-mill string translation process.

Related posts:

1. Contribute: how your language identifies its arguments
2. Writing commands with semantic roles
3. Ubiquity in Firefox: Focus on Japanese

Related posts brought to you by Yet Another Related Posts Plugin.