This week I implemented a new way to customize and extend Ubiquity commands: CmdUtils.CreateAlias.

The use case for and importance of CreateAlias

CreateAlias lets you easily create a special-case alias of another, more generic verb. Ubiquity comes bundled with useful verbs like translate and search which can be used for a number of different uses based on their arguments. In some cases, and in some languages, though, typing out translate to English or search with Google is unnatural, though, as there is a more succinct and direct way to make that request. For example, in English one could say “anglicize” or “google”, respectively, for the verbs and arguments above. Indeed, in order to support both search with Google and google, Ubiquity traditionally has implemented two different verbs, search and google, which duplicate functionality and code.

CreateAlias lets us create such natural aliases without repeating ourselves. We can easily create an anglicize verb which, in one word, does the work of translate to English, or google which is semantically equivalent to search with Google.

These sorts of aliases become particularly important in our perpetual quest to internationalize Ubiquity. One discussion that came up early on on our Ubiquity-i18n list is the fact that not all languages have the verb “Google”: in many languages it is necessary to explicitly say “search with Google”. Moreover, other languages may have other domain-specific verbs which English doesn’t have either. Maybe some language has a special verb for “email with Hotmail” or “map Denmark”. Who knows? With CreateAlias we can easily enable such localizations based on the more generic commands bundled with Ubiquity.

Creating an alias

CreateAlias was designed to be incredibly simple to use. Here’s an example that will be bundled (but not installed by default) in Ubiquity:

CmdUtils.CreateAlias({
  names: ["anglicize"],
  verb: "translate",
  givenArgs: { goal: "English" }
});

As you see, this syntax is incredibly straightforward. There are two required properties, names, an array of names for the alias, and verb, a reference to the target verb that this alias should use.¹

The alias can also have a givenArgs property which is a hash of pre-specified arguments with their semantic roles. Because translate takes three arguments (an object text, a goal language, and a source language) but we have pre-specified a goal in the givenArgs, the new anglicize command will only take two arguments: the object text and a source language. Of course, if you specify no givenArgs, you’ll get a simple synonym without having access to the original verb’s code.

As you see, the preview of this command is simply the preview of the translate verb. Its preview and execution is just as if you had entered translate こんにちは to English.

Just like other commands created with CreateCommand, the object specifying the alias can also have properties like help, description, author information, and so on. I used the icon property to add a Union Jack to it so that it was easily identifiable.

Bonus: using CmdUtils.previewCommand and CmdUtils.executeCommand

On the road to implementing CreateAlias, I also implemented the CmdUtils.previewCommand and CmdUtils.executeCommand functions. The majority of this code comes from previous work by Louis-Rémi Babé, though I adapted it to the modern Ubiquity system. Using previewCommand and executeCommand you can take advantage of the preview or execute functionality of another command. In the new alias-commands feed I included a command called germanize which essentially is a straightforward analogy to anglicize, seen above, but using these functions within a CreateCommand. While CreateAlias is much more straightforward for simple aliases, for more complex subcommands where you would like to adapt another verb’s execution or preview, or only take one of those but re-implement the other part, previewCommand and executeCommand are the way to go.

Related posts:

1. Ubiquity Localization: What’s New, What’s Next
2. Rolling out the Roles
3. Localizing Commands for Ubiquity 0.5

Related posts brought to you by Yet Another Related Posts Plugin.

Granted, these are only Ubiquity’s interface strings (for example, the about and settings pages)… the parser localization and command localization have their own processes.² But this is still a tremendous accomplishment!

Hopefully we can roll some of these complete or almost-complete interface localizations with Ubiquity 0.5.4 which is a minor bugfix update coming soon. If you would like to get involved with localizing the Ubiquity interface strings into your language, get a BabelZilla login and sign up on the Ubiquity project page. Thanks again to our rockin’ localizers!

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. A Visual Guide to Community Command Localization
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

Command localization in Ubiquity 0.5

A few important design decisions have already been made in implementing command localization in Ubiquity 0.5. The first was the choice of the gettext po (portable object) file format. The po format is a de facto industry standard with many tools built for the format and this design choice hopefully lower the bar for prospective localizers.

Second, in order to simplify the matching of localizations and commands, we require that each command feed have one localization po file, rather than splitting the localizations of different commands across multiple po files.

Finding the appropriate localization po file for a JavaScript command feed is very simple with bundled commands as the files are managed in our main repository so we know exactly where to find them. We just take the command feed’s filename sans extension—the feed key—and look for localizations/lang/feed key.po, where lang is the active language code. This gives us a simple one to one relationship between the JavaScript command feed source and the appropriate localization.

Background

The ability of users to easily write their own Ubiquity commands has always been a huge strength of the Ubiquity platform. Users can also “subscribe” to commands written by other users on other servers. In this case, local copies of those command sources are made.

The herd was developed as a dynamic aggregator of community Ubiquity commands. The herd keeps its own copy of each command source. The herd groups mirrors of command feeds on multiple servers together as well, giving each unique command feed a unique ID.

Localizing distributed resources

This distributed nature of the command feeds is exactly what complicates the localization of community commands. The original sources themselves are distributed, and we also want a community of localizers to be able to localize the commands—i.e., for the localizations to be (in some sense) distributed. Here I will present three options which I believe could be the basis for a winning solution, in order of more distributed to less distributed.

All three of these options have the property that localizations do not need to be “approved” or “managed” by the command author. For example, a possible standard where URL’s of localizations must be in the command feed is not considered. I believe this is a crucial property of any approach we decide on.¹

Option 1: a completely distributed option

In this option, localizers simply put the po files on their own servers (or some code snippet site such as github) and the user must then subscribe to those localizations much as they subscribe to commands now.

Option 2: registration and discovery through the herd

In this option, localizers put po files up on their own servers and then register that po file’s URL with the herd. The herd keeps track of each command feed’s localizations in different languages.

Option 3: localizations on the herd

In this option, po localizations are uploaded right onto the herd. The herd is the centralized repository of all localizations for each command feed.

Users first

In coming up with a criteria for judging different models of community command localization, I think it is helpful to think of the end-user experience. Right now to subscribe to a new command a user must find the command (perhaps via the herd) and click on the subscribe button, then in most cases confirm that they are aware of the possible dangers and confirm subscription.² What work is required for a user to subscribe to a new command and get it in their language?

Under option one, the user would somehow have to find the localization scattered someplace on the internet of their own accord, and then install that localization by themself. In my mind, this is clearly a no go. With options two or three, however, when a user subscribes to a command feed Ubiquity can check with the herd to see whether there are any localizations available. The localizations could be offered to the user or the localization for the currently active language could be automatically installed. There are, on the other hand, disadvantages to requiring a centralized authority, exemplified by the fact that the current iteration of the herd itself has often been down.

Summary and a call for comments

As laid out in this visual guide, I personally have a couple main criteria which I believe we should follow:

1. localization independence: Command authors should not have to manage their commands’ localizations. (See footnote 1.)
2. friendly discovery and subscription: Users should not have to go out and find localizations by themselves. Localizations should be offered to the user.

With these criteria, I’m pretty sure the only logical conclusion is that we need some level of centralization, pointing to options two and three. If you think of another option which satisfies these criteria, or disagree with the criteria above, I would love to know. In addition, have you encountered similar problems of localizing distributed resources elsewhere? What worked there?

If this problem has been solved before, there is no need to reinvent the wheel. As far as I can tell, however, this could be a particularly hairy problem.

Related posts:

1. Ubiquity Localization Update
2. Localizing Commands for Ubiquity 0.5
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

Of course, one of the best ways to learn about the new features is to experience them… try Ubiquity 0.5 now!

Install now!

Preface: What’s What

To give users a completely localized experience, there are many different components that need to be made to work with different languages. In a single Ubiquity input, like

1

translate hello from English to Spanish

there are actually many different components that need to all be localized in order to comprehend the equivalent sentence in a different language. The diagram below will give you a sense for the different components that need to be localized: the parser, verbs, and nountypes.

What’s New

Ubiquity 0.5’s improved language support can be thought of as the product of two more or less orthogonal developments: the brand-new parser, Parser 2, as well as local command localization support.

Parser 2

Parser 2 (née Parser: The Next Generation) is a completely new parser designed to support different languages easily. Taking a serious look at the similarities and differences between different languages, we created a universal parser design which takes a minimal set of settings for particular languages to “learn” that language’s grammar.

The key insight to Parser 2’s design is that, for the limited range of inputs Ubiquity should understand, languages deal with them in remarkably similar ways. The input we’re dealing with here are all commands or actions without quantification or negation. These are all comprised of a single verb and a series of arguments with certain markings to designate their roles in the sentence. For example, here’s our example Ubiquity input:

1

translate hello from English to Spanish

In this example, “translate” is the verb, which we recognize by looking at our bank of known verbs, and the rest of the input can be split up into three different arguments: “hello,” “from English,” and “to Spanish.” Of these, the markers “from” and “to” tell us that “English” is a source of some sort and “Spanish” is a goal, while the unmarked “hello” is simply an object—the target of the action. By identifying arguments by these abstract semantic roles, we’re able to quickly identify different kinds of arguments in different languages. For example, the following is the exact same example but using the Japanese syntax and markers:

2

helloをEnglishからSpanishにtranslate

Ubiquity knows what the different markers mean in Japanese, like “を” > object, “から” > source, “に” > goal, and can easily interpret this to mean the exact same command as (1). With just a few lines of code, you can teach Ubiquity how to recognize these different semantic roles in your language. This innovation also means that Ubiquity commands can be written once for one language and automatically used with another language’s parser, bringing us half-way to the goal of command localization.

Note also that Japanese (as in example (2)) is verb-final and uses no spaces between words. We’ve tried to make Parser 2 itself agnostic towards these types of different ways in which languages vary.

Parser 2 also adds better argument-first suggestions, inspired by some earlier thoughts on Ubiquity in Japanese. Ubiquity will now start to parse arguments in the input even if a verb isn’t found, and suggest verbs based on that input. For example, if you enter “hello to Spanish,” it’ll recognize that you have an object of “hello” and a goal of “Spanish” which can be understood as a language name, so it’ll suggest the verb “translate.” This is the way it should be.

For more information and background, feel free to check out some of my previous blog posts on the new parser and on the different linguistic considerations. I also have a four-page academic paper giving an overview of some innovations in the parser—email me at x@x.com where x=mitcho if you’d like to get a copy.

Internationalization of bundled commands

The move to use semantic roles in the new command API, described above, means that the same Ubiquity command code can be used with inputs in different languages. Two things are left, then, to make a completely localized input work: (1) translation (localization) of different strings in the commands and (2) localization of the nountypes.

In Ubiquity 0.5, we built a localization infrastructure for commands (1, above) but have not yet tackled the nountypes (2). Ubiquity 0.5 uses the gettext po (portable object) file format for localizations, which many localizers in the UNIX world are very familiar with. This choice of file format potentially opens Ubiquity localization up to many who are new to localization or are unfamiliar with other Mozilla localization. Ubiquity is able to produce localization templates by itself and we also have a great tool to check the completeness of different localizations.

A huge caveat, however, is that this localization support currently only works with the commands bundled with Ubiquity itself.

What’s Next

We’re going to continue working to make Ubiquity more natural for more users. The tasks we have ahead of us are the localization of nountypes and community commands.

Nountype localization

With the new semantic role argument specifications, command localization simply became a question of translating some strings, which many localizers are used to. After all, we want localizations to affect the presentation of commands, not the logic of the commands. When it comes to nountypes, however, it is quite possible that we would actually want the nountype localization to affect its behavior.

Consider, for example, an imaginary day_of_the_week nountype. In English, this nountype might accept or suggest strings like “Monday” or “Tuesday,” while a French localization would accept “lundi” or “mardi.” More complicated still, consider a date nountype. In English this nountype may have custom logic to parse strings like “June 1st” while another language may have to parse very different kinds of strings. These nountype localizations thus involve not just string translations, but actual changes in their logic, making the po format approach we took to command localization a poor fit.

Making nountypes localizable, however, will make Ubiquity significantly more “natural” for many users. In the coming weeks and months we’ll be discussing and debating different options to accomplish this.

Community command localization

Even though the file format and infrastructure for command localization itself has been fleshed out with Ubiquity 0.5, the distributed nature of all these community commands adds an additional complication. Do we want community command localizations to be completely distributed, or should they be centralized? If they’re distributed, how do you find them? These are the types of questions we’ll need to ask and answer. The ease of creating a new Ubiquity command and sharing it with the world is a huge asset of the platform, so we’ll definitely be thinking about how best to localize these community commands as well. In the next day or two I’ll be writing up a more detailed blog post on what we need from a good community command localization solution.

Summary

For the more visually inclined (including myself), here’s a handy diagram to summarize what components are localizable now, what will be in the future, and what this means for Ubiquity users of different languages.

Get Involved

Whether you’re a speaker of an as-yet unsupported language, a veteran localization contributor, or simply interested in seeing how we can offer this natural language interface to more languages and more users, there are lots of ways to get involved. If you have some JavaScript experience and want to teach Ubiquity your native languages’ grammar, read our parser localization tutorial. If you would like to contribute localizations for our built-in commands, there’s a command localization tutorial. To discuss how best to localize nountypes and community commands, or to ask questions about or discuss command and parser localization, join us on the Ubiquity-i18n mailing list.

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. Localizing Ubiquity: commands and nountypes
3. Big Issues and Small Issues with Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.

It’s not too late to add localizations for other languages to 0.5, though. Localizations help make Ubiquity more “natural” for more users, offering a new level of ease and familiarity to the already powerful Ubiquity. We have a new tutorial to help you localize commands.

To help encourage command localization, we now have gettext-style po template files for all the bundled command feeds in the hg repository. You can find these files in the ubiquity/localization/templates directory of the repository, or on our online hg repository.

If you complete some localizations (even incomplete) for your language and would like to submit them into the repository, for the time being, you can post them on this trac ticket.¹

I’ll be looking forward to seeing your localizations! If you have any questions, feel free to ask on the ubiquity-i18n Google group or on irc.mozilla.org#ubiquity. ^^

Related posts:

1. Localizing Ubiquity: commands and nountypes
2. Ubiquity Localization Update
3. A Visual Guide to Community Command Localization

Related posts brought to you by Yet Another Related Posts Plugin.

• Goals for 0.5
  • Parser 2 (on by default)
  • underlying support for localization of commands
  • localization of standard feed commands for a few languages
  • Parser 2 language files for those same languages
• Nongoals for 0.5
  • distribution/sharing of localizations
  • localization of nountypes

The overall goal for this release of Ubiquity is to come up with a format and standard for localization. Localizations in Ubiquity 0.5 will only apply to commands bundled with Ubiquity, and the localization files themselves will be distributed with Ubiquity. In a future release we will tackle the problem of localizations for commands in the wild and truly croud-source¹ this process.

Localization Architecture

The localization of Ubiquity commands will use a gettext-style approach where localization files list key-value pairs for different properties and messages of the commands. For Ubiquity 0.5, where we only deal with the standard command feeds bundled Ubiquity, we can simply place all the localization files in ubiquity/standard-feeds/localization. Localization files are organized by source feed, with one localization file per source feed, per language.

The localizable components of commands will include the names, contributors, and help properties, as well as any localizable strings in the command’s preview() and execute() methods. To make strings localizable in preview() and execute(), they must be wrapped in the localize function, _().²

Other localizable components, like names, contributors, and help will not need to be wrapped in the _() function. In addition, as the localization files can only hold values of strings, for values such as names and contributors, the delimiter | can be used to delimit multiple values.

zoom.names=ズーム|ズームして|ズームする|ズームしろ

The Localization Experience

One tool we have planned to help kickstart the localization process is a tool that will automatically create a template of strings that need localization in a user’s commands. I took a first stab at this tool today. Clicking on the “get localization template” link next to each feed in the Ubiquity command list will give you a template which you can then copy into a text file:

Additionally, instructions will later be added to this page to specify how and where to save localizations to test them or perhaps we can add a button that will automatically save it in the right location.

Open Questions

Localization file formats

There are two kinds of file formats for localizations we are considering: .properties and .po, the native gettext format. As an example, here is the same key-value pair in the two formats:

.properties:

# This is a comment
welcomeMessage=Hello, world!

.po:

#. This is a comment (the . is actually optional)
msgid "welcomeMessage"
msgstr "Hello, world!"

The advantage of .properties over .po is that Mozilla natively supports this format with an XUL/XPCOM interface called stringbundle and it is what is used to localize JavaScript in Firefox itself. We actually already have the _() localization function working with the properties file format, following gomita’s great instructions (Japanese) on how to load properties files in using Mozilla’s native stringbundle tools via JavaScript.

The advantage of .po over .properties is that it is the de-facto standard in localization, particularly in the UNIX world. Lots of great tools have been written for it. The adoption of .po could make Ubiquity localization more accessible for more people. Another advantage is that .po files can have keys with spaces, as I note below.

If we do opt to work with .po files, the two libraries I see out in the wild for dealing with .po files are gettext-js (MIT) and jsgettext (LGPL). While I haven’t looked at the libraries in depth yet, so far jsgettext seems to be the winner, as some sections of gettext-js require the use of the prototype.js library.

A “key” question

In either file format, we need a unique way to refer to each localizable string—a key format. As each localization file refers to a command feed, the first collision we must avoid is the command name. With this in mind, we can come up with some trivial keys for the localizable properties: (here, consider the command hello)

• hello.names
• hello.contributors
• hello.help

However, we run into difficulty when we try to come up with keys for the arbitrary text in previews and executes. For example, for a message like “Hello world!” in the preview, we could simply make the key hello.preview.Hello world! but this may be unruly and be prone to typos. In addition, in .properties files keys cannot have certain characters in them, like spaces, so we would have to make the key something like hello.preview.Hello_world! or, stripping symbols and standardizing case, hello.preview.HELLO_WORLD.

Keys could also get very long with this type of key format, although here again .po files may have an advantage as they can stay relatively more legible even with long keys. One option to deal with this would be to optionally supply a key argument to _() so that it is used instead of the automatic key. For example, suppose the hello command’s preview() included this code:

_('This is a really long greeting message. Hello there!','longmessage')

then a localizer would only have to refer to hello.preview.longmessage, not hello.preview.THIS_IS_A_REALLY_LONG_GREETING_MESSAGE_HELLO_THERE.

satyr points out that some commands use another function to incorporate similar actions and messages in both preview() and execute(). In this case, he argues, it wouldn’t make sense to have to keep both localizations (hello.preview.… and hello.execute.…). He suggests that optional keys (mentioned above) could be used without the preview. or execute. infixes, as in hello.longmessage. By taking out the preview and execute namespacing in the localization keys, though, it becomes the command author’s responsibility to not accidentally use strings named “names”, “help”, etc. that will have unintended consequences.

Conclusion

I hope that this blog post gives people an idea of the progress we’ve made in the localization area and gets people thinking about the challenges we still face. We’d love to get your feedback on the localization format and process in Ubiquity, as well as the open problems of the file format and keys.

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. A Visual Guide to Community Command Localization
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

昨日は文部科学省の先端ITスペシャリスト育成プログラムの学生の為に講義をしました。ちょっと長めの時間だったので、二つのトークに分けてみました。第一部は「オープン」と言うアイデア、特にオープンなデザイン過程の利点について。第二部は未来のインターフェースと国際化とUbiquityの紹介でした。スライドをここにslideshareにあげておきました。

Design processes in the open-source era オープンソース時代のデザインプロセス

Ubiquity: Interfaces and Internationalization インターフェースと国際化

Related posts:

1. Notes from BarCamp Tokyo 2009
2. This week on Ubiquity Parser: The Next Generation
3. Ubiquity presentation at Tokyo 2.0

Related posts brought to you by Yet Another Related Posts Plugin.

I’ve been framing this problem as two subproblems:

1. What will be the data structure of localized commands/nountypes within Ubiquity?
2. How do we distribute/share these localizations?

We’ve mostly been discussing the first problem, weighing the merits of unified objects (with different localized text as different JS properties) as opposed to a gettext-style approach, and noting that our requirements for commands and nountypes may be different. I hope we can discuss the second issue more in the coming week.

Should everything go through the command author? Should localization be centralized through some web tool? Should it be completely distributed like commands currently are? I invite you to join us in this conversation on the Google group. ^^

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. Ubiquity Commands by The Numbers
3. Ubiquity Localization Update

Related posts brought to you by Yet Another Related Posts Plugin.

1. Some commands’ preview’s and execute’s are not working properly (trac #652). This could be an underlying issue with some pipes not rerouted correctly in Parser 2, or it could be that the commands have not been rewritten correctly to take advantage of Parser 2.
2. Flesh out how to localize resources, like commands and nountypes. We started a conversation on this subject a few weeks ago but we never reached a resolution. This blocks issues 3 and 4 below.
3. We need to standardize a format for commands for Parser 2. As noted in last week’s meeting (among other places) Parser 2 will require at least some modification to all commands. Jono and I came up with a simple hybrid format for commands which specify takes and modifiers for Parser 1 and arguments for Parser 2, but until we figure out how exactly the localization of commands will work, we can’t write a definitive standard.
4. Enable nountype localization. While the most popular nountypes used are those that ship with Ubiquity, it is important to come up with a localization process which can apply to custom nountypes as well. Nountype localizations need the ability to either (1) replace the _name only, or (2) replace both the _name and the suggest() logic, as both cases will be necessary.

Given that Big Issue 3 and Big Issue 4 are both dependent on Big Issue 2, there clearly needs to be a continued public discussion of how we should make these resources localizable. I look forward to this discussion taking place at tomorrow’s joint (general + i18n) Ubiquity meeting.

In other news, here are some Small Issues:

1. Add a switch for parser version and language settings: Jono’s already made a space for this in the new “Settings and Skins” page in about:ubiquity. He’s on it. Like a bonnet.
2. Magic word (anaphor) substitution is not yet working properly. This needs to work both when there is an explicit magic word and when there are simply missing arguments.
3. The position of suggested verbs is always sentence-initial (trac #655). This also requires that we can specify whether verb name localizations are sentence-initial forms or sentence-final forms.¹
4. …

Let’s hit the code!

Related posts:

1. Localizing Ubiquity: commands and nountypes
2. Ubiquity Localization: What’s New, What’s Next
3. This week on Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

Foxkeh demos Ubiquity Parser: The Next Generation from mitcho on Vimeo.

There are some details which are not covered in this introductory video, such as how to deal with case marking languages or languages without spaces. Hopefully this’ll inspire some people to play with the demo, though. I’d love to hear your comments! ^^

Related posts:

1. This week on Ubiquity Parser: The Next Generation
2. Adding Your Language to Ubiquity Parser 2
3. Ubiquity Parser: The Next Generation Demo

Related posts brought to you by Yet Another Related Posts Plugin.