We spent a good chunk of time with a team from Invisible Elephant who came in to give us some training on making technical presentations, and then dug into the code on Day 2. It was great to have the geniuses at Mozilla Labs like Atul and Myk show us the latest Jetpack code as well as get the latest project direction from Daniel, Aza, and Nick, from which we could see the amount of careful consideration and effort that’s gone into the Jetpack reboot.

The best part of the whole experience, though, has to be the fellowship with the other Jetpack Ambassadors. The Ambassadors came from all over the world, encompassing Europe, Asia, S. America, and of course N. America. Each are involved with some really exciting projects and have each made a name for themselves in their respective communities. I’ve put together a twitter list of all the Jetpack Ambassadors and the core team members and invite you to follow them.

We also had the greatest number of Ubiquity core developers to have ever been in the same place at the same time, which of course had to be documented.

(More photos can be seen in my gallery.)

I had a fantastic time in MV and it was a shame I could only be there for such a short time. I feel honored to be a part of this group and am looking forward to speaking on Jetpack soon at an event near you!

Related posts:

1. Contribute to Ubiquity! No Coding Required!
2. Spring is for Speaking: JSConf, WordCamp SF, IACL
3. Beginning development with Jetpack SDK 0.2

Related posts brought to you by Yet Another Related Posts Plugin.

Being a longer time slot than I previously have used to talk about Ubiquity, I decided to dedicate a good portion of the talk to Jetpack. Being outside of Mozilla for the past few months, this gave me an opportunity to get reacquainted with the Jetpack APIs. I myself was impressed by how easy it was to develop a quick Jetpack. I ended up preparing two to live-code during the talk: one called Helvetica which, with one click, replaces all fonts on the current page with Helvetica; and You Are Here which uses an open API from IPinfoDB to display the physical location of the domain you are currently visiting in the status bar. Both are now on the Jetpack Gallery.

Unfortunately there was a bit of a snowstorm leading up to the event, but there was still a nice turnout and I got to meet some fantastic people there. Ken Shoemake of slerp and quaternion fame came up to me after my talk and said “the Ubiquity parser reminded me of the dancing bear… it’s less surprising that it works well as that it works at all.” I also enjoyed the other great presentations in the technology track, covering the virtues of REST and basic iPhone development.

Mashup the Browser with Ubiquity and Jetpack

Related posts:

1. Jetpacking in Boston
2. Jetpack Ambassadors in MV
3. Spring is for Speaking: JSConf, WordCamp SF, IACL

Related posts brought to you by Yet Another Related Posts Plugin.

Time working for Mozilla: 6.5 months

Mozilla-related blog posts written: 69

Academic papers written on Ubiquity: 1

Ubiquity presentations given: 5

Screencasts made: 8

Most popular video on Vimeo: Ubiquity 0.5 日本語紹介ビデオ, the Japanese Ubiquity 0.5 introduction video: 2252 views

Languages Ubiquity commands and parser now support: 6

Commits to the Ubiquity repository: 492

Other web projects started during this period: 2+ (Ten Grand Is Buried There, HookPress)

TechCrunch references: 2 (1, 2)

Countries worked in: 2

Mythical Kiwis worked with: 1

References to bugs I introduced as “glitcho”s: 1

Extremely disturbing homages to me and Django: 1

Friends made; experience gained; lessons on Open-ness learned; personal growth: priceless enumerable

Thanks to all who made this experience amazing, beginning with Aza, Jono, Atul, Blair and the rest of the Labs team; intern extraordinaire Brandon; the always thoughtful and friendly Mozilla Japan team; and of course the fantastic Ubiquity community! Please visit me in Boston—I should be around for a while.

Related posts:

1. Jetpack Ambassadors in MV
2. Report from Mozilla Party JP 10!
3. This week on Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

This week I implemented a new way to customize and extend Ubiquity commands: CmdUtils.CreateAlias.

The use case for and importance of CreateAlias

CreateAlias lets you easily create a special-case alias of another, more generic verb. Ubiquity comes bundled with useful verbs like translate and search which can be used for a number of different uses based on their arguments. In some cases, and in some languages, though, typing out translate to English or search with Google is unnatural, though, as there is a more succinct and direct way to make that request. For example, in English one could say “anglicize” or “google”, respectively, for the verbs and arguments above. Indeed, in order to support both search with Google and google, Ubiquity traditionally has implemented two different verbs, search and google, which duplicate functionality and code.

CreateAlias lets us create such natural aliases without repeating ourselves. We can easily create an anglicize verb which, in one word, does the work of translate to English, or google which is semantically equivalent to search with Google.

These sorts of aliases become particularly important in our perpetual quest to internationalize Ubiquity. One discussion that came up early on on our Ubiquity-i18n list is the fact that not all languages have the verb “Google”: in many languages it is necessary to explicitly say “search with Google”. Moreover, other languages may have other domain-specific verbs which English doesn’t have either. Maybe some language has a special verb for “email with Hotmail” or “map Denmark”. Who knows? With CreateAlias we can easily enable such localizations based on the more generic commands bundled with Ubiquity.

Creating an alias

CreateAlias was designed to be incredibly simple to use. Here’s an example that will be bundled (but not installed by default) in Ubiquity:

CmdUtils.CreateAlias({
  names: ["anglicize"],
  verb: "translate",
  givenArgs: { goal: "English" }
});

As you see, this syntax is incredibly straightforward. There are two required properties, names, an array of names for the alias, and verb, a reference to the target verb that this alias should use.¹

The alias can also have a givenArgs property which is a hash of pre-specified arguments with their semantic roles. Because translate takes three arguments (an object text, a goal language, and a source language) but we have pre-specified a goal in the givenArgs, the new anglicize command will only take two arguments: the object text and a source language. Of course, if you specify no givenArgs, you’ll get a simple synonym without having access to the original verb’s code.

As you see, the preview of this command is simply the preview of the translate verb. Its preview and execution is just as if you had entered translate こんにちは to English.

Just like other commands created with CreateCommand, the object specifying the alias can also have properties like help, description, author information, and so on. I used the icon property to add a Union Jack to it so that it was easily identifiable.

Bonus: using CmdUtils.previewCommand and CmdUtils.executeCommand

On the road to implementing CreateAlias, I also implemented the CmdUtils.previewCommand and CmdUtils.executeCommand functions. The majority of this code comes from previous work by Louis-Rémi Babé, though I adapted it to the modern Ubiquity system. Using previewCommand and executeCommand you can take advantage of the preview or execute functionality of another command. In the new alias-commands feed I included a command called germanize which essentially is a straightforward analogy to anglicize, seen above, but using these functions within a CreateCommand. While CreateAlias is much more straightforward for simple aliases, for more complex subcommands where you would like to adapt another verb’s execution or preview, or only take one of those but re-implement the other part, previewCommand and executeCommand are the way to go.

Related posts:

1. Ubiquity Localization: What’s New, What’s Next
2. Rolling out the Roles
3. Localizing Commands for Ubiquity 0.5

Related posts brought to you by Yet Another Related Posts Plugin.

I recently have begun giving serious thought to what command chaining might look like in Ubiquity and the various considerations which must be made to make it happen. The “command chaining,” or “piping,” described here always involves (at least) two verbs acting sequentially on a passed target—that is, the first command performs some action or lookup and the second command acts on the first command’s output.

A few days ago I penned some initial technical considerations regarding command chaining. In this post I’ll be point out some linguistic considerations involved in supporting a natural syntax for chaining.

Simple syntaxes: sequential vs embedding strategies

When it comes to creating a natural language interface, there’s always a decision to make between requiring a certain kind of input, or working a little harder to understand the user’s natural input. From an implementation point of view, adopting certain programmatic conventions is of course simpler and to this end, there have been a couple different “unnatural” command chaining syntaxes suggested. While these both go against Ubiquity’s basic tenet of natural syntax — that is, to not introduce rules which contradict the user’s natural language — which gives Ubiquity its strengths of usability and memorability, I’ll entertain them here as they illustrate two different structural relationships that we will want to consider.

The first suggestion is to adopt the shell pipe (|), which would lead to input such as

1

translate hello to Spanish | email to Jono

While this itself is pretty unnatural unless you speak shell, note that this syntax is similar to the more natural “, and” syntax, yielding translate hello to Spanish, and email to Jono, which we will consider below. I’ll refer to this strategy as the sequential strategy.

Another very interesting proposal by Alex Fritze is to embed each subordinate computation into an argument position, marked by parentheses. This could also be parsed relatively straightforwardly by writing a noun type which first checks for parentheses and then runs the content of the argument through another ParseQuery.

2

email (translate hello to Spanish) to Jono

I’ll refer to this pattern as the embedding strategy.

Sequential and embedding strategies in natural language

What’s interesting about the two proposals above is that both strategies are seen in natural language. The sequential strategy could correspond to the following linguistic phenomena:

1. coordination: a non-hierarchical joining of two or more clauses, often marked by a conjunction. Here’s an example from English:
   • “[I made a sandwich] and [you will eat it]” where [] represent clause boundaries. Here, “and” is the conjunction.
2. serial verb and converb constructions: a joining of multiple verbs or verb phrases within a single clause, with shared subject and tense/aspect values, with no particular conjugation or delimiter between them. Such constructions are common in many African and east Asian languages. Here are two examples:¹
   

• A converbal construction in Japanese:
  

  3 4 5

  僕は サンドイッチを 作って 食べる boku-wa sandiʔchi-o tsuku-ʔte tabe-ru I-TOP sandwich-ACC make-CON eat

  
  “I (will) make a sandwich and eat [it].” (Here, `TOP` = topic, `ACC` = accusative, `CON` = converbal ending)[^3]
• A serial verb construction in Mandarin Chinese:
  

  6 7 8

  我 作 三明治 吃 wǒ zùo sānmíngzhì chī I make sandwich eat

  
  “I (will) make a sandwich and eat [it]” or “I (will) make a sandwich [in order to] eat [it].”

Note that in both the converb and serial verb construction, the second verb (eat) takes shares its object (sandwich) with the first verb and there is no need for a pronoun such as “it” to introduce that argument as it is with coordination, above.²

The embedding strategy is observed in natural language as well, in the form of the following phenomena:

1. embedded clauses: a sentence is itself the argument of another verb. Example:
   

• “John says [he likes sandwiches].”

Embedded clauses, however, clearly have no relation to command chaining and does not require our attention. 2. relative clauses: a partial sentence³ is attached to a noun in order to describe it or distinguish it from other possible referents. Example:

• “You ate the sandwich that I made” where “sandwich” is called the “head” of the relative clause, and “I made” is what I here call the “partial sentence” (see footnote). The “relative clause” is used here to distinguish “the sandwich that I made” from other sandwiches.

The natural syntax of chaining

So which strategy is used in complex natural language commands: the sequential strategy or the embedding strategy? Both the sequential strategy and embedding strategy can be involved with commands:

9
10

[Make a sandwich] and [eat it]!
Eat (the sandwich that I made)!

These two commands do not mean the same thing, though, and only (9) is the kind of command we would want to give Ubiquity. The problem with relative clauses, as in (10), is that it presupposes the existence of the sandwich in the context. If we both know you just made a sandwich, saying (10) is perfectly appropriate, but out of the blue it doesn’t make sense. For this reason, only the sequential strategy is used in the natural syntax of chaining.

Parsing the sequential strategy

In natural language, unlike the initial simple proposals laid out above, there is often no clear delimiter marking the boundary between the two parts in a sequential relation (e.g. examples (3) and (6) above, particularly given that neither Japanese and Chinese normally break words with spaces). How would we parse a sequential string of commands?

Let’s assume for our purposes here that we can identify find all verbs within the input string.⁴ Parsing a sequential strategy string is not particularly difficult if we can also assume that the verb in any particular language is either always verb-initial or always verb-final. Let’s look at both cases:

• Always verb-initial: Mandarin Chinese:
  

11
12
13

翻譯       hello 到  西班牙語   送    給  Juanito
fānyì     hello dào xībānyáyǔ sòng gěi Juanito
translate hello to  Spanish   send to  Juanito

“Translate hello to Spanish [and] send [it] to Juanito” 1. find every possible verb:
翻譯hello到西班牙語送給Juanito 2. as every verb marks the beginning of a sentence, we now have our two commands: “翻譯hello到西班牙語” (translate hello to Spanish) and “送給Juanito” (send to Juanito). * Always verb-final: Japanese

14
15
16

helloを スペイン語に 訳して Juanitoに 送って
hello-o supeingo-ni yakus-ite Juanito-ni oku-ʔte
hello-ACC Spanish-DAT translate-CON Juanito-DAT send-CON

“Translate hello to Spanish [and] send [it] to Juanito” 1. find every possible verb:
helloをスペイン語に訳してJuanitoに送って 2. as every verb marks the end of a sentence, we now have our two commands: “helloをスペイン語に訳して” (translate hello to Spanish) and “Juanitoに送って” (send to Juanito).

For languages where there is a clear conjunction between the two commands, such as English “and”, we can also use that conjunction as a delimiter as well. We then simply execute the first command and then execute the second with the first command’s output in its interpolation context. This way the output of the first command will be picked up both by an overt pronoun such as “it” in the second command and without it, such as in the Chinese and Japanese examples above.⁵

The only potential problem with this approach is in the case of languages where some commands are verb-initial while others are verb-final. I note that such languages do exist in a previous blog post, Where’s The Verb. In these languages, commands can be expressed by more than one verb form (such as infinitive, imperative, subjunctive, etc.) and some of those verb forms are sentence-initial while others are sentence-final. Here’s one such example from German:

“search hello with google” (German) 1. Infinitive: hello mit google suchen 2. Imperative: suche hello mit google

Here the verb for “search” is “suchen” (infinitive) or “suche” (imperative). I know that this same type of phenomena occurs in other Germanic languages such as Dutch with infinitive and imperative and also other languages such as Modern Greek with infinitive and subjunctive forms. If you are a speaker of one of these lanuages (German, Dutch, Greek, etc.) I would love to know whether you can chain verb-final and verb-initial commands together.

Conclusion

In this blog post I examined command chaining in natural language, focusing on data from English, Mandarin Chinese, and Japanese, which exhibit three linguistically different approaches to chaining. What we found is that the sequential strategy—that of listing the commands one by one, in order of execution—is what is used in natural languages, rather than any sort of embedding. This fact, combined with the fact that our parser can recognize every available verb, offers a simple approach to doing a naive parse of natural command chains in most languages, even without explicit delimiters.

In a final installation of this series on “exploring command chaining,” I hope to consider how the Ubiquity interface itself could present command chains and aid in its input.

Related posts:

1. Exploring Command Chaining in Ubiquity: Part 1
2. Command Chaining with Oni?
3. Ubiquity in Italian

Related posts brought to you by Yet Another Related Posts Plugin.

In order to facilitate this discussion, we created the Persistence tool. With the Persistence tool you can quickly try out new design and interaction ideas, mocking things up with some simple jQuery-powered JavaScript and CSS and see your changes live. The Persistence tool is bundled with our latest Ubiquity beta (install link).

The Ubiquity Persistence Project: exploring a persistent Ubiquity in the toolbar from mitcho on Vimeo.

I just put together a screencast introducing the initiative, demoing the Persistence tool, as well as talking about this project’s relation to the ongoing work on Taskfox. We’ll look forward to your comments and designs!

Related posts:

1. User-Aided Disambiguation: a demo
2. Count command for Ubiquity
3. Exploring Command Chaining in Ubiquity: Part 1

Related posts brought to you by Yet Another Related Posts Plugin.

Today I hope to cover some of the technical considerations required in implementing command chaining in Ubiquity, and I will follow up soon with a blog post on the linguistic considerations required as well.

Technical considerations: hooking the pipes together

I’d first like to lay out some technical challenges and questions. These can be broken into two different categories: (1) how the parse and display of suggestions is affected and (2) how the execution is affected.

Matching inputs and outputs

We’ll first consider how command chaining may affect the parsing. Ubiquity commands each specify the types of argument inputs that it expects using different noun types, such as noun_arb_text which accepts anything, noun_type_number which accepts numbers, or noun_type_language, which takes the name of a language. For example, the translate verb takes maximally three arguments: a noun_arb_text object, a noun_type_language goal (the language to translate into), and a noun_type_language source (the source language). In implementing command chaining, it will be necessary to identify the appropriate noun types for the output of a command.

The first question we must address here is “what is the chaining output of a command”? Is it the preview text? Some text output from the execution?

Big fish eat da lil fish by joemud, CC-SA-NC

To put this question into perspective, we note that Ubiquity commands can be broadly classified into two types: lookup and action execution. Here’s a classification which I believe to be exhaustive:²

In light of this classification I believe we can say that lookup commands are much more likely to be the first verb in a command chain—conversely, chains such as email hello to Blair and then do ... or twitter hello and then ... are quite unlikely. What is much more likely is for the first verb to be a lookup function.

Thus in the same way that not all commands have a useful execution³ perhaps only lookup commands will have a chainable output: the results of the lookup. Even with this restriction, we will most likely need to implement a new “chainable output” method or getter in these commands. This means that commands will need to opt-in to become chainable, but I believe this is a necessary evil.

The second question we must address is “when do we establish the noun type of a command’s chainable output?” One unsung but crucial feature of the way Ubiquity works now is that suggestions’ previews are not computed until that suggestion is selected (except for the first suggestion, which in most skins gets previewed immediately). Should we wait for all of the first verbs’ chainable output to be computed and then run them through the noun type detection system? Or should verbs with chainable output also a priori specify what noun types their output will be?

Both of these approaches have their problems. If we compute the chainable output of the first verb, run a noun type detection on it and then suggest the full combination if it matches what the second verb was expecting, this will have clear performance implications, not to mention that it could greatly complicate our parsing algorithm. While the latter approach doesn’t have these performance implications, it does mean that it will have to list (by name or reference) the noun types that will match its output, meaning that if a command author is unaware of someone else’s noun type, that chain will be impossible, even if the chainable output itself does indeed match that noun type. The a posteriori approach would never have this issue. What other benefits or problems do you forsee with either of these approaches? Is there another approach which avoids these pitfalls?

(A)synchronous composability

Once we have the noun types, parsing, and suggestions down, all that remains is to compute the previews and implement the composite execution. Since the Ubiquity command manager already wraps the preview and execute functions in a wrapper to facilitate localization, among other uses, it would be easy to make the command manager compose asynchronous processes pseudo-synchronously. No major changes should be necessary to do the previews and executions, though, again, there will be a performance cost.

Conclusion

There are a number of technical questions which must be answered, mostly in the parsing/suggesting stage. The key questions to answer are:

1. What is the chaining output of a command?
2. When do we establish the noun type of a command’s chainable output?

I’ll make another post soon on the linguistic considerations necessary in making command chaining happen in a natural fashion.

Related posts:

1. Exploring Command Chaining in Ubiquity: Part 2
2. Command Chaining with Oni?
3. Ubiquity Commands by The Numbers

Related posts brought to you by Yet Another Related Posts Plugin.

The short story:

Inspired by some comments by Blair, yesterday I was able to make significant (roughly 100%) performance gains in Parser 2, resulting in 40-60% faster parses, depending on the query. This change has been committed and will be released as part of our forthcoming minor update, Ubiquity 0.5.4. Yay!

The long story: asynchronous parsing

Given that parsing in Ubiquity, combined with the post-parse of displaying suggestions, takes a good few dozen milliseconds, it is important to make sure it doesn’t block the main execution thread in order for the UI to stay responsive throughout. In other words, we needed to make it asynchronous.¹

When we began work on Parser 2 a few months ago, Blair stepped up to the plate to make it run asynchronously. For various reasons, the parser doesn’t run in a Worker thread for truer threading. Instead, what we did was to put the parser’s steps into a generator called _yieldingParse. The keyword yield is scattered in points throughout this generator.

function _yieldingParse(...) {
  // step 1
  ...
  yield true;
 
  // step 2
  ...
  {
    ...
    yield true;
  }
  ...
 
}

We then iterate over a _yieldingParse object in a function called doAsyncParse. Each time we go invoke doAsyncParse, it invokes next which advances from the last yield point of the parse to the next one. doAsyncParse checks after each step whether we should keepworking or not and then asynchronously advances to the next step by calling itself with a setTimeout. (Note the code below is a simplification.)

var parseGenerator = _yieldingParse();
 
function doAsyncParse() {
  var ok = parseGenerator.next();
  if (ok && keepworking)
    Utils.setTimeout(doAsyncParse,0);
}
 
// get this party started
Utils.setTimeout(doAsyncParse,0);

The more often we yield, the more responsive the UI would be. However, there is a certain overhead to yielding each time due to the setTimeouts we call. This point hit home when Blair told me the other day that the parser was much faster without any of the setTimeouts. Indeed, in my own testing running queries completely synchronously (replacing out all the setTimeouts), parses would run in roughly half the original time. However, by virtue of being completely synchronous, the parser would then completely lock up the UI.

I thus set out to strike a balance between performance and responsiveness by taking out and moving some of the yields in our _yieldingParse (#856).

Tests, tests, tests

Keeping this in mind, I ran a number of tests as I proceeded with my “refactoring.”

Here are some final parse time results:²

Four different queries (“hello to span”, “goo hello”, “22.7”, “tw as test” with a selection context of “hello world”) were run using each algorithm. The blue bar is the performance of the algorithm prior to adjustment of setTimeouts—that of Ubiquity 0.5.3. The gold bar is the time from a completely synchronous parse where all the setTimeouts were replaced. This algorithm completely locks up the UI, but is clearly the fastest, and should be seen as a baseline for all other yielding optimizations. The green bar is our new algorithm. As you can see, the parser is now roughly twice as fast.

Moreover, the average time difference between yields went from 0.7ms to 3.9ms which still should be no problem in terms of responsiveness.

Cancellability

This doAsyncParse is also the key for cancellability of the query. When a user changes the input or closes Ubiquity while a query is in progress, we want to cancel that query as soon as possible so the user and UI can advance. keepworking is set to false when the query is cancelled, so making sure that we yield early enough and often enough in the parse are important for issues like keystrokes being lost when typing too fast.

While the parser was indeed yielding often enough (in fact, more often than necessary) before, I noticed that our first yield was often 15-20 ms into the parse. This was because step 1 of our parse derivation was happening outside of the doAsyncParse loop. By moving this into that loop, I was able to bring this initial synchronous time down to around 10 ms. Of course, setting up the parse generator itself takes a little overhead, so this can never go down to 0, but perhaps this will improve the keystroke issue as well. I’d love to get anecdotal feedback on whether this update improves the disappearing keystrokes issue from 0.5.4 users.

Related posts:

1. Ubiquity Parser: The Next Generation Demo
2. This week on Ubiquity Parser: The Next Generation
3. Changes to Ubiquity Parser 2 and the Playpen

Related posts brought to you by Yet Another Related Posts Plugin.

Granted, these are only Ubiquity’s interface strings (for example, the about and settings pages)… the parser localization and command localization have their own processes.² But this is still a tremendous accomplishment!

Hopefully we can roll some of these complete or almost-complete interface localizations with Ubiquity 0.5.4 which is a minor bugfix update coming soon. If you would like to get involved with localizing the Ubiquity interface strings into your language, get a BabelZilla login and sign up on the Ubiquity project page. Thanks again to our rockin’ localizers!

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. A Visual Guide to Community Command Localization
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

It’s time for one more installment of Nountype Quirks, where I review and tweak Ubiquity’s built-in nountypes. For an introduction to this effort, please read Judging Noun Types and my updates from Day 1 and Day 2.

Today I ended up spending most of the day attempting to implement (but not yet completing) major improvements to the geolocation-related nountypes whose plans I lay out here.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

noun_type_geolocation

noun_type_geolocation is the nountype used by the weather command for its location argument in input like “weather near Chicago”. The neat feature of noun_type_geolocation is that it has a smart default value which uses Firefox’s geolocation system to give you your current location by default, so I can enter “weather” and get the suggestion “weather near Broomfield, Colorado” (not completely correct, but close enough for the weather). Otherwise, however, noun_type_geolocation does not do too hot… for any input you give it, it’ll just accept it with a score of 0.3, much like noun_arb_text. We could do better.

One issue with this noun_type_geolocation is a conceptual one. Is this nountype supposed to accept only municipalities? Countries? Or should it accept landmarks or addresses as well? Part of the issue is that it’s only used by one built-in command in Ubiquity now, weather. But to be called a general “geolocation” nountype, its output should not be specific to weather’s usage, which is to throw the result at the Weather Underground API.

I propose that we change this to be something like noun_type_geo_town and also make similar nountypes like noun_type_geo_country, noun_type_geo_region, going all the way down to noun_type_address (which already exists—see below). All of the nountypes in this family could use a geocoding API such as Google’s or Yahoo’s. Their data properties could include all of this geocoded geographic data (in English) and also the latitude/longitude coordinate data.

The weather command could then accept noun_type_geo_town but, as some municipalities are not in Weather Underground or, for some countries, it is only as granular as administrative districts, we could just display the results of the geocoding API but then give Weather Underground the geocoded latitude/longitude data.

noun_type_async_address

noun_type_async_address attempts to do exactly what I’ve laid out above for the most granular level: that of geolocations with data all the way down to the street level. This is the nountype which is used for the built-in map command and uses the Yahoo geocoding service to accomplish this. Let’s see what kinds of results it returns:

input	suggestion
mitcho	mitcho	0.5
grenada	grenada	0.9
jono	jono	0.9
mountain view	mountain view	0.9

Let’s lay out some immediate quirks:

1. All scores are either 0.5 or 0.9. In general, if the Yahoo API returns some geocoded interpretation, it gets 0.9, but otherwise it accepts everything with 0.5.
2. The results that came back from the Yahoo service doesn’t add any useful information like the country or administrative region. Even the case stays lowercase.
3. Since when is Jono a location!? I’ll get back to this later.

For starters, the Yahoo! Maps API terms of service dictate that we can’t use its geocoding service if we’re not also displaying Yahoo maps, so I rewrote it using the Google API which also had the advantage of offering JSON output.

One quirk of the Google Geocoding API, though, is that all of the resulting municipality names are only in English. Try for example queries for Wien or 東京 (Tokyo). Since we want our suggestions to only add information to our input, not replace the input entirely (and especially not in another language), we’ll then only take results which have the input as an initial substring. On the other hand, if none of the results have the input as a proper prefix of the return value, we will take the geocoding information from the first result but with the original input as the display text. Such results will have a markedly lower score.¹

As this is the address nountype, we’ll penalize results which do not have detailed information such as street address or town-level information. All of this is very easy to judge as every result from the API has a geocoding accuracy value.

The best laid plans of mice and men…

I spent a good few hours this afternoon and evening attempting to implement this new family of nountypes, including this new nountype_geo_address, but also nountype_geo_subregion, nountype_geo_region, and nountype_geo_country. Some of the quirks of the weather and map commands, however, have prevented me from completely replacing the legacy noun_type_address and noun_type_geolocation described above. I hope to continue this work again soon and actually make this transition, ideally before 0.5.2.

Look forward to one (or maybe two?) more episode(s) of Nountype Quirks where I hope to definitively explain, analyze, and tweak matchScore, the scoring algorithm which underlies the majority of the nountypes in Ubiquity. As always, I look forward to your comments and feedback.

Bonus: Where’s Jono?

It turns out that noun_type_async_address was recognizing “Jono” as an address because Jono is actually a location afterall! Not only that, but Jono is in Japan!!

You clearly can’t take Japan out of Jono, but it turns out you can’t take Jono out of Japan either.

Related posts:

1. Nountype Quirks: Day 2
2. Nountype Quirks: Day 1
3. Judging Noun Types

Related posts brought to you by Yet Another Related Posts Plugin.

Today I’m continuing the process of reviewing and tweaking all of the nountypes built-in to Ubiquity. For a more respectable introduction to this endeavor, please read my blog post from a couple days ago, Judging Noun Types and my status update from yesterday, Nountype Quirks: Day 1.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

noun_type_twitter_user

Let’s begin again by considering the suggestions and scores that a variety of different inputs to this nountype return and see what quirks we find.

To test this nountype, I made sure I had logged into Twitter once with the login mitchoyoshitaka.

input	suggestion
mitcho	mitchoyoshitaka	0.85
	mitcho	0.5
mitchoyoshi	mitchoyoshitaka	0.94
	mitcho	0.5
test	test	0.5
テスト	none
hello world	none
@test	none

As nountypes go, this is looking pretty good. For usernames which look like logins we’ve saved before, we’re using matchScore to get decent differential scores.¹ It’s even ruling out impossible twitter username strings, according to Twitter’s own restriction:

One possible improvement we could make is to let @ strings be accepted. I went ahead and made this improvement. The initial @ will be stripped off and then will be checked as normal, but the final score will receive a slight boost using an nth root formula. The twitter command was also updated to deal with inputs with and without the initial @.

input	suggestion
mitcho	mitchoyoshitaka	0.85
	mitcho	0.5
@mitcho	@mitchoyoshitaka	0.88
	@mitcho	0.57
test	test	0.5
@test	@test	0.57

Although the noun_type_twitter_user nountype is currently most used by the built-in twitter command to specify the user’s username, in theory it could also be used for example in a command which pulls up another user’s tweets. With that in mind, perhaps in the future we could check the browser history and/or bookmarks for entries of the form http://twitter.com/... and suggest those as well (trac #846).

noun_type_number

input	suggestion
text	none
0.5	0.5	1
0.5.1	none

This nountype has an incredibly simple job and does it with ease. I’m going to leave it alone.

noun_type_date and noun_type_time

noun_type_date and noun_type_time both use the magical Date.parse method to parse date- and time-like strings. Let’s first take a look at some of its suggestions:

input	date suggestion	time suggestion
June 8th 5pm	2009-06-08	05:00 PM	1
5pm	2009-07-30	05:00 PM	1
5	2009-07-05	12:00 AM	1
June 8th	2009-06-08	12:00 AM	1
today	2009-07-30	12:00 AM	1
now	2009-07-30	02:40 PM	1
5pm is a good time	none	none

The quirks in these outputs can be summed up into these two factors:

1. There is no differential scoring at all.
2. Both nountypes parse the input with Date.parse and then just spit out the date or time components of the result. Thus time-only inputs get the default date and date-only inputs get the default time with equal scores.

I just rewrote both nountypes and also added a new noun_type_date_time. Here are some of the features of the new implementation:

1. If the input only contains digits and spaces, it is marked down.
2. With the exception of the outputs ‘today’ and ‘now’, if the resulting Date object’s date is today, its date suggestion is scored lower; equivalently for time being the default value, “12:00 AM”.
3. Scores (with the exception of ‘today’ and ‘now’) which are shorter than the output string get a slight penalty. This factor reflects the intuition that a longer output than input means some generic information was added and thus there is less confidence in the output.

Here’s what some of the inputs give now:

input	suggestion
June 8th 5pm	date: 2009-06-08	0.7
	time: 05:00 PM	0.7
	date_time: 2009-06-08 05:00 PM	0.86
5pm	date: 2009-07-30	0.27
	time: 05:00 PM	0.81
	date_time: 2009-07-30 05:00 PM	0.49
5	date: 2009-07-05	0.53
	time: 12:00 AM	0.19
	date_time: 2009-07-05 12:00 AM	0.34
June 8th	date: 2009-06-08	0.95
	time: 12:00 AM	0.35
	date_time: 2009-06-08 12:00 AM	0.58
today	date: 2009-07-30	1
	time: 12:00 AM	0.45
	date_time: 2009-06-08 12:00 AM	0.7
now	date: 2009-07-30	0.7
	time: 12:00 AM	1
	date_time: 2009-06-08 04:34 PM	1

In addition, looking to the future we’d like to make nountypes localizable as well, and these two nountypes in particular will surely require some good thinking and planning to make localizable.

noun_type_email and noun_type_contact

noun_type_email and noun_type_contact are two closely related nountypes. noun_type_email simply validates email address-looking strings, while noun_type_contact will return the noun_type_email suggestions and additionally return contacts from GMail if available.

The first thing to note is that I’ve often found the GMail contact lookup to be finicky in my own use. Reading through the code, I discovered the solution: GMail must either be open in a tab or you must use the “stay signed in” option and close the GMail tab.² With this mystery solved, and some code cleanup done to this contact fetching, let’s take a look at some example suggestions: (suggestions overlapping with noun_type_email are not listed here)

input	suggestion
aza@m	aza@mozilla.com	0.42
jono	jdicarlo@mozilla.com	0.28
jdicarlo	jdicarlo@mozilla.com	0.19

In general, we see that these scores all look pretty poor. In particular, though, note that the “jono” input yielded a higher score for the same suggestion than “jdicarlo”, even though “jdicarlo” is longer and thus, intuitively, has more informational content and should maybe do better. Digging into the code I realized why this is. It was computing the scores by comparing “jono” and “jdicarlo” not simply to “Jono DiCarlo” and “jdicarlo@mozilla.com” respectively, but to the combined string “Jono DiCarlo <jdicarlo@mozilla.com>”. Now with this change in place, both the email address and name are analyzed individually and, due to the way nountype detection works in Parser 2, no duplicates are returned. Here are the updated results:

input	suggestion
jono	jdicarlo@mozilla.com	0.83
jdicarlo	jdicarlo@mozilla.com	0.85

That’s much better!

Now let’s consider the suggestions from noun_type_email. Here are what they originally looked like:

input	suggestion
bpung	none
bpung@m	bpung@m	1
bpung@mozilla.com	bpung@mozilla.com	1

noun_type_email is based on a very robust regular expression for RFC 2822. Unfortunately this means that it completely rules out strings such as “bpung” which could be a proper prefix of an email address—something that I’ve advocated for avoiding before (see footnote 2 of Judging Noun Types). Moreover, due to a quirk of how nountypes based on regular expressions are scored, all results are given the score of 1.

I just committed a change so that this behavior is improved. The new version accepts strings which match the username part of the email address spec sans @ and domain, but with a great score penalty.³ Moreover, domains which do not have a final label (the top level domain) with more than one letter (unless it’s an IP address) or do not have any periods (.) in the domain will be penalized as well. Here’s what the same inputs produce now:

input	suggestion
bpung	bpung	0.3
bpung@m	bpung@m	0.8
bpung@mozilla.com	bpung@mozilla.com	1

Same time, same channel

I hope this post sheds light on the many changes I made together as well as the underlying thought process. If you don’t agree with any particular fix or analysis, please comment! I’ll be back again tomorrow with another installment of Nountype Quirks. Stay tuned!

Related posts:

1. Nountype Quirks: Day 3: Geo Day
2. Nountype Quirks: Day 1
3. Judging Noun Types

Related posts brought to you by Yet Another Related Posts Plugin.

Today I began the process of going through all of the nountypes built-in to Ubiquity using the principles and criteria I laid out yesterday—a task I’ve had in planning for a while now. As I explained yesterday, improved suggestions and scoring from the built-in nountypes could directly translate to better and smarter suggestions, resulting in a better experience for all users. Here I’ll document some of the nountype quirks I’ve discovered so far and what remedy has been implemented or is planned.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

noun_type_percentage

Here’s what a few different inputs originally returned:

input	suggestion
20	20%	1
20%	20%	1
0.2	20%	1
0.2%	20%	1
20.0	2000%	1
2 hens in the garden	2%	1

Let me highlight a couple obvious quirks:

1. In certain cases, where the numerical expression includes a decimal and is less than one, it is interpreted as a proportional, rather than percent, value, e.g. “0.2” → “20%”. “0.2%” is not even an option. This is the case even when explicitly adding a % sign.
2. All suggestions, including those where the numeral was extracted from a long string of text (e.g. “2 hens in the garden”), get the same score of 1.

I just committed a fix so noun_type_percentage now…

1. Counts the number of characters in the input which match [\d.%] and caps the score by (number of acceptable characters)/(length of input).
2. Strings which do not include “%” get a 10% penalty.
3. In the case of decimals less than 1 without a % sign, the proportion interpretation is also suggested (e.g. “0.2” → “20%”) in addition to the original suggestion (“0.2%”), but with a slight penalty.

Here is what they now return:

input	suggestion
20	20%	0.9
20%	20%	1
0.2	0.2%	0.9
	20%	0.81
0.2%	0.2%	1
20.0	20%	0.9
2 hens in the garden	2%	0.05

noun_type_tag

Here’s what a few different inputs originally returned. Keep in mind that currently in this test profile, the preexisting tags are “animal”, “help”, “test”, and “ubiquity”.

input	suggestion
animal	animal	0.3
mineral	mineral	0.3
anim	animal	0.7
	anim	0.3
help, test, ubiq	help,test,ubiquity	0.7
	help,test,ubiq	0.3
google, yahoo, ubiq	google,yahoo,ubiquity	0.7
	google,yahoo,ubiq	0.3
google, , yahoo	google,yahoo	0.3

Here are a few of noun_type_tag’s quirks:

1. There are only two scores ever given out: 0.3 and 0.7.
2. Only the last tag in the list and whether it exists or not is taken into account.
3. When the last tag is incomplete, the completion is suggested with a higher score, but if the last tag is exactly equal to an existing tag, it gets the lower score.

Ideally, we want noun_type_tag to look at each of the tags given to it, with higher scores for when there are more preexisting tags and fewer new ones. Keep in mind, though, that we only have to suggest the completion of the very last tag as that may be one where the user hasn’t completed typing yet… for earlier tags, we can assume (safely or not) that the user placed the comma where they meant to. We can’t teach Ubiquity to read minds, after all.¹

With this in mind, I just made a change to noun_type_tag which aims to follow these principles. The basic idea is that we start with a base score of 0.3 but then raise it via nth root for every tag in the sequence which is preexisting. Here’s what the same inputs return now. Recall that the preexisting tags are “animal”, “help”, “test”, and “ubiquity”.

input	suggestion
animal	animal	0.55
mineral	mineral	0.3
anim	animal	0.55
	anim	0.3
help, test, ubiq	help,test,ubiquity	0.86
	help,test,ubiq	0.74
google, yahoo, ubiq	google,yahoo,ubiquity	0.55
	google,yahoo,ubiq	0.3
google, , yahoo	google,yahoo	0.3

noun_type_awesomebar

input	suggestion
moz	http://www.mozilla.com/	0.8
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.8
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.8
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.8

There are a couple quirks here:

1. All suggestions are returned with the same scores.
2. The nountype returns the URL of the entry as the HTML-formatted result and the title as the text-formatted result, which clearly does not make sense. However, it’s not clear to me whether the title, URL, or some combination of both is what we should be returning as the suggestion text presented to the user.²

I just rewrote noun_type_awesomebar to actually do some differential scoring. This new version also presents the URL or title depending on whichever had a better match using the matchScore function.³

input	suggestion
moz	www.mozilla.com	0.7
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.63
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.61
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.6

noun_type_url

The purpose of noun_type_url’s suggest function is two-fold: first, to accept strings which may look like a URL and, second, to suggest URL’s from the history just like noun_type_url, but only based on URL matches and not title matches.⁴ Here are a few sample inputs:

input	suggestion
moz	http://www.mozilla.com/	0.9
	http://moz	0.5
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.9
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.9
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.9
test	http://test	0.5
http://	http://	0.5
http:	http:	0.5
http	http	0.5
_test	http://_test	0.5
hello world!	http://hello world!	0.5

Oh, where to begin!? Here are some initial quirks… it’s possible that you could think of more!

1. There is no differential scoring… only 0.9 for suggestions from history and 0.5 for URL-like strings.
2. A number of invalid domain names are being accepted and turned into suggestions (“hello world!”, “_test”, etc.).
3. It’s trying to be smart by suggesting “http://” as a default URI scheme but doing so even for prefixes (initial substrings) of the word “http” itself.

With these thoughts in mind, I just took a first stab at improving this situation. Here are some features of the new implementation:

1. History entries are scored in the same way as in noun_type_awesomebar, using matchScore.
2. URLs without an explicit URI scheme (like “http://”) get a 10% penalty.
3. “http://” is only suggested if one of a long list of common URI schemes are not detected.
4. It repairs schemes which are missing a slash or two, suggesting for example “http:hello.com” → “http://hello.com”.
5. It actually uses Firefox’s own IDNService to check if the domain name is a valid internationalized domain name. If it’s an IDN as opposed to LDH (“letters, digits, and hyphens”), it gets a 10% penalty. If it’s not even a valid IDN, it is ruled out (see last two example inputs below).
6. There are also penalties for only being a domain name with no path and for the domain not having any periods (.) in it.

Here is what our suggestions now look like:

input	suggestion
moz	http://www.mozilla.com/	0.6
	http://moz	0.65
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.63
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.61
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.6
test	http://test	0.65
http://	http://	1
	shttp://	0.75
http:	http://	0.9
	shttp://	0.7
http	http://	0.72
	https://	0.71
	shttp://	0.68
	http://http	0.65
_test	none
hello world!	none

See you tomorrow~

Alright, enough nountype wrangling for one day. I’ll be back again tomorrow for another installment.

Related posts:

1. Nountype Quirks: Day 3: Geo Day
2. Nountype Quirks: Day 2
3. Localizing Commands for Ubiquity 0.5

Related posts brought to you by Yet Another Related Posts Plugin.

Different arguments are classified into different kinds of nouns in Ubiquity using noun types.¹ For example, a string like “Spanish” could be construed as a language, while “14.3” should not be. These kinds of relations are then used by the parser to introduce, for example, language-related verbs (like translate) using the former argument, and number-related verbs (like zoom or calculate) based on the latter. Ubiquity nountypes aren’t exclusive—a single string can count as valid for a number of different nountypes and in particular the “arbitrary text” nountype (noun_arb_text) will always accept any string given.

In addition to the various built-in nountypes, Ubiquity lets command authors write their own nountypes as well.

The functions of a noun type

Nountypes have two functions: the first is accepting and suggesting suggestions and the second is scoring.

Accepting and suggesting

Nountypes don’t just have to accept the exact string they were given—they can also return suggestions which are based on that input. For example, the noun_type_language can take the input “span” and return “Spanish.” A nountype can return multiple suggestions which may or may not include the trivial suggestion, i.e. the original input as is. If there is no way that that input could possibly be part of an accepted value, it should return no suggestions, i.e. [].²

Scoring

Ubiquity 0.5 with Parser 2 introduced the notion of a nountype suggestion score. For example, two different nountypes can accept the same input, but with different scores. Scores range from 0 to 1 where 1 is a perfect or exact suggestion and 0.1 or so is a very very improbable suggestion.³ These scores are used in the scoring of parses. Because verbs specify certain nountypes for each of their arguments, the scores that individual nountypes return for each argument are a crucial component of the scoring algorithm and can even determine whether a parse is returned or not.

With this in mind, you may be tempted to make your nountype return a score of 1 on any input so your verb will show up in the suggestions highly. While this would work, it will only act to make your verb annoying and a poor Ubiquity citizen. Appropriate scores must be given to noun suggestions, with higher values reflecting confidence and lower values reflecting imprecision. But how exactly do you figure out what’s an appropriate value?

Judging nountypes with the Nountype Tuner

The Nountype Tuner is a new tool I’ve been building to help both Ubiquity core developers and command authors to check their nountypes against others and to “tune” their behavior and scores. The nountype tuner will take your input and throw it against all of the nountypes referenced in your active verbs and display the suggestions returned with their scores. You can think of it as the Playpen’s little sister.

The Nountype Tuner can be found at chrome://ubiquity/content/tuner.html, though I am pretty sure it is broken in Ubiquity 0.5 and 0.5.1. It has been fixed now and I will make sure it’s in good shape for 0.5.2.

The heart and soul of the Nountype Tuner is this scale:

This scale tells you, in plain English, what different scores represent and correspond to, in two sets of vocabulary: “in terms of a guess” and “in terms of a match.” While still subjective, this scale helps developers just different input/output pairs and their scores. For example, “lian” → “http://lian” is given 0.5, so it’s an okay guess or a possible match… does that seem right to you? Or “lian” → “Italian” being between “okay” and “good.” Appropriate? We can look at such statements, decide how we feel about them, and tweak if necessary.

Good nountype scores have roots

CC-BY Aaron Escobar

…not that kind of root, but more like this kind of root… let me explain…

When comparing the scores that individual nountypes return for different inputs, we must compare those scores within the same nountype’s family of suggestions to see if higher scores truly correspond to higher confidence. For example, the language nountype should give the suggestion “French” for both the inputs “f” and “fren,” but the scores of these suggestions should be different—i.e. the score of “f” → “French” should be much lower than the score for “fren” → “French,” reflecting the additional informational value. We refer to this relation of the scores of successive prefixes of a single suggestion all returning that same suggestion as the score curve and in general it should be non-decreasing.⁴

One could say the most trivial score function then is the linear one. For a series of converging prefixes of the same suggestion (“Dutch”), under a linear approach we could naively let the score be (length of the input)/(length of the suggestion), as below:

This linear model is represented below by the black line.

The problem with the linear model is that earlier transitions (additional keystrokes) add more information than the later ones. Once we’ve entered “dutc,” after all, we would like to be pretty darn sure that we mean “Dutch,” so the score difference between “dutc” and “dutch” should be less than the score difference between, say, “d” and “du.” We want a score curve that looks more like the solid or dotted red lines above.

For this reason, I strongly advocate the incorporation of an nth-root in the score computation. Nth-rooted score functions over [0,1] have the feature that they are increasing but also that earlier transitions affect the score more than later ones, which is exactly what we’d like to see. (The solid red line above is x^1/2 and the dotted one is x^1/3.)⁵

Conclusion

Properly tuning both the built-in nountypes and custom nountypes is crucial to producing more accurate and relevant parse suggestions. I’ll be using the principles and criteria laid out above, combined with the new Nountype Tuner, to tune the built-in nountypes (trac #746) in the coming days in preparation for our 0.5.2 release. I invite you to use the Nountype Tuner in 0.5.2 to tune your custom nountypes as well.

Related posts:

1. Nountype Quirks: Day 1
2. Nountype Quirks: Day 2
3. Nountype Quirks: Day 3: Geo Day

Related posts brought to you by Yet Another Related Posts Plugin.

The keynote speaker was Ralf Steinberger of the European Commission’s Joint Research Centre, presenting the European Media Monitor family of applications. EMM is a suite of different applications all based on a core platform which aggregates news stories from a variety of sources around the world in a few dozen different languages. The system uses various CLIR techniques to then cluster stories by event, regardless of language or source country. Large news agencies and organizations, as well as the European Commission itself, use the system to track upcoming news stories as well as health concerns. In the European Union which has over a dozen different “official” languages, there is a great need for this type of service. The applications are available to the public so I invite you to play around with them at the EMM site.¹

The workshop’s paper presentations were quite diverse and all interesting. Elena Filatova presented an interesting approach to measuring the “trustworthiness” of statements in Wikipedia entries, by comparing the overlap in content statements between different language entries (more overlap => more trustworthy) which can be used to create inverted pyramid summaries. Perspectives from different use cases were also examined through presentations on news analysts and health organizations, patent searches, and medical information for personal use, as well as discussion of the need for CLIR for historical and religious texts.

One interesting thread throughout the day’s sessions was the issue of loanword processes and the possible use of romanization as an interlingua. Cross-language named entity recognition is a major problem in CLIR. Many novel words and names (common in news articles) go through processes of loanword adaptation and transliteration and are hard to identify and also are not in the systems’ dictionaries. A few of the talks touched on these problems, including a great talk by Kashif Riaz of the U of M on the salient (and great) differences between Hindi and Urdu. Frederic Gey described a number of different approaches to comparing strings via a romanization interlingua.

Another interesting thread was the idea of the target user. The needs of CLIR applications can vary greatly depending on its use case and the users’ savvy. The needs of a patent search office, where many professional searchers are already multilingual, is clearly different from the needs of an executive hoping to stay on top of the world news related to their organization. It was brought up, however, that in this age of open data and API’s, if a CLIR resource provides a good API, it need not necessarily supply the perfect interface for every type of user, as different third parties could also develop such targeted interfaces.

While most of the presentations and research interests in the room were on users accessing resources in various languages, I presented on Ubiquity. The talk was intended to highlight the idea that the opposite problem of users with different languages getting at the same kinds of information and getting equally powerful user experiences is also a different but worthwhile problem. Below are the slides from this talk. As a web archive of all the papers will be set up soon, I believe it’s safe to put up my paper as well, so please check it out. It’s a good short (four pages) overview of the innovative approaches we’ve taken to build a localizable natural language interface.

Related posts:

1. Presenting in Boston at SIGIR Workshop
2. The Hit List: Better Software Through Less UI
3. Mozilla By The Numbers

Related posts brought to you by Yet Another Related Posts Plugin.

This video walks through the process of converting your Ubiquity commands to Ubiquity 0.5 with Parser 2. For more information, please consult the command conversion tutorial.

Related posts:

1. A Demonstration of Ubiquity Parser 2
2. Changes to Ubiquity Parser 2 and the Playpen
3. Foxkeh demos Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

Command localization in Ubiquity 0.5

A few important design decisions have already been made in implementing command localization in Ubiquity 0.5. The first was the choice of the gettext po (portable object) file format. The po format is a de facto industry standard with many tools built for the format and this design choice hopefully lower the bar for prospective localizers.

Second, in order to simplify the matching of localizations and commands, we require that each command feed have one localization po file, rather than splitting the localizations of different commands across multiple po files.

Finding the appropriate localization po file for a JavaScript command feed is very simple with bundled commands as the files are managed in our main repository so we know exactly where to find them. We just take the command feed’s filename sans extension—the feed key—and look for localizations/lang/feed key.po, where lang is the active language code. This gives us a simple one to one relationship between the JavaScript command feed source and the appropriate localization.

Background

The ability of users to easily write their own Ubiquity commands has always been a huge strength of the Ubiquity platform. Users can also “subscribe” to commands written by other users on other servers. In this case, local copies of those command sources are made.

The herd was developed as a dynamic aggregator of community Ubiquity commands. The herd keeps its own copy of each command source. The herd groups mirrors of command feeds on multiple servers together as well, giving each unique command feed a unique ID.

Localizing distributed resources

This distributed nature of the command feeds is exactly what complicates the localization of community commands. The original sources themselves are distributed, and we also want a community of localizers to be able to localize the commands—i.e., for the localizations to be (in some sense) distributed. Here I will present three options which I believe could be the basis for a winning solution, in order of more distributed to less distributed.

All three of these options have the property that localizations do not need to be “approved” or “managed” by the command author. For example, a possible standard where URL’s of localizations must be in the command feed is not considered. I believe this is a crucial property of any approach we decide on.¹

Option 1: a completely distributed option

In this option, localizers simply put the po files on their own servers (or some code snippet site such as github) and the user must then subscribe to those localizations much as they subscribe to commands now.

Option 2: registration and discovery through the herd

In this option, localizers put po files up on their own servers and then register that po file’s URL with the herd. The herd keeps track of each command feed’s localizations in different languages.

Option 3: localizations on the herd

In this option, po localizations are uploaded right onto the herd. The herd is the centralized repository of all localizations for each command feed.

Users first

In coming up with a criteria for judging different models of community command localization, I think it is helpful to think of the end-user experience. Right now to subscribe to a new command a user must find the command (perhaps via the herd) and click on the subscribe button, then in most cases confirm that they are aware of the possible dangers and confirm subscription.² What work is required for a user to subscribe to a new command and get it in their language?

Under option one, the user would somehow have to find the localization scattered someplace on the internet of their own accord, and then install that localization by themself. In my mind, this is clearly a no go. With options two or three, however, when a user subscribes to a command feed Ubiquity can check with the herd to see whether there are any localizations available. The localizations could be offered to the user or the localization for the currently active language could be automatically installed. There are, on the other hand, disadvantages to requiring a centralized authority, exemplified by the fact that the current iteration of the herd itself has often been down.

Summary and a call for comments

As laid out in this visual guide, I personally have a couple main criteria which I believe we should follow:

1. localization independence: Command authors should not have to manage their commands’ localizations. (See footnote 1.)
2. friendly discovery and subscription: Users should not have to go out and find localizations by themselves. Localizations should be offered to the user.

With these criteria, I’m pretty sure the only logical conclusion is that we need some level of centralization, pointing to options two and three. If you think of another option which satisfies these criteria, or disagree with the criteria above, I would love to know. In addition, have you encountered similar problems of localizing distributed resources elsewhere? What worked there?

If this problem has been solved before, there is no need to reinvent the wheel. As far as I can tell, however, this could be a particularly hairy problem.

Related posts:

1. Ubiquity Localization Update
2. Localizing Commands for Ubiquity 0.5
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

Mozilla Labs の実験的プロジェクトのひとつ、 Ubiquity の最新版、バージョン 0.5 を昨日リリースしました。 (Mozilla Labs 正式発表 [英文])

Ubiquity はウェブをより有益に、より使いやすくするために自然言語で Firefox を操作するインターフェースを提供します。ウェブ上のオープン API と機能が増えて行く一方でどのようなインターフェースが必要であるのか。その答えを追求した結果、テキスト入力の正確さとスピードと自然言語の心地よさを合わせたインターフェースができあがりました。例えば「麹町を地図で表示」、「これを (誰々) へメール」などを自分の言葉で入力してブラウザを操作することができます。新しいコマンド (動詞) も簡単に JavaScript で書けるので、拡張性も非常に高いプラットフォームです。

ユーザにとって「自然な構文」 (“natural syntax” [英文]) という目標の下、数ヶ月の研究の結果、Ubiquity 0.5 では複数の言語の異なる構文に対応できるパーサを実装しました。Ubiquity 内蔵のコマンドもローカライズ可能になり、0.5 ではすべての内蔵コマンドの日本語、デンマーク語とポルトガル語版が搭載されています。

リリース直前に Ubiquity の日本語紹介ビデオを作成しましたので、どうぞご覧ください。日本語モードでの使用方法も説明されています。

Ubiquity 0.5 日本語紹介ビデオ from mitcho on Vimeo.

日本語サポートが入った Ubiquity 0.5 を是非ご使用ください。このインターフェースをより多くのユーザが「自然に」使えるよう、これからも開発を続けていきたいと思います。

Related posts:

1. Contribute to Ubiquity! No Coding Required!
2. Talking Ubiquity in Japan: 拡張機能勉強会にて発表
3. Ubiquity interview in Chinese

Related posts brought to you by Yet Another Related Posts Plugin.

Of course, one of the best ways to learn about the new features is to experience them… try Ubiquity 0.5 now!

Install now!

Preface: What’s What

To give users a completely localized experience, there are many different components that need to be made to work with different languages. In a single Ubiquity input, like

1

translate hello from English to Spanish

there are actually many different components that need to all be localized in order to comprehend the equivalent sentence in a different language. The diagram below will give you a sense for the different components that need to be localized: the parser, verbs, and nountypes.

What’s New

Ubiquity 0.5’s improved language support can be thought of as the product of two more or less orthogonal developments: the brand-new parser, Parser 2, as well as local command localization support.

Parser 2

Parser 2 (née Parser: The Next Generation) is a completely new parser designed to support different languages easily. Taking a serious look at the similarities and differences between different languages, we created a universal parser design which takes a minimal set of settings for particular languages to “learn” that language’s grammar.

The key insight to Parser 2’s design is that, for the limited range of inputs Ubiquity should understand, languages deal with them in remarkably similar ways. The input we’re dealing with here are all commands or actions without quantification or negation. These are all comprised of a single verb and a series of arguments with certain markings to designate their roles in the sentence. For example, here’s our example Ubiquity input:

1

translate hello from English to Spanish

In this example, “translate” is the verb, which we recognize by looking at our bank of known verbs, and the rest of the input can be split up into three different arguments: “hello,” “from English,” and “to Spanish.” Of these, the markers “from” and “to” tell us that “English” is a source of some sort and “Spanish” is a goal, while the unmarked “hello” is simply an object—the target of the action. By identifying arguments by these abstract semantic roles, we’re able to quickly identify different kinds of arguments in different languages. For example, the following is the exact same example but using the Japanese syntax and markers:

2

helloをEnglishからSpanishにtranslate

Ubiquity knows what the different markers mean in Japanese, like “を” > object, “から” > source, “に” > goal, and can easily interpret this to mean the exact same command as (1). With just a few lines of code, you can teach Ubiquity how to recognize these different semantic roles in your language. This innovation also means that Ubiquity commands can be written once for one language and automatically used with another language’s parser, bringing us half-way to the goal of command localization.

Note also that Japanese (as in example (2)) is verb-final and uses no spaces between words. We’ve tried to make Parser 2 itself agnostic towards these types of different ways in which languages vary.

Parser 2 also adds better argument-first suggestions, inspired by some earlier thoughts on Ubiquity in Japanese. Ubiquity will now start to parse arguments in the input even if a verb isn’t found, and suggest verbs based on that input. For example, if you enter “hello to Spanish,” it’ll recognize that you have an object of “hello” and a goal of “Spanish” which can be understood as a language name, so it’ll suggest the verb “translate.” This is the way it should be.

For more information and background, feel free to check out some of my previous blog posts on the new parser and on the different linguistic considerations. I also have a four-page academic paper giving an overview of some innovations in the parser—email me at x@x.com where x=mitcho if you’d like to get a copy.

Internationalization of bundled commands

The move to use semantic roles in the new command API, described above, means that the same Ubiquity command code can be used with inputs in different languages. Two things are left, then, to make a completely localized input work: (1) translation (localization) of different strings in the commands and (2) localization of the nountypes.

In Ubiquity 0.5, we built a localization infrastructure for commands (1, above) but have not yet tackled the nountypes (2). Ubiquity 0.5 uses the gettext po (portable object) file format for localizations, which many localizers in the UNIX world are very familiar with. This choice of file format potentially opens Ubiquity localization up to many who are new to localization or are unfamiliar with other Mozilla localization. Ubiquity is able to produce localization templates by itself and we also have a great tool to check the completeness of different localizations.

A huge caveat, however, is that this localization support currently only works with the commands bundled with Ubiquity itself.

What’s Next

We’re going to continue working to make Ubiquity more natural for more users. The tasks we have ahead of us are the localization of nountypes and community commands.

Nountype localization

With the new semantic role argument specifications, command localization simply became a question of translating some strings, which many localizers are used to. After all, we want localizations to affect the presentation of commands, not the logic of the commands. When it comes to nountypes, however, it is quite possible that we would actually want the nountype localization to affect its behavior.

Consider, for example, an imaginary day_of_the_week nountype. In English, this nountype might accept or suggest strings like “Monday” or “Tuesday,” while a French localization would accept “lundi” or “mardi.” More complicated still, consider a date nountype. In English this nountype may have custom logic to parse strings like “June 1st” while another language may have to parse very different kinds of strings. These nountype localizations thus involve not just string translations, but actual changes in their logic, making the po format approach we took to command localization a poor fit.

Making nountypes localizable, however, will make Ubiquity significantly more “natural” for many users. In the coming weeks and months we’ll be discussing and debating different options to accomplish this.

Community command localization

Even though the file format and infrastructure for command localization itself has been fleshed out with Ubiquity 0.5, the distributed nature of all these community commands adds an additional complication. Do we want community command localizations to be completely distributed, or should they be centralized? If they’re distributed, how do you find them? These are the types of questions we’ll need to ask and answer. The ease of creating a new Ubiquity command and sharing it with the world is a huge asset of the platform, so we’ll definitely be thinking about how best to localize these community commands as well. In the next day or two I’ll be writing up a more detailed blog post on what we need from a good community command localization solution.

Summary

For the more visually inclined (including myself), here’s a handy diagram to summarize what components are localizable now, what will be in the future, and what this means for Ubiquity users of different languages.

Get Involved

Whether you’re a speaker of an as-yet unsupported language, a veteran localization contributor, or simply interested in seeing how we can offer this natural language interface to more languages and more users, there are lots of ways to get involved. If you have some JavaScript experience and want to teach Ubiquity your native languages’ grammar, read our parser localization tutorial. If you would like to contribute localizations for our built-in commands, there’s a command localization tutorial. To discuss how best to localize nountypes and community commands, or to ask questions about or discuss command and parser localization, join us on the Ubiquity-i18n mailing list.

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. Localizing Ubiquity: commands and nountypes
3. Big Issues and Small Issues with Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.

If you would like to see a preprint of the paper, please contact me at x@x.com where x=mitcho.

Related posts:

1. Report from SIGIR Workshop on Information Access in a Multilingual World
2. WordCamp Boston 2010
3. Jetpacking in Boston

Related posts brought to you by Yet Another Related Posts Plugin.

追伸： ７月３日現在、 Ubiquity 0.5 のリリースを遅らせる方向になったので、残念ながら今日はリリースされません。是非リリース後インストールしてみてください。

Ubiquity 0.5 日本語紹介ビデオ from mitcho on Vimeo.

As Ubiquity 0.5 will be released soon (Thursday morning in Mountain View), I decided it was a good time to put together a screencast in Japanese demoing the use of the new Japanese parser and commands.

Related posts:

1. Changes to Ubiquity Parser 2 and the Playpen
2. Foxkeh demos Ubiquity Parser: The Next Generation
3. A Demonstration of Ubiquity Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.