mitcho.com » Mozilla Planet

Checking mochitest test coverage

mitcho — Tue, 22 Mar 2011 23:05:19 +0000

One of the last bugs for Firefox Panorama was bug 625818: “Check Panorama mochitest test suite coverage”. Our automated tests ensure that we do not regress on existing functionality, but it’s only as good as its coverage: how much of the Panorama code base is actually being “hit” through the process of running the test suite.

Panorama went through a pretty rapid development cycle, making it into Firefox 4 which was released today (yay!). Moreover, for a while we were developing outside of mozilla-central, without the regular “patches require tests” requirement. This makes checking its test coverage particularly important.

Check out the final result, the Panorama test coverage report. The good news: our code coverage is 86%! (Some notes on what improvements can be made are in the bug.)

PhiliKON had previously worked on hooking into the JS Debugger service’s interruptHook to test xpcshell tests. I modified this code to run instead in the Mochitest browser chrome tests. This code can be found on the bug.

With this patch applied, I invoked the test suite with the following code: TEST_PATH=browser/base/content/tests/tabview COVERAGE_FILTER="*tabview*" COVERAGE=true make -C obj-ff-dbg mochitest-browser-chrome . That’s a regular mochitest-browser-chrome invocation with the COVERAGE=true flag which turns on code coverage checking, and COVERAGE_FILTER=*tabview* which filters out results from files which don’t have “tabview” in their paths. This creates a file called coverage.json in the working directory of the test suite, meaning, for me, obj-ff-dbg/_tests/testing/mochitest/.

This JSON file is a multidimensional array, with file paths and then line numbers as keys. The file paths here, as best as possible, have been converted into local filesystem paths. PhiliKON built a script which produces beautiful reports based on this output.

A word of warning: running with this JSD interruptHook is ridiculously slow. A number of tests for Panorama are timing-dependent (drag-drop tests, for example), making some of them fail, but that’s okay… as long as it completed not via a timeout, it actually did run through all the code. In order to get this to run through everything with some degree of control, I split up the mochitest tabview suite in to a few chunks. I then took the multiple resulting coverage.json files and passed them into another script, in tools/coverage/aggregate.py, which takes multiple JSON results like this and puts them together into a single JSON file. I then passed this aggregate JSON file to PhiliKON’s wonderful report script and—voila—the Panorama test coverage report! Easy as pie.

Related posts brought to you by Yet Another Related Posts Plugin.

Happy Halloween!

mitcho — Fri, 29 Oct 2010 20:22:51 +0000

Happy Halloween from the Firefox Panorama team!

We carved some pumpkins a couple days ago in my department. I carved the Panorama logo above, but also one of the Stata Center.

More Jack-O-Lantern photos, including great ones of Chomsky and Norvin Richards, are up on Flickr.

Related posts brought to you by Yet Another Related Posts Plugin.

Beginning development with Jetpack SDK 0.2

mitcho — Wed, 14 Apr 2010 16:20:47 +0000

This article is a translation of a recent article in Japanese by fellow Jetpack Ambassador Gomita which was published on the Mozilla Labs Jetpack blog. I’m cross-posting it here for posterity.

Mozilla Labs recently released version 0.2 of the Jetpack SDK, which fixes some issues of the 0.1 release such as a glitch regarding development with Windows. SDK 0.2 doesn’t include the planned APIs for rapid development of new browser functionality, but you can still play with SDK 0.2 to get a flavor for development with the Jetpack SDK.

In this article we begin by setting up an SDK 0.2 development environment and explain the steps required to develop a simple, practical add-on using SDK 0.2. The instructions here are for Windows, but the basic steps are the same in every platform.

Installing Python

The first step to using the Jetpack SDK is to install Python. How to install Python depends on your OS, but in Windows you can choose the “Python 2.6.5 Windows installer” from the Python site and follow the installation wizard. Here, I’ll use C:\Python26\ as the installation path.

After the install, you can activate the python command in your command line by adding C:\Python26 to the Windows Path preference. (If there is already another value, delimit with a semicolon: “;”.) Run the command “cmd” from the Start menu to start the command prompt and run python -V to confirm the Python version, Python 2.6.2:

C:\>python -V
Python 2.6.2

Note, the Jetpack SDK Docs state that Python 2.5+ is required, but there seem to be some incompatibilities with Python 3.0.1 at this time. In addition, in my experience the SDK worked fine without the “Python for Windows extensions.”

Setting up the Jetpack SDK

Next, we set up the Jetpack SDK. Download the Jetpack SDK 0.2 package from the Jetpack site, unzip it, and place it somewhere convenient. Here, I used C:\jetpack-sdk-0.2.

To use the Jetpack SDK, it must be “activated.” From the command prompt, go to the Jetpack SDK folder and run bin\activate:

C:\jetpack-sdk-0.2>bin\activate
Welcome to the Jetpack SDK. Run 'cfx docs' for assistance.
(C:\jetpack-sdk-0.2) C:\jetpack-sdk-0.2>

Next, run cfx docs to open the SDK documentation in the browser. The SDK documentation starts a local server on port 8888.

(C:\jetpack-sdk-0.2) C:\jetpack-sdk-0.2>cfx docs
One moment.
Opening web browser to http://127.0.0.1:8888.

The package directory structure

Addons built with the Jetpack SDK are called “packages.” Let’s try building a simple “hello world”-style package, but first let’s see what the final directory structure of this package will look like:

directory/file	Note
jetpack-sdk-0.2	the Jetpack SDK folder
packages	the main packages folder
hello-world	package root
package.json	package manifest file
README.md	package documentation
lib	the package code directory
main.js	main program code
simple-dialog.js	a custom code library

The package’s root directory is placed in the “packages” directory in the Jetpack SDK folder, and includes the package.json manifest file and the README.md documentation file. The lib folder includes the package’s main program code and any custom libraries used by our addon.

Creating the package

We begin by creating the hello-world directory in C:\jetpack-sdk-0.2\packages . Next the manifest file package.json is created. The manifest file includes metadata about our package in JSON format. If you’ve ever created a XUL-style addon before, you can think of this as similar to the install.rdf file. Here, I used the following as the manifest:

{
    "id": "helloworld@xuldev.org",
    "version": "0.1",
    "description": "This is my first package.",
    "author": "Gomita "
}

The id property is used as a unique ID for all addons including Jetpack packages and is often formatted as an email address. This corresponds to XUL-based addons’ <em:id> tag.

Next, reload the SDK documentation in the browser and confirm that “hello-world” shows up under “Package Reference.”

Writing the main code

The next step is to add some working code to the hello-world package. Create a lib folder under the package root and create a main.js under lib with the following code:

exports.main = function(options, callbacks) {
    console.log("Hello, World!");
};

The main program code is always loaded as a module called main. This main property is made accessible from outside code using the CommonJS-style code exports.main = .... console.log is a global function made available by Jetpack and the SDK prints the string to the command prompt.

It’s worth noting that, in the current Jetpack SDK, calling “console.log("こんにちは");” doesn’t yield the expected Japanese output. In the future such output will be handled through the planned localization API.

Testing our package

With some simple code in our main function, it’s time to try this code out. To test this code, we run cfx run -a firefox in the command prompt. By running cfx run with the -a firefox option, we load our package into a brand new Firefox profile and launch Firefox.

(C:\jetpack-sdk-0.2) C:\jetpack-sdk-0.2>cd packages\hello-world

(C:\jetpack-sdk-0.2) C:\jetpack-sdk-0.2\packages\hello-world>cfx run -a firefox
info: Hello, World!
OK
Total time: 1.531000 seconds
Program terminated unsuccessfully.

After Firefox loads, confirm that the command prompt reads info: Hello, World! When you quit Firefox, the testing will end.

Using a standard library

Now we’ll edit our code to invoke the timer library which is one of the Jetpack SDK’s standard libraries. The timer library is a module which abstracts various timer-related functionality, similar to the DOM’s window.setTimeout, window.clearTimeout. Details on this library are available in the SDK documentation. Moreover, although not in the documentation, timer.setInterval and timer.clearInterval also work in this version.

To use this library in our main program code, we first must invoke this library with the CommonJS require function. We modify the main.js file as follows:

var timer = require("timer");
exports.main = function(options, callbacks) {
    timer.setInterval(function() {
        console.log(new Date().toLocaleTimeString());
    }, 1000);
};

After this change, run cfx run -a firefox in the command prompt to test it. Check to make sure that the current time is being printed to the command prompt once a second:

(C:\jetpack-sdk-0.2) C:\jetpack-sdk-0.2\packages\hello-world>cfx run -a firefox
info: 10:37:21
info: 10:37:22
info: 10:37:23
info: 10:37:24
info: 10:37:25

Creating a custom library

Next we’ll create a custom library to add some functionality not currently included in the Jetpack standard library. Implementing advanced functionality in add-ons, like filesystem access, involves using XPCOM components. Jetpack encourages seprarating the use of XPCOM components into separate modules which are then used by the main program code. The Jetpack SDK doesn’t currently disallow direct XPCOM access within Jetpack add-on code, but such a restriction is forthcoming. Modularizing XPCOM code into separate libraries now allow you to easily migrate to equivalent standard libraries in the future.

Let’s create a simple-dialog library to display a modal dialog much like window.alert does. The Jetpack code’s runtime environment doesn’t include access to the regular window or document objects, so just calling window.alert doesn’t work. To create an alert from this context, we use the nsIPromptService XPCOM component. In our package’s lib folder, create a simple-dialog.js file. Just like our main program code, we implement this library as a CommonJS module using exports.methodname = function(...){...}.

The simple-dialog library will have these two methods:

Method	Note
`alert(text)`	Displays an alert dialog with the string in text and an OK button. Equivalent to the DOM’s `window.alert`.
`confirmYesNo(text)`	Displays a confirmation dialog with the string in text and Yes and No buttons. The method returns `true` if the user presses “yes” and `false` if “no.”

Here is the code for simple-dialog.js:

var promptSvc = Cc["@mozilla.org/embedcomp/prompt-service;1"].
                getService(Ci.nsIPromptService);

exports.alert = function(text) {
    promptSvc.alert(null, "[Jetpack]", text);
};

exports.confirmYesNo = function(text) {
    var pos = promptSvc.confirmEx(
        null, "[Jetpack]", text, promptSvc.STD_YES_NO_BUTTONS,
        null, null, null, null, {}
    );
    return pos == 0;
};

Lines 1-2 are for calling nsIPromptService. Note that Cc, Ci are aliases for Components.classes and Components.interfaces, respectively, and are made available by Jetpack as global variables. Lines 4-6 implement the alert method for showing alert dialogs using nsIPromptService’s alert method. Lines 8-14 implement simple-dialog’s confirmYesNo method using nsIPromptService’s confirmEx method to display the dialog with yes and no buttons. nsIPromptservice’s confirmEx method returns 0 if the user presses “yes” and 1 if “no”, so we modify this value and return it.

Using our custom library

Let’s call this new custom library from our main program code and verify that it works. Here’s our updated main.js file:

var simpleDialog = require("simple-dialog");

exports.main = function(options, callbacks) {
    var adult = simpleDialog.confirmYesNo("Are you over 18 years old?");
    if (adult) {
        simpleDialog.alert("Welcome!");
    }
    else {
        simpleDialog.alert("Good bye!");
    }
};

Run cfx run -a firefox and confirm that a confirmation dialog, as seen below, is displayed. Pressing “yes” and “no” should give you the appropriate alert dialogs as well.

Implementing a network status observer

Now let’s use this hello-world package as a foundation for a more practical add-on. Using the observer-service module included with the Jetpack SDK, we can monitor Firefox’s online/offline network status changes.

Firefox internally broadcasts various application events to observers via the nsIObserverService XPCOM component. When Firefox goes offline, a network:offline-status-changed notification is broadcast. To subscribe this notification and act on it, we use the observer-service library’s add method. add’s first argument is the name of the notification we want to subscribe to and the second argument is a callback function. The callback function is given two arguments, of which the second is a string equal to either “online” or “offline.” In our add-on, we’ll check this value and display an appropriate alert using simple-dialog.

var simpleDialog = require("simple-dialog");
var observer = require("observer-service");

exports.main = function(options, callbacks) {
    observer.add("network:offline-status-changed", function(sbj, data) {
        if (data == "online") {
            simpleDialog.alert("Firefox is now online.");
        }
        else if (data == "offline") {
            simpleDialog.alert("Firefox is now offline.");
        }
    });
};

Launch Firefox by running cfx run -a firefox and then choose “File” > “Work Offline” and you should get a notification:

Adding documentation

If you add documentation to a package, you can view it by clicking that package in the SDK Documentation. To add documentation, create a README.md file in the package root directory. README.md is written in Markdown format which looks like this:

This is my *first* package.
* foo
* bar
* baz

Now if you load the SDK documentation using cfx docs and click on the “hello-world” link, you’ll see this documentation together with the package metadata.

Exporting an install package

Jetpack add-ons which are created in this way can then be exported into Firefox-standard XPI files. To export an XPI, go to the package’s root directory in the command prompt and run cfx xpi.

(C:\jetpack-sdk-0.2) C:\jetpack-sdk-0.2\packages\hello-world>cfx xpi
Exporting extension to hello-world.xpi.

This creates an XPI file called hello-world.xpi. Opening this file in any Firefox profile will let you install it using the regular add-on install mechanism.

Related posts brought to you by Yet Another Related Posts Plugin.

Spring is for Speaking: JSConf, WordCamp SF, IACL

mitcho — Sat, 20 Mar 2010 04:37:04 +0000

I recently confirmed three different very exciting speaking gigs which I’ll be doing this spring:

JSConf.us:

I’ll be putting my Mozilla Jetpack Ambassador hat on to represent Mozilla Labs’ Jetpack project at the premier Javascript conference in North America, JSConf.us, which this year will be April 17-18 in Washington D.C. and has a pirate theme.¹ I’ll be giving a short talk in the main session and will also lead a hands-on Jetpack workshop in the hacker lounge. I’ve heard that JSConf is a lot of fun and I’m really looking forward to it!

WordCamp San Francisco:

I’m honored to have been invited to give a talk at WordCamp San Francisco 2010. WordCamps are community-organized events for the WordPress community, and the San Francisco WordCamp is the original and biggest. WordCamp SF will be at the Mission Bay Conference Center on May 1st. Tickets available.

My talk is tentatively titled “Abstract Your Code.”² WordPress is a great platform to build amazing content-rich applications on, and many of us have written new functionality in the form of plugins. I hope to encourage developers to make their code more portable and reusable after the project is done—or, ideally, to even start with abstraction in mind—to add to the “life” of the code and to consider then open-sourcing that functionality.

Hope to see you there!

International Association of Chinese Linguistics (IACL) 18:

Finally, I’m thrilled to say that I got a paper accepted to the annual meeting of the International Association of Chinese Linguistics which this year is at Harvard on May 20-22. IACL is the big conference for Chinese linguistics, with about 180 papers presenting. I’ll be presenting Two Onlys in Mandarin Chinese, my recent work on the formal syntax/semantics of two only words in Chinese: zhǐ (只) and éryǐ (而已). I’ve put up a handout of some of this material in work-in-progress form which I recently presented at SNEWS.

I’ll fit right in. ↩
Sexier title suggestions welcome. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Jetpacking in Boston

mitcho — Sat, 13 Mar 2010 21:19:06 +0000

A couple weeks ago I gave a talk at the Boston Javascript meetup introducing Jetpack and filling people in in the latest developments in the project, including the Reboot. Between 20 to 30 people came to the talk which was at Microsoft Cambridge. Here are the slides from the talk:¹

Extend the Browser with Jetpack

I felt the audience was very engaged and we had almost a dozen questions in the question period and throughout. Most of the examples centered around the use of capabilities and the new security model. For example:

What level of granularity will the manifests have? Do you have to just say “I use xhr” or “I talk to twitter but not facebook”?
What will be the review process for capabilities?
What license will capabilities use?

Perhaps I simply did not present this material in the clearest manner (it didn’t help that I didn’t have any particular slides covering the capability and manifest model) but it also reflects the reality that these particulars of the manifests’ security sections are still being fleshed out. I’m looking forward to the next time I speak on Jetpack when hopefully the discussion of the Reboot will be more concrete and I’ll be able to give a clearer overview and demo of these aspects of the Reboot.

If anyone would like the Keynote deck, just let me know. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Jetpack Ambassadors in MV

mitcho — Sun, 21 Feb 2010 20:55:23 +0000

A couple weeks ago I went out to Mozilla HQ in Mountain View for a Jetpack Ambassador meetup. Jetpack is a project at Mozilla labs intended to make writing Firefox add-ons easier, and shares some ancestry with the Ubiquity project dear to my heart. The Jetpack Ambassadors are a team of Mozilla community members who will be involved with Jetpack marketing, evangelizing Jetpack and writing about our own experiences working with the exciting new Jetpack architecture.

We spent a good chunk of time with a team from Invisible Elephant who came in to give us some training on making technical presentations, and then dug into the code on Day 2. It was great to have the geniuses at Mozilla Labs like Atul and Myk show us the latest Jetpack code as well as get the latest project direction from Daniel, Aza, and Nick, from which we could see the amount of careful consideration and effort that’s gone into the Jetpack reboot.

The best part of the whole experience, though, has to be the fellowship with the other Jetpack Ambassadors. The Ambassadors came from all over the world, encompassing Europe, Asia, S. America, and of course N. America. Each are involved with some really exciting projects and have each made a name for themselves in their respective communities. I’ve put together a twitter list of all the Jetpack Ambassadors and the core team members and invite you to follow them.

We also had the greatest number of Ubiquity core developers to have ever been in the same place at the same time, which of course had to be documented.

(More photos can be seen in my gallery.)

I had a fantastic time in MV and it was a shame I could only be there for such a short time. I feel honored to be a part of this group and am looking forward to speaking on Jetpack soon at an event near you!

Related posts brought to you by Yet Another Related Posts Plugin.

Mashing up the browser in Maine

mitcho — Sat, 19 Dec 2009 19:00:32 +0000

Last week I was invited to give a talk at the TechMaine annual conference in Portland, Maine.

Being a longer time slot than I previously have used to talk about Ubiquity, I decided to dedicate a good portion of the talk to Jetpack. Being outside of Mozilla for the past few months, this gave me an opportunity to get reacquainted with the Jetpack APIs. I myself was impressed by how easy it was to develop a quick Jetpack. I ended up preparing two to live-code during the talk: one called Helvetica which, with one click, replaces all fonts on the current page with Helvetica; and You Are Here which uses an open API from IPinfoDB to display the physical location of the domain you are currently visiting in the status bar. Both are now on the Jetpack Gallery.

Unfortunately there was a bit of a snowstorm leading up to the event, but there was still a nice turnout and I got to meet some fantastic people there. Ken Shoemake of slerp and quaternion fame came up to me after my talk and said “the Ubiquity parser reminded me of the dancing bear… it’s less surprising that it works well as that it works at all.” I also enjoyed the other great presentations in the technology track, covering the virtues of REST and basic iPhone development.

Mashup the Browser with Ubiquity and Jetpack

Related posts brought to you by Yet Another Related Posts Plugin.

Mozilla By The Numbers

mitcho — Sun, 06 Sep 2009 04:26:54 +0000

About six months ago I started working for Mozilla Labs full-time, focusing on Ubiquity, the multilingual natural language interface for the browser. This week marked my last week on contract as I go back to grad school next week. While the work will go on and I hope to continue to stay involved as time allows, here’s a quick bird’s eye view of my activities in my Mozilla tenure:

Time working for Mozilla: 6.5 months

Mozilla-related blog posts written: 69

Academic papers written on Ubiquity: 1

Ubiquity presentations given: 5

Screencasts made: 8

Most popular video on Vimeo: Ubiquity 0.5 日本語紹介ビデオ, the Japanese Ubiquity 0.5 introduction video: 2252 views

Languages Ubiquity commands and parser now support: 6

Commits to the Ubiquity repository: 492

Other web projects started during this period: 2+ (Ten Grand Is Buried There, HookPress)

TechCrunch references: 2 (1, 2)

Countries worked in: 2

Mythical Kiwis worked with: 1

References to bugs I introduced as “glitcho”s: 1

Extremely disturbing homages to me and Django: 1

Friends made; experience gained; lessons on Open-ness learned; personal growth: ~~priceless~~ enumerable

Thanks to all who made this experience amazing, beginning with Aza, Jono, Atul, Blair and the rest of the Labs team; intern extraordinaire Brandon; the always thoughtful and friendly Mozilla Japan team; and of course the fantastic Ubiquity community! Please visit me in Boston—I should be around for a while.

Related posts brought to you by Yet Another Related Posts Plugin.

The Aliens Aliases Have Landed

mitcho — Sat, 05 Sep 2009 00:46:56 +0000

This week I implemented a new way to customize and extend Ubiquity commands: CmdUtils.CreateAlias.

The use case for and importance of `CreateAlias`

CreateAlias lets you easily create a special-case alias of another, more generic verb. Ubiquity comes bundled with useful verbs like translate and search which can be used for a number of different uses based on their arguments. In some cases, and in some languages, though, typing out translate to English or search with Google is unnatural, though, as there is a more succinct and direct way to make that request. For example, in English one could say “anglicize” or “google”, respectively, for the verbs and arguments above. Indeed, in order to support both search with Google and google, Ubiquity traditionally has implemented two different verbs, search and google, which duplicate functionality and code.

CreateAlias lets us create such natural aliases without repeating ourselves. We can easily create an anglicize verb which, in one word, does the work of translate to English, or google which is semantically equivalent to search with Google.

These sorts of aliases become particularly important in our perpetual quest to internationalize Ubiquity. One discussion that came up early on on our Ubiquity-i18n list is the fact that not all languages have the verb “Google”: in many languages it is necessary to explicitly say “search with Google”. Moreover, other languages may have other domain-specific verbs which English doesn’t have either. Maybe some language has a special verb for “email with Hotmail” or “map Denmark”. Who knows? With CreateAlias we can easily enable such localizations based on the more generic commands bundled with Ubiquity.

Creating an alias

CreateAlias was designed to be incredibly simple to use. Here’s an example that will be bundled (but not installed by default) in Ubiquity:

CmdUtils.CreateAlias({
  names: ["anglicize"],
  verb: "translate",
  givenArgs: { goal: "English" }
});

As you see, this syntax is incredibly straightforward. There are two required properties, names, an array of names for the alias, and verb, a reference to the target verb that this alias should use.¹

The alias can also have a givenArgs property which is a hash of pre-specified arguments with their semantic roles. Because translate takes three arguments (an object text, a goal language, and a source language) but we have pre-specified a goal in the givenArgs, the new anglicize command will only take two arguments: the object text and a source language. Of course, if you specify no givenArgs, you’ll get a simple synonym without having access to the original verb’s code.

As you see, the preview of this command is simply the preview of the translate verb. Its preview and execution is just as if you had entered translate こんにちは to English.

Just like other commands created with CreateCommand, the object specifying the alias can also have properties like help, description, author information, and so on. I used the icon property to add a Union Jack to it so that it was easily identifiable.

Bonus: using `CmdUtils.previewCommand` and `CmdUtils.executeCommand`

On the road to implementing CreateAlias, I also implemented the CmdUtils.previewCommand and CmdUtils.executeCommand functions. The majority of this code comes from previous work by Louis-Rémi Babé, though I adapted it to the modern Ubiquity system. Using previewCommand and executeCommand you can take advantage of the preview or execute functionality of another command. In the new alias-commands feed I included a command called germanize which essentially is a straightforward analogy to anglicize, seen above, but using these functions within a CreateCommand. While CreateAlias is much more straightforward for simple aliases, for more complex subcommands where you would like to adapt another verb’s execution or preview, or only take one of those but re-implement the other part, previewCommand and executeCommand are the way to go.

The verb reference can be the canonical or reference name of a command, which is the first name in the names of a command (also the name listed in the command list when Ubiquity is running in English) or the actual internal ID of the command, which looks like resource://ubiquity/standard-feeds/general.html#translate. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Exploring Command Chaining in Ubiquity: Part 2

mitcho — Sun, 23 Aug 2009 23:14:07 +0000

Introduction

I recently have begun giving serious thought to what command chaining might look like in Ubiquity and the various considerations which must be made to make it happen. The “command chaining,” or “piping,” described here always involves (at least) two verbs acting sequentially on a passed target—that is, the first command performs some action or lookup and the second command acts on the first command’s output.

A few days ago I penned some initial technical considerations regarding command chaining. In this post I’ll be point out some linguistic considerations involved in supporting a natural syntax for chaining.

Simple syntaxes: sequential vs embedding strategies

When it comes to creating a natural language interface, there’s always a decision to make between requiring a certain kind of input, or working a little harder to understand the user’s natural input. From an implementation point of view, adopting certain programmatic conventions is of course simpler and to this end, there have been a couple different “unnatural” command chaining syntaxes suggested. While these both go against Ubiquity’s basic tenet of natural syntax — that is, to not introduce rules which contradict the user’s natural language — which gives Ubiquity its strengths of usability and memorability, I’ll entertain them here as they illustrate two different structural relationships that we will want to consider.

The first suggestion is to adopt the shell pipe (|), which would lead to input such as

1	translate hello to Spanish \| email to Jono

While this itself is pretty unnatural unless you speak shell, note that this syntax is similar to the more natural “, and” syntax, yielding translate hello to Spanish, and email to Jono, which we will consider below. I’ll refer to this strategy as the sequential strategy.

Another very interesting proposal by Alex Fritze is to embed each subordinate computation into an argument position, marked by parentheses. This could also be parsed relatively straightforwardly by writing a noun type which first checks for parentheses and then runs the content of the argument through another ParseQuery.

2	email (translate hello to Spanish) to Jono

I’ll refer to this pattern as the embedding strategy.

Sequential and embedding strategies in natural language

What’s interesting about the two proposals above is that both strategies are seen in natural language. The sequential strategy could correspond to the following linguistic phenomena:

coordination: a non-hierarchical joining of two or more clauses, often marked by a conjunction. Here’s an example from English:
- “[I made a sandwich] and [you will eat it]” where [] represent clause boundaries. Here, “and” is the conjunction.
serial verb and converb constructions: a joining of multiple verbs or verb phrases within a single clause, with shared subject and tense/aspect values, with no particular conjugation or delimiter between them. Such constructions are common in many African and east Asian languages. Here are two examples:¹

A converbal construction in Japanese:
3 4 5
僕はサンドイッチを作って食べる boku-wa sandiʔchi-o tsuku-ʔte tabe-ru I-TOP sandwich-ACC make-CON eat
“I (will) make a sandwich and eat [it].” (Here, `TOP` = topic, `ACC` = accusative, `CON` = converbal ending)[^3]
A serial verb construction in Mandarin Chinese:
6 7 8
我作三明治吃 wǒ zùo sānmíngzhì chī I make sandwich eat
“I (will) make a sandwich and eat [it]” or “I (will) make a sandwich [in order to] eat [it].”

Note that in both the converb and serial verb construction, the second verb (eat) takes shares its object (sandwich) with the first verb and there is no need for a pronoun such as “it” to introduce that argument as it is with coordination, above.²

The embedding strategy is observed in natural language as well, in the form of the following phenomena:

embedded clauses: a sentence is itself the argument of another verb. Example:

“John says [he likes sandwiches].”

Embedded clauses, however, clearly have no relation to command chaining and does not require our attention. 2. relative clauses: a partial sentence³ is attached to a noun in order to describe it or distinguish it from other possible referents. Example:

“You ate the sandwich that I made” where “sandwich” is called the “head” of the relative clause, and “I made” is what I here call the “partial sentence” (see footnote). The “relative clause” is used here to distinguish “the sandwich that I made” from other sandwiches.

The natural syntax of chaining

So which strategy is used in complex natural language commands: the sequential strategy or the embedding strategy? Both the sequential strategy and embedding strategy can be involved with commands:

9 10	[Make a sandwich] and [eat it]! Eat (the sandwich that I made)!

These two commands do not mean the same thing, though, and only (9) is the kind of command we would want to give Ubiquity. The problem with relative clauses, as in (10), is that it presupposes the existence of the sandwich in the context. If we both know you just made a sandwich, saying (10) is perfectly appropriate, but out of the blue it doesn’t make sense. For this reason, only the sequential strategy is used in the natural syntax of chaining.

Parsing the sequential strategy

In natural language, unlike the initial simple proposals laid out above, there is often no clear delimiter marking the boundary between the two parts in a sequential relation (e.g. examples (3) and (6) above, particularly given that neither Japanese and Chinese normally break words with spaces). How would we parse a sequential string of commands?

Let’s assume for our purposes here that we can identify find all verbs within the input string.⁴ Parsing a sequential strategy string is not particularly difficult if we can also assume that the verb in any particular language is either always verb-initial or always verb-final. Let’s look at both cases:

Always verb-initial: Mandarin Chinese:

11
12
13

翻譯       hello 到  西班牙語   送    給  Juanito
fānyì     hello dào xībānyáyǔ sòng gěi Juanito
translate hello to  Spanish   send to  Juanito

“Translate hello to Spanish [and] send [it] to Juanito” 1. find every possible verb:
翻譯hello到西班牙語送給Juanito 2. as every verb marks the beginning of a sentence, we now have our two commands: “翻譯hello到西班牙語” (translate hello to Spanish) and “送給Juanito” (send to Juanito). * Always verb-final: Japanese

14
15
16

helloを スペイン語に 訳して Juanitoに 送って
hello-o supeingo-ni yakus-ite Juanito-ni oku-ʔte
hello-ACC Spanish-DAT translate-CON Juanito-DAT send-CON

“Translate hello to Spanish [and] send [it] to Juanito” 1. find every possible verb:
helloをスペイン語に訳してJuanitoに送って 2. as every verb marks the end of a sentence, we now have our two commands: “helloをスペイン語に訳して” (translate hello to Spanish) and “Juanitoに送って” (send to Juanito).

For languages where there is a clear conjunction between the two commands, such as English “and”, we can also use that conjunction as a delimiter as well. We then simply execute the first command and then execute the second with the first command’s output in its interpolation context. This way the output of the first command will be picked up both by an overt pronoun such as “it” in the second command and without it, such as in the Chinese and Japanese examples above.⁵

The only potential problem with this approach is in the case of languages where some commands are verb-initial while others are verb-final. I note that such languages do exist in a previous blog post, Where’s The Verb. In these languages, commands can be expressed by more than one verb form (such as infinitive, imperative, subjunctive, etc.) and some of those verb forms are sentence-initial while others are sentence-final. Here’s one such example from German:

“search hello with google” (German) 1. Infinitive: hello mit google suchen 2. Imperative: suche hello mit google

Here the verb for “search” is “suchen” (infinitive) or “suche” (imperative). I know that this same type of phenomena occurs in other Germanic languages such as Dutch with infinitive and imperative and also other languages such as Modern Greek with infinitive and subjunctive forms. If you are a speaker of one of these lanuages (German, Dutch, Greek, etc.) I would love to know whether you can chain verb-final and verb-initial commands together.

Conclusion

In this blog post I examined command chaining in natural language, focusing on data from English, Mandarin Chinese, and Japanese, which exhibit three linguistically different approaches to chaining. What we found is that the sequential strategy—that of listing the commands one by one, in order of execution—is what is used in natural languages, rather than any sort of embedding. This fact, combined with the fact that our parser can recognize every available verb, offers a simple approach to doing a naive parse of natural command chains in most languages, even without explicit delimiters.

In a final installation of this series on “exploring command chaining,” I hope to consider how the Ubiquity interface itself could present command chains and aid in its input.

The distinction between serial verb and converb constructions (as well as other forms of complex predication) hinge on structural distinctions which are not of importance for our purposes. ↩
Some people (Baker 1989 and others), in fact, list this object sharing as a necessary part of the notion of a “serial verb construction.” ↩
“Partial sentence” is used in a descriptive sense here to reflect that the relative clause, such as “I made” in the example given cannot stand as its own sentence, as the verb’s argument is clearly missing. This type of pattern is also seen in questions (“What did [you make]?”) and topicalization (“That sandwich, [I made].”) and is a great focus of theoretical linguistics research. See wh-movement on wikipedia for more examples and information on theoretical approaches to such constructions. ↩
We don’t do this right now as there hasn’t been a use for it—right now Parser 2 simply looks for known verbs at the beginning and end of the input. The parser does build a nice regular expression to find known verbs, however, so finding verbs input-medially would also be easy to do, though. ↩
Note that even though the linguistic relation between the two commands is non-hierarchical, we interpret the sentences to mean “translate hello to Spanish and then email it to Juanito”, rather than “translate hello to Spanish and email it (hello) to Juanito at the same time.” This observed universal property that, ceteris paribus, the linear speech order of verbs reflects the conceptual order of events is known as the Temporal Iconicity Condition (Li 1993 and others). ↩

Related posts brought to you by Yet Another Related Posts Plugin.

The Ubiquity Persistence Project: exploring a persistent Ubiquity in the toolbar

mitcho — Thu, 20 Aug 2009 19:50:13 +0000

It’s often hard to remember Ubiquity’s presence and keystroke without a visual reminder—even I often forget that I could use Ubiquity and end up going to a search engine or using the search bar for some quick lookup task. What if the Ubiquity input were in the toolbar and always visible? How would that affect people’s use of Ubiquity? And what could we make that look like and how would it behave? Today we’re kicking off the Ubiquity Persistence Project, a new Ubiquity initiative to explore what a persistent Ubiquity might look like in the Firefox toolbar.

In order to facilitate this discussion, we created the Persistence tool. With the Persistence tool you can quickly try out new design and interaction ideas, mocking things up with some simple jQuery-powered JavaScript and CSS and see your changes live. The Persistence tool is bundled with our latest Ubiquity beta (install link).

The Ubiquity Persistence Project: exploring a persistent Ubiquity in the toolbar from mitcho on Vimeo.

I just put together a screencast introducing the initiative, demoing the Persistence tool, as well as talking about this project’s relation to the ongoing work on Taskfox. We’ll look forward to your comments and designs!

Related posts brought to you by Yet Another Related Posts Plugin.

Exploring Command Chaining in Ubiquity: Part 1

mitcho — Wed, 19 Aug 2009 20:35:37 +0000

Since the dawn of time people have been asking about command chaining in Ubiquity. If you have a translate command and an email command, it would be great to be able to, for example, translate hello to Spanish and email to Juanito. This is what we call command chaining or piping: in a single complex query, specifying multiple (probably two) actions and using the first’s output as the second’s input.¹

Today I hope to cover some of the technical considerations required in implementing command chaining in Ubiquity, and I will follow up soon with a blog post on the linguistic considerations required as well.

Technical considerations: hooking the pipes together

I’d first like to lay out some technical challenges and questions. These can be broken into two different categories: (1) how the parse and display of suggestions is affected and (2) how the execution is affected.

Matching inputs and outputs

We’ll first consider how command chaining may affect the parsing. Ubiquity commands each specify the types of argument inputs that it expects using different noun types, such as noun_arb_text which accepts anything, noun_type_number which accepts numbers, or noun_type_language, which takes the name of a language. For example, the translate verb takes maximally three arguments: a noun_arb_text object, a noun_type_language goal (the language to translate into), and a noun_type_language source (the source language). In implementing command chaining, it will be necessary to identify the appropriate noun types for the output of a command.

The first question we must address here is “what is the chaining output of a command”? Is it the preview text? Some text output from the execution?

Big fish eat da lil fish by joemud, CC-SA-NC

To put this question into perspective, we note that Ubiquity commands can be broadly classified into two types: lookup and action execution. Here’s a classification which I believe to be exhaustive:²

classification	preview	execution	example
lookup	data lookup	inserting result into page	`translate`
		opening a website	`weather`, most search commands
		copying result to pasteboard	`get email address`
		nothing
action	nothing (maybe a description of what the action will do)	an action which changes some state (in the browser or on the web)	`quit firefox`, `email`, `twitter`

In light of this classification I believe we can say that lookup commands are much more likely to be the first verb in a command chain—conversely, chains such as email hello to Blair and then do ... or twitter hello and then ... are quite unlikely. What is much more likely is for the first verb to be a lookup function.

first verb type	second verb type	example
lookup	action	`translate this to Spanish and email to Aza`
lookup	lookup	`translate this to English and then find it with Amazon`
action	action/lookup	no use case?

Thus in the same way that not all commands have a useful execution³ perhaps only lookup commands will have a chainable output: the results of the lookup. Even with this restriction, we will most likely need to implement a new “chainable output” method or getter in these commands. This means that commands will need to opt-in to become chainable, but I believe this is a necessary evil.

The second question we must address is “when do we establish the noun type of a command’s chainable output?” One unsung but crucial feature of the way Ubiquity works now is that suggestions’ previews are not computed until that suggestion is selected (except for the first suggestion, which in most skins gets previewed immediately). Should we wait for all of the first verbs’ chainable output to be computed and then run them through the noun type detection system? Or should verbs with chainable output also a priori specify what noun types their output will be?

Both of these approaches have their problems. If we compute the chainable output of the first verb, run a noun type detection on it and then suggest the full combination if it matches what the second verb was expecting, this will have clear performance implications, not to mention that it could greatly complicate our parsing algorithm. While the latter approach doesn’t have these performance implications, it does mean that it will have to list (by name or reference) the noun types that will match its output, meaning that if a command author is unaware of someone else’s noun type, that chain will be impossible, even if the chainable output itself does indeed match that noun type. The a posteriori approach would never have this issue. What other benefits or problems do you forsee with either of these approaches? Is there another approach which avoids these pitfalls?

(A)synchronous composability

Once we have the noun types, parsing, and suggestions down, all that remains is to compute the previews and implement the composite execution. Since the Ubiquity command manager already wraps the preview and execute functions in a wrapper to facilitate localization, among other uses, it would be easy to make the command manager compose asynchronous processes pseudo-synchronously. No major changes should be necessary to do the previews and executions, though, again, there will be a performance cost.

Conclusion

There are a number of technical questions which must be answered, mostly in the parsing/suggesting stage. The key questions to answer are:

What is the chaining output of a command?
When do we establish the noun type of a command’s chainable output?

I’ll make another post soon on the linguistic considerations necessary in making command chaining happen in a natural fashion.

We’re going to limit our discussion here to this restriction that the two verbs are not simply two simultaneous commands, but two commands which operate successively on an input, i.e., that it is true piping. This for example rules out input such as google dogs and translate cat to Spanish, as the second command’s execution does not semantically depend on the first’s execution. This (hopefully uncontroversial) decision also affects the linguistic considerations to be made in my next post. ↩
If you know of a command which doesn’t neatly fit into “lookup” or “action”, please let me know. ↩
I believe we should mark these no-execution lookup commands visually so the user does not expect anything to happen if they execute it. This is trac #651. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Performance vs Responsiveness —or— How I Made the Parser Twice As Fast in One Day

mitcho — Thu, 13 Aug 2009 06:16:54 +0000

Since we launched Ubiquity 0.5, the issue of Parser 2 performance has been brought up over and over within the community. By virtue of having a more flexible and localizable design, Parser 2 was expected to be slower than our original parser, but its current implementation felt noticeably—perhaps unnecessarily—slow compared to Parser 1. Parser 2 performance has been identified as one of the blockers for pushing Ubiquity 0.5+ to all of our 0.1.x users, and has thus been one of my recent foci.

The short story:

Inspired by some comments by Blair, yesterday I was able to make significant (roughly 100%) performance gains in Parser 2, resulting in 40-60% faster parses, depending on the query. This change has been committed and will be released as part of our forthcoming minor update, Ubiquity 0.5.4. Yay!

The long story: asynchronous parsing

Given that parsing in Ubiquity, combined with the post-parse of displaying suggestions, takes a good few dozen milliseconds, it is important to make sure it doesn’t block the main execution thread in order for the UI to stay responsive throughout. In other words, we needed to make it asynchronous.¹

When we began work on Parser 2 a few months ago, Blair stepped up to the plate to make it run asynchronously. For various reasons, the parser doesn’t run in a Worker thread for truer threading. Instead, what we did was to put the parser’s steps into a generator called _yieldingParse. The keyword yield is scattered in points throughout this generator.

function _yieldingParse(...) {
  // step 1
  ...
  yield true;
 
  // step 2
  ...
  {
    ...
    yield true;
  }
  ...
 
}

We then iterate over a _yieldingParse object in a function called doAsyncParse. Each time we go invoke doAsyncParse, it invokes next which advances from the last yield point of the parse to the next one. doAsyncParse checks after each step whether we should keepworking or not and then asynchronously advances to the next step by calling itself with a setTimeout. (Note the code below is a simplification.)

var parseGenerator = _yieldingParse();
 
function doAsyncParse() {
  var ok = parseGenerator.next();
  if (ok && keepworking)
    Utils.setTimeout(doAsyncParse,0);
}
 
// get this party started
Utils.setTimeout(doAsyncParse,0);

The more often we yield, the more responsive the UI would be. However, there is a certain overhead to yielding each time due to the setTimeouts we call. This point hit home when Blair told me the other day that the parser was much faster without any of the setTimeouts. Indeed, in my own testing running queries completely synchronously (replacing out all the setTimeouts), parses would run in roughly half the original time. However, by virtue of being completely synchronous, the parser would then completely lock up the UI.

I thus set out to strike a balance between performance and responsiveness by taking out and moving some of the yields in our _yieldingParse (#856).

Tests, tests, tests

Keeping this in mind, I ran a number of tests as I proceeded with my “refactoring.”

Here are some final parse time results:²

Four different queries (“hello to span”, “goo hello”, “22.7”, “tw as test” with a selection context of “hello world”) were run using each algorithm. The blue bar is the performance of the algorithm prior to adjustment of setTimeouts—that of Ubiquity 0.5.3. The gold bar is the time from a completely synchronous parse where all the setTimeouts were replaced. This algorithm completely locks up the UI, but is clearly the fastest, and should be seen as a baseline for all other yielding optimizations. The green bar is our new algorithm. As you can see, the parser is now roughly twice as fast.

Moreover, the average time difference between yields went from 0.7ms to 3.9ms which still should be no problem in terms of responsiveness.

Cancellability

This doAsyncParse is also the key for cancellability of the query. When a user changes the input or closes Ubiquity while a query is in progress, we want to cancel that query as soon as possible so the user and UI can advance. keepworking is set to false when the query is cancelled, so making sure that we yield early enough and often enough in the parse are important for issues like keystrokes being lost when typing too fast.

While the parser was indeed yielding often enough (in fact, more often than necessary) before, I noticed that our first yield was often 15-20 ms into the parse. This was because step 1 of our parse derivation was happening outside of the doAsyncParse loop. By moving this into that loop, I was able to bring this initial synchronous time down to around 10 ms. Of course, setting up the parse generator itself takes a little overhead, so this can never go down to 0, but perhaps this will improve the keystroke issue as well. I’d love to get anecdotal feedback on whether this update improves the disappearing keystrokes issue from 0.5.4 users.

This is analogous to a recent discussion of the asynchronous AwesomeBar. ↩
A note on methodology: the Parser 2 Playpen (chrome://ubiquity/content/playpen.html) was used for all testing and timing. All tests were in Firefox 3.5 on Mac OS X Leopard. My machine is a 2.4 GHz Intel Core 2 Duo MacBook Pro. No other (non-OS/daemon) apps were running. No other tabs were open and no other add-ons were installed. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Let’s talk about how cool our localizers are

mitcho — Tue, 11 Aug 2009 20:17:53 +0000

I uploaded Ubiquity to BabelZilla, an online community and tool for localizing Mozilla-style strings, just a couple days ago and we already have French and Polish complete.¹ WOW!

Granted, these are only Ubiquity’s interface strings (for example, the about and settings pages)… the parser localization and command localization have their own processes.² But this is still a tremendous accomplishment!

Hopefully we can roll some of these complete or almost-complete interface localizations with Ubiquity 0.5.4 which is a minor bugfix update coming soon. If you would like to get involved with localizing the Ubiquity interface strings into your language, get a BabelZilla login and sign up on the Ubiquity project page. Thanks again to our rockin’ localizers!

I received notification that the Polish localization in particular has completed testing and is now ready for release, as I was writing this blog post. ↩
Perhaps this anecdote is telling us that having a nice centralized web interface for localizers to work together and without messing with the files directly is a plus. Perhaps we should put up the builtin commands for localization on something like Pootle or Launchpad. Thoughts, anyone? ↩

Related posts brought to you by Yet Another Related Posts Plugin.

HookPress: Webhooks for WordPress

mitcho — Fri, 07 Aug 2009 02:23:25 +0000

I recently have spent a little time putting together a new WordPress plugin called HookPress. HookPress lets you add webhooks to WordPress, letting you easily develop push notifications or extend WordPress in languages other than PHP.

WordPress itself is built on a powerful plugin API which provides actions and filters. Actions correspond to events, so you can set a webhook to fire when a post is published or a comment is made.¹ Filters let you modify some text when it is saved or displayed, so you can have your external webhook script reformat some text or insert some other content dynamically. Not all actions and filters are supported at this time, but I will continue to add more in.

There’s a webhooks meetup in San Francisco today but I unfortunately left SF this morning, so I created a video which will be played there as a lightning talk. A demo of both types of webhooks are in the video as well.

HookPress: add webhooks to WordPress from mitcho on Vimeo.

I’m really excited by this very simple but potentially high-impact plugin. I’d love to get your comments and feedback on this new plugin and hope to hear how you’re using HookPress!

ADDENDUM: Please also follow HookPress on twitter!

My friend Abi actually has already blogged about HookPress and how it can be used to tweet on post publication. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Nountype Quirks: Day 3: Geo Day

mitcho — Sat, 01 Aug 2009 04:20:22 +0000

It’s time for one more installment of Nountype Quirks, where I review and tweak Ubiquity’s built-in nountypes. For an introduction to this effort, please read Judging Noun Types and my updates from Day 1 and Day 2.

Today I ended up spending most of the day attempting to implement (but not yet completing) major improvements to the geolocation-related nountypes whose plans I lay out here.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

`noun_type_geolocation`

noun_type_geolocation is the nountype used by the weather command for its location argument in input like “weather near Chicago”. The neat feature of noun_type_geolocation is that it has a smart default value which uses Firefox’s geolocation system to give you your current location by default, so I can enter “weather” and get the suggestion “weather near Broomfield, Colorado” (not completely correct, but close enough for the weather). Otherwise, however, noun_type_geolocation does not do too hot… for any input you give it, it’ll just accept it with a score of 0.3, much like noun_arb_text. We could do better.

One issue with this noun_type_geolocation is a conceptual one. Is this nountype supposed to accept only municipalities? Countries? Or should it accept landmarks or addresses as well? Part of the issue is that it’s only used by one built-in command in Ubiquity now, weather. But to be called a general “geolocation” nountype, its output should not be specific to weather’s usage, which is to throw the result at the Weather Underground API.

I propose that we change this to be something like noun_type_geo_town and also make similar nountypes like noun_type_geo_country, noun_type_geo_region, going all the way down to noun_type_address (which already exists—see below). All of the nountypes in this family could use a geocoding API such as Google’s or Yahoo’s. Their data properties could include all of this geocoded geographic data (in English) and also the latitude/longitude coordinate data.

The weather command could then accept noun_type_geo_town but, as some municipalities are not in Weather Underground or, for some countries, it is only as granular as administrative districts, we could just display the results of the geocoding API but then give Weather Underground the geocoded latitude/longitude data.

`noun_type_async_address`

noun_type_async_address attempts to do exactly what I’ve laid out above for the most granular level: that of geolocations with data all the way down to the street level. This is the nountype which is used for the built-in map command and uses the Yahoo geocoding service to accomplish this. Let’s see what kinds of results it returns:

input	suggestion
mitcho	mitcho	0.5
grenada	grenada	0.9
jono	jono	0.9
mountain view	mountain view	0.9

Let’s lay out some immediate quirks:

All scores are either 0.5 or 0.9. In general, if the Yahoo API returns some geocoded interpretation, it gets 0.9, but otherwise it accepts everything with 0.5.
The results that came back from the Yahoo service doesn’t add any useful information like the country or administrative region. Even the case stays lowercase.
Since when is Jono a location!? I’ll get back to this later.

For starters, the Yahoo! Maps API terms of service dictate that we can’t use its geocoding service if we’re not also displaying Yahoo maps, so I rewrote it using the Google API which also had the advantage of offering JSON output.

One quirk of the Google Geocoding API, though, is that all of the resulting municipality names are only in English. Try for example queries for Wien or 東京 (Tokyo). Since we want our suggestions to only add information to our input, not replace the input entirely (and especially not in another language), we’ll then only take results which have the input as an initial substring. On the other hand, if none of the results have the input as a proper prefix of the return value, we will take the geocoding information from the first result but with the original input as the display text. Such results will have a markedly lower score.¹

As this is the address nountype, we’ll penalize results which do not have detailed information such as street address or town-level information. All of this is very easy to judge as every result from the API has a geocoding accuracy value.

The best laid plans of mice and men…

I spent a good few hours this afternoon and evening attempting to implement this new family of nountypes, including this new nountype_geo_address, but also nountype_geo_subregion, nountype_geo_region, and nountype_geo_country. Some of the quirks of the weather and map commands, however, have prevented me from completely replacing the legacy noun_type_address and noun_type_geolocation described above. I hope to continue this work again soon and actually make this transition, ideally before 0.5.2.

Look forward to one (or maybe two?) more episode(s) of Nountype Quirks where I hope to definitively explain, analyze, and tweak matchScore, the scoring algorithm which underlies the majority of the nountypes in Ubiquity. As always, I look forward to your comments and feedback.

Bonus: Where’s Jono?

It turns out that noun_type_async_address was recognizing “Jono” as an address because Jono is actually a location afterall! Not only that, but Jono is in Japan!!

You clearly can’t take Japan out of Jono, but it turns out you can’t take Jono out of Japan either.

If this crazy algorithm raises a red flag for anyone, you’re not alone… if you think of a more elegant solution, please let me know. This will no doubt be an issue when it comes to localizing the address nountype as well. I wish we could specify an output language for the Google Geocoding API… ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Nountype Quirks: Day 2

mitcho — Thu, 30 Jul 2009 22:44:52 +0000

Today I’m continuing the process of reviewing and tweaking all of the nountypes built-in to Ubiquity. For a more respectable introduction to this endeavor, please read my blog post from a couple days ago, Judging Noun Types and my status update from yesterday, Nountype Quirks: Day 1.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

`noun_type_twitter_user`

Let’s begin again by considering the suggestions and scores that a variety of different inputs to this nountype return and see what quirks we find.

To test this nountype, I made sure I had logged into Twitter once with the login mitchoyoshitaka.

input	suggestion
mitcho	mitchoyoshitaka	0.85
mitcho	mitcho	0.5
mitchoyoshi	mitchoyoshitaka	0.94
mitchoyoshi	mitcho	0.5
test	test	0.5
テスト	none
hello world	none
@test	none

As nountypes go, this is looking pretty good. For usernames which look like logins we’ve saved before, we’re using matchScore to get decent differential scores.¹ It’s even ruling out impossible twitter username strings, according to Twitter’s own restriction:

One possible improvement we could make is to let @ strings be accepted. I went ahead and made this improvement. The initial @ will be stripped off and then will be checked as normal, but the final score will receive a slight boost using an nth root formula. The twitter command was also updated to deal with inputs with and without the initial @.

input	suggestion
mitcho	mitchoyoshitaka	0.85
mitcho	mitcho	0.5
@mitcho	@mitchoyoshitaka	0.88
@mitcho	@mitcho	0.57
test	test	0.5
@test	@test	0.57

Although the noun_type_twitter_user nountype is currently most used by the built-in twitter command to specify the user’s username, in theory it could also be used for example in a command which pulls up another user’s tweets. With that in mind, perhaps in the future we could check the browser history and/or bookmarks for entries of the form http://twitter.com/... and suggest those as well (trac #846).

`noun_type_number`

input	suggestion
text	none
0.5	0.5	1
0.5.1	none

This nountype has an incredibly simple job and does it with ease. I’m going to leave it alone.

`noun_type_date` and `noun_type_time`

noun_type_date and noun_type_time both use the magical Date.parse method to parse date- and time-like strings. Let’s first take a look at some of its suggestions:

input	`date` suggestion	`time` suggestion
June 8th 5pm	2009-06-08	05:00 PM	1
5pm	2009-07-30	05:00 PM	1
5	2009-07-05	12:00 AM	1
June 8th	2009-06-08	12:00 AM	1
today	2009-07-30	12:00 AM	1
now	2009-07-30	02:40 PM	1
5pm is a good time	none	none

The quirks in these outputs can be summed up into these two factors:

There is no differential scoring at all.
Both nountypes parse the input with Date.parse and then just spit out the date or time components of the result. Thus time-only inputs get the default date and date-only inputs get the default time with equal scores.

I just rewrote both nountypes and also added a new noun_type_date_time. Here are some of the features of the new implementation:

If the input only contains digits and spaces, it is marked down.
With the exception of the outputs ‘today’ and ‘now’, if the resulting Date object’s date is today, its date suggestion is scored lower; equivalently for time being the default value, “12:00 AM”.
Scores (with the exception of ‘today’ and ‘now’) which are shorter than the output string get a slight penalty. This factor reflects the intuition that a longer output than input means some generic information was added and thus there is less confidence in the output.

Here’s what some of the inputs give now:

input	suggestion
June 8th 5pm	`date`: 2009-06-08	0.7
	`time`: 05:00 PM	0.7
	`date_time`: 2009-06-08 05:00 PM	0.86
5pm	`date`: 2009-07-30	0.27
	`time`: 05:00 PM	0.81
	`date_time`: 2009-07-30 05:00 PM	0.49
5	`date`: 2009-07-05	0.53
	`time`: 12:00 AM	0.19
	`date_time`: 2009-07-05 12:00 AM	0.34
June 8th	`date`: 2009-06-08	0.95
	`time`: 12:00 AM	0.35
	`date_time`: 2009-06-08 12:00 AM	0.58
today	`date`: 2009-07-30	1
	`time`: 12:00 AM	0.45
	`date_time`: 2009-06-08 12:00 AM	0.7
now	`date`: 2009-07-30	0.7
	`time`: 12:00 AM	1
	`date_time`: 2009-06-08 04:34 PM	1

In addition, looking to the future we’d like to make nountypes localizable as well, and these two nountypes in particular will surely require some good thinking and planning to make localizable.

`noun_type_email` and `noun_type_contact`

noun_type_email and noun_type_contact are two closely related nountypes. noun_type_email simply validates email address-looking strings, while noun_type_contact will return the noun_type_email suggestions and additionally return contacts from GMail if available.

The first thing to note is that I’ve often found the GMail contact lookup to be finicky in my own use. Reading through the code, I discovered the solution: GMail must either be open in a tab or you must use the “stay signed in” option and close the GMail tab.² With this mystery solved, and some code cleanup done to this contact fetching, let’s take a look at some example suggestions: (suggestions overlapping with noun_type_email are not listed here)

input	suggestion
aza@m	aza@mozilla.com	0.42
jono	jdicarlo@mozilla.com	0.28
jdicarlo	jdicarlo@mozilla.com	0.19

In general, we see that these scores all look pretty poor. In particular, though, note that the “jono” input yielded a higher score for the same suggestion than “jdicarlo”, even though “jdicarlo” is longer and thus, intuitively, has more informational content and should maybe do better. Digging into the code I realized why this is. It was computing the scores by comparing “jono” and “jdicarlo” not simply to “Jono DiCarlo” and “jdicarlo@mozilla.com” respectively, but to the combined string “Jono DiCarlo ”. Now with this change in place, both the email address and name are analyzed individually and, due to the way nountype detection works in Parser 2, no duplicates are returned. Here are the updated results:

input	suggestion
jono	jdicarlo@mozilla.com	0.83
jdicarlo	jdicarlo@mozilla.com	0.85

That’s much better!

Now let’s consider the suggestions from noun_type_email. Here are what they originally looked like:

input	suggestion
bpung	none
bpung@m	bpung@m	1
bpung@mozilla.com	bpung@mozilla.com	1

noun_type_email is based on a very robust regular expression for RFC 2822. Unfortunately this means that it completely rules out strings such as “bpung” which could be a proper prefix of an email address—something that I’ve advocated for avoiding before (see footnote 2 of Judging Noun Types). Moreover, due to a quirk of how nountypes based on regular expressions are scored, all results are given the score of 1.

I just committed a change so that this behavior is improved. The new version accepts strings which match the username part of the email address spec sans @ and domain, but with a great score penalty.³ Moreover, domains which do not have a final label (the top level domain) with more than one letter (unless it’s an IP address) or do not have any periods (.) in the domain will be penalized as well. Here’s what the same inputs produce now:

input	suggestion
bpung	bpung	0.3
bpung@m	bpung@m	0.8
bpung@mozilla.com	bpung@mozilla.com	1

Same time, same channel

I hope this post sheds light on the many changes I made together as well as the underlying thought process. If you don’t agree with any particular fix or analysis, please comment! I’ll be back again tomorrow with another installment of Nountype Quirks. Stay tuned!

Again, matchScore will be the subject of another blog post in the near future. ↩
Moreover, due to the way noun_type_contact caches the contact list internally, as long as GMail’s contacts are available once, you should be able to continue accessing those contacts’ suggestions after logging out of GMail. There are also great performance benefits to this caching. The downside is that we currently have no way to know when to clear the cache, so even if you update your contacts in GMail, those new contacts won’t appear in Ubiquity until you restart Firefox. ↩
Perhaps this is a horrible idea, because if executed or previewed, any verb which uses these nountypes would have to deal with arguments which are not valid email addresses. In my mind, though, as long as it doesn’t actually cause any error, this should be okay. Keep in mind that, given the very low scores given to these suggestions, parses using it would most likely only show up if the verb which requires these nountypes was explicitly given and there are other arguments as well, for example in input like “email hello to bpung”. In such a situation, we would rather this suggestion not disappear until we type “@m”. If executed, the built-in email verb, for instance, will deal with this gracefully by simply putting the incomplete email address in the To field. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Nountype Quirks: Day 1

mitcho — Wed, 29 Jul 2009 23:00:56 +0000

Today I began the process of going through all of the nountypes built-in to Ubiquity using the principles and criteria I laid out yesterday—a task I’ve had in planning for a while now. As I explained yesterday, improved suggestions and scoring from the built-in nountypes could directly translate to better and smarter suggestions, resulting in a better experience for all users. Here I’ll document some of the nountype quirks I’ve discovered so far and what remedy has been implemented or is planned.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

`noun_type_percentage`

Here’s what a few different inputs originally returned:

input	suggestion
20	20%	1
20%	20%	1
0.2	20%	1
0.2%	20%	1
20.0	2000%	1
2 hens in the garden	2%	1

Let me highlight a couple obvious quirks:

In certain cases, where the numerical expression includes a decimal and is less than one, it is interpreted as a proportional, rather than percent, value, e.g. “0.2” → “20%”. “0.2%” is not even an option. This is the case even when explicitly adding a % sign.
All suggestions, including those where the numeral was extracted from a long string of text (e.g. “2 hens in the garden”), get the same score of 1.

I just committed a fix so noun_type_percentage now…

Counts the number of characters in the input which match [\d.%] and caps the score by (number of acceptable characters)/(length of input).
Strings which do not include “%” get a 10% penalty.
In the case of decimals less than 1 without a % sign, the proportion interpretation is also suggested (e.g. “0.2” → “20%”) in addition to the original suggestion (“0.2%”), but with a slight penalty.

Here is what they now return:

input	suggestion
20	20%	0.9
20%	20%	1
0.2	0.2%	0.9
0.2	20%	0.81
0.2%	0.2%	1
20.0	20%	0.9
2 hens in the garden	2%	0.05

`noun_type_tag`

Here’s what a few different inputs originally returned. Keep in mind that currently in this test profile, the preexisting tags are “animal”, “help”, “test”, and “ubiquity”.

input	suggestion
animal	animal	0.3
mineral	mineral	0.3
anim	animal	0.7
anim	anim	0.3
help, test, ubiq	help,test,ubiquity	0.7
help, test, ubiq	help,test,ubiq	0.3
google, yahoo, ubiq	google,yahoo,ubiquity	0.7
google, yahoo, ubiq	google,yahoo,ubiq	0.3
google, , yahoo	google,yahoo	0.3

Here are a few of noun_type_tag’s quirks:

There are only two scores ever given out: 0.3 and 0.7.
Only the last tag in the list and whether it exists or not is taken into account.
When the last tag is incomplete, the completion is suggested with a higher score, but if the last tag is exactly equal to an existing tag, it gets the lower score.

Ideally, we want noun_type_tag to look at each of the tags given to it, with higher scores for when there are more preexisting tags and fewer new ones. Keep in mind, though, that we only have to suggest the completion of the very last tag as that may be one where the user hasn’t completed typing yet… for earlier tags, we can assume (safely or not) that the user placed the comma where they meant to. We can’t teach Ubiquity to read minds, after all.¹

With this in mind, I just made a change to noun_type_tag which aims to follow these principles. The basic idea is that we start with a base score of 0.3 but then raise it via nth root for every tag in the sequence which is preexisting. Here’s what the same inputs return now. Recall that the preexisting tags are “animal”, “help”, “test”, and “ubiquity”.

input	suggestion
animal	animal	0.55
mineral	mineral	0.3
anim	animal	0.55
anim	anim	0.3
help, test, ubiq	help,test,ubiquity	0.86
help, test, ubiq	help,test,ubiq	0.74
google, yahoo, ubiq	google,yahoo,ubiquity	0.55
google, yahoo, ubiq	google,yahoo,ubiq	0.3
google, , yahoo	google,yahoo	0.3

`noun_type_awesomebar`

input	suggestion
moz	http://www.mozilla.com/	0.8
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.8
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.8
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.8

There are a couple quirks here:

All suggestions are returned with the same scores.
The nountype returns the URL of the entry as the HTML-formatted result and the title as the text-formatted result, which clearly does not make sense. However, it’s not clear to me whether the title, URL, or some combination of both is what we should be returning as the suggestion text presented to the user.²

I just rewrote noun_type_awesomebar to actually do some differential scoring. This new version also presents the URL or title depending on whichever had a better match using the matchScore function.³

input	suggestion
moz	www.mozilla.com	0.7
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.63
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.61
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.6

`noun_type_url`

The purpose of noun_type_url’s suggest function is two-fold: first, to accept strings which may look like a URL and, second, to suggest URL’s from the history just like noun_type_url, but only based on URL matches and not title matches.⁴ Here are a few sample inputs:

input	suggestion
moz	http://www.mozilla.com/	0.9
	http://moz	0.5
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.9
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.9
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.9
test	http://test	0.5
http://	http://	0.5
http:	http:	0.5
http	http	0.5
_test	http://_test	0.5
hello world!	http://hello world!	0.5

Oh, where to begin!? Here are some initial quirks… it’s possible that you could think of more!

There is no differential scoring… only 0.9 for suggestions from history and 0.5 for URL-like strings.
A number of invalid domain names are being accepted and turned into suggestions (“hello world!”, “_test”, etc.).
It’s trying to be smart by suggesting “http://” as a default URI scheme but doing so even for prefixes (initial substrings) of the word “http” itself.

With these thoughts in mind, I just took a first stab at improving this situation. Here are some features of the new implementation:

History entries are scored in the same way as in noun_type_awesomebar, using matchScore.
URLs without an explicit URI scheme (like “http://”) get a 10% penalty.
“http://” is only suggested if one of a long list of common URI schemes are not detected.
It repairs schemes which are missing a slash or two, suggesting for example “http:hello.com” → “http://hello.com”.
It actually uses Firefox’s own IDNService to check if the domain name is a valid internationalized domain name. If it’s an IDN as opposed to LDH (“letters, digits, and hyphens”), it gets a 10% penalty. If it’s not even a valid IDN, it is ruled out (see last two example inputs below).
There are also penalties for only being a domain name with no path and for the domain not having any periods (.) in it.

Here is what our suggestions now look like:

input	suggestion
moz	http://www.mozilla.com/	0.6
	http://moz	0.65
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.63
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.61
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.6
test	http://test	0.65
http://	http://	1
http://	shttp://	0.75
http:	http://	0.9
http:	shttp://	0.7
http	http://	0.72
	https://	0.71
	shttp://	0.68
	http://http	0.65
_test	none
hello world!	none

See you tomorrow~

Alright, enough nountype wrangling for one day. I’ll be back again tomorrow for another installment.

If we could make assumptions about what tags look like, for example that they are always pretty short, or use certain character classes, we could use such factors as well to judge non-preexisting tags for “tagginess” but unfortunately it’s possible (though unlikely) that a user would prefer really long tag strings and of course Firefox allows tags in any unicode code range. The only strings we can immediately rule out as impossible are ones which are purely whitespace. ↩
It’s actually unclear whether the method we’re using (nsIAutoCompleteSearch) is actually searching titles or not… it currently looks like it’s only looking at the URL’s. Perhaps the title query is what we’re supposed to enter in the mystery parameter. ↩
I hope to discuss the matchScore function in a separate blog post later. ↩
While writing up this section I ran into a bug whereby when both noun_type_awesomebar and noun_type_url are active, only one of their async callbacks from Utils.history.search are returned. Thus, if lucky, only one of the nountypes will return the history results and if unlucky the parse query will not complete. Filed as trac #845. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Judging Noun Types

mitcho — Wed, 29 Jul 2009 06:39:11 +0000

Introduction

Different arguments are classified into different kinds of nouns in Ubiquity using noun types.¹ For example, a string like “Spanish” could be construed as a language, while “14.3” should not be. These kinds of relations are then used by the parser to introduce, for example, language-related verbs (like translate) using the former argument, and number-related verbs (like zoom or calculate) based on the latter. Ubiquity nountypes aren’t exclusive—a single string can count as valid for a number of different nountypes and in particular the “arbitrary text” nountype (noun_arb_text) will always accept any string given.

In addition to the various built-in nountypes, Ubiquity lets command authors write their own nountypes as well.

The functions of a noun type

Nountypes have two functions: the first is accepting and suggesting suggestions and the second is scoring.

Accepting and suggesting

Nountypes don’t just have to accept the exact string they were given—they can also return suggestions which are based on that input. For example, the noun_type_language can take the input “span” and return “Spanish.” A nountype can return multiple suggestions which may or may not include the trivial suggestion, i.e. the original input as is. If there is no way that that input could possibly be part of an accepted value, it should return no suggestions, i.e. [].²

Scoring

Ubiquity 0.5 with Parser 2 introduced the notion of a nountype suggestion score. For example, two different nountypes can accept the same input, but with different scores. Scores range from 0 to 1 where 1 is a perfect or exact suggestion and 0.1 or so is a very very improbable suggestion.³ These scores are used in the scoring of parses. Because verbs specify certain nountypes for each of their arguments, the scores that individual nountypes return for each argument are a crucial component of the scoring algorithm and can even determine whether a parse is returned or not.

With this in mind, you may be tempted to make your nountype return a score of 1 on any input so your verb will show up in the suggestions highly. While this would work, it will only act to make your verb annoying and a poor Ubiquity citizen. Appropriate scores must be given to noun suggestions, with higher values reflecting confidence and lower values reflecting imprecision. But how exactly do you figure out what’s an appropriate value?

Judging nountypes with the Nountype Tuner

The Nountype Tuner is a new tool I’ve been building to help both Ubiquity core developers and command authors to check their nountypes against others and to “tune” their behavior and scores. The nountype tuner will take your input and throw it against all of the nountypes referenced in your active verbs and display the suggestions returned with their scores. You can think of it as the Playpen’s little sister.

The Nountype Tuner can be found at chrome://ubiquity/content/tuner.html, though I am pretty sure it is broken in Ubiquity 0.5 and 0.5.1. It has been fixed now and I will make sure it’s in good shape for 0.5.2.

The heart and soul of the Nountype Tuner is this scale:

This scale tells you, in plain English, what different scores represent and correspond to, in two sets of vocabulary: “in terms of a guess” and “in terms of a match.” While still subjective, this scale helps developers just different input/output pairs and their scores. For example, “lian” → “http://lian” is given 0.5, so it’s an okay guess or a possible match… does that seem right to you? Or “lian” → “Italian” being between “okay” and “good.” Appropriate? We can look at such statements, decide how we feel about them, and tweak if necessary.

Good nountype scores have roots

CC-BY Aaron Escobar

…not that kind of root, but more like this kind of root… let me explain…

When comparing the scores that individual nountypes return for different inputs, we must compare those scores within the same nountype’s family of suggestions to see if higher scores truly correspond to higher confidence. For example, the language nountype should give the suggestion “French” for both the inputs “f” and “fren,” but the scores of these suggestions should be different—i.e. the score of “f” → “French” should be much lower than the score for “fren” → “French,” reflecting the additional informational value. We refer to this relation of the scores of successive prefixes of a single suggestion all returning that same suggestion as the score curve and in general it should be non-decreasing.⁴

One could say the most trivial score function then is the linear one. For a series of converging prefixes of the same suggestion (“Dutch”), under a linear approach we could naively let the score be (length of the input)/(length of the suggestion), as below:

the linear model

input	d	du	dut	dutc	dutch
output	Dutch	Dutch	Dutch	Dutch	Dutch
score	0.2	0.4	0.6	0.8	1

This linear model is represented below by the black line.

The problem with the linear model is that earlier transitions (additional keystrokes) add more information than the later ones. Once we’ve entered “dutc,” after all, we would like to be pretty darn sure that we mean “Dutch,” so the score difference between “dutc” and “dutch” should be less than the score difference between, say, “d” and “du.” We want a score curve that looks more like the solid or dotted red lines above.

For this reason, I strongly advocate the incorporation of an nth-root in the score computation. Nth-rooted score functions over [0,1] have the feature that they are increasing but also that earlier transitions affect the score more than later ones, which is exactly what we’d like to see. (The solid red line above is x^1/2 and the dotted one is x^1/3.)⁵

Conclusion

Properly tuning both the built-in nountypes and custom nountypes is crucial to producing more accurate and relevant parse suggestions. I’ll be using the principles and criteria laid out above, combined with the new Nountype Tuner, to tune the built-in nountypes (trac #746) in the coming days in preparation for our 0.5.2 release. I invite you to use the Nountype Tuner in 0.5.2 to tune your custom nountypes as well.

Or, as I often write them, “nountypes.” ↩
Note that I didn’t just say “if the input is not an accepted value…” That’s because, based on the left-to-right nature of text input, an argument may later become a valid input of a certain nountype with a few more keystrokes. For example, if we had a URL nountype which accepted “http://mitcho.com” but not “http://mitcho”, any command which took this nountype would not show up in the suggestions while we were typing out “http://mitcho”… but would suddenly appear when we completed the “.com”. The best practice here is to suggest a valid value for the initial “http://mitcho”, like “http://mitcho.com”.
(In reality, I should have said “initial-to-later nature” to be fair to right-to-left languages, but you get the idea. Speaking of which, serious consideration of Ubiquity in right-to-left languages is long overdue.) ↩
In reality, due to the way parses are scored and the fact that noun_arb_text accepts anything with score 0.3, a suggestion with score below 0.3 is probably not worth even giving out. Notable exceptions are for custom noun types which are used in commands which take multiple arguments… in these cases, even scores below 0.3 could add up and overtake a noun_arb_text parse, but it’s rare. ↩
The idea that successively longer inputs should yield successively higher scores only makes sense (1) when they are converging on the same suggestion output and (2) when these are truly suggestions, not just acceptances. For nountypes which accept the input verbatim, suggestion scores need not increase… for example “1” is just as good a “number” as “1234” is, so both of their respective suggestions, “1” and “1234” could be given the same score. ↩
Unfortunately the Nountype Tuner currently only compares the suggestions of one input across a number of nountypes, not a number of inputs across the same nountype. In the future I’d like to make the Nountype Tuner be able to produce these sorts of score curves as well. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Report from SIGIR Workshop on Information Access in a Multilingual World

mitcho — Fri, 24 Jul 2009 19:42:27 +0000

Yesterday I participated in and presented at a workshop on Information Access in a Multilingual World at ACM SIGIR in Boston. The focus of the workshop was on cross-language information retrieval (CLIR). Cross-language information retrieval systems enable users to retrieve relevant information across different languages for a certain task or query. Even if you have a budget to translate some documents from a foreign language to your language, how do you find the relevant documents to translate in the first place if you don’t speak (or read) that source language? This is the type of problem that CLIR aims to solve.

The keynote speaker was Ralf Steinberger of the European Commission’s Joint Research Centre, presenting the European Media Monitor family of applications. EMM is a suite of different applications all based on a core platform which aggregates news stories from a variety of sources around the world in a few dozen different languages. The system uses various CLIR techniques to then cluster stories by event, regardless of language or source country. Large news agencies and organizations, as well as the European Commission itself, use the system to track upcoming news stories as well as health concerns. In the European Union which has over a dozen different “official” languages, there is a great need for this type of service. The applications are available to the public so I invite you to play around with them at the EMM site.¹

The workshop’s paper presentations were quite diverse and all interesting. Elena Filatova presented an interesting approach to measuring the “trustworthiness” of statements in Wikipedia entries, by comparing the overlap in content statements between different language entries (more overlap => more trustworthy) which can be used to create inverted pyramid summaries. Perspectives from different use cases were also examined through presentations on news analysts and health organizations, patent searches, and medical information for personal use, as well as discussion of the need for CLIR for historical and religious texts.

One interesting thread throughout the day’s sessions was the issue of loanword processes and the possible use of romanization as an interlingua. Cross-language named entity recognition is a major problem in CLIR. Many novel words and names (common in news articles) go through processes of loanword adaptation and transliteration and are hard to identify and also are not in the systems’ dictionaries. A few of the talks touched on these problems, including a great talk by Kashif Riaz of the U of M on the salient (and great) differences between Hindi and Urdu. Frederic Gey described a number of different approaches to comparing strings via a romanization interlingua.

Another interesting thread was the idea of the target user. The needs of CLIR applications can vary greatly depending on its use case and the users’ savvy. The needs of a patent search office, where many professional searchers are already multilingual, is clearly different from the needs of an executive hoping to stay on top of the world news related to their organization. It was brought up, however, that in this age of open data and API’s, if a CLIR resource provides a good API, it need not necessarily supply the perfect interface for every type of user, as different third parties could also develop such targeted interfaces.

While most of the presentations and research interests in the room were on users accessing resources in various languages, I presented on Ubiquity. The talk was intended to highlight the idea that the opposite problem of users with different languages getting at the same kinds of information and getting equally powerful user experiences is also a different but worthwhile problem. Below are the slides from this talk. As a web archive of all the papers will be set up soon, I believe it’s safe to put up my paper as well, so please check it out. It’s a good short (four pages) overview of the innovative approaches we’ve taken to build a localizable natural language interface.

Ubiquity: Designing a Multilingual Natural Language Interface

“Ubiquity: Designing a Multilingual Natural Language Interface.” Presented at SIGIR Workshop on Information Access in a Multilingual World, Boston, July 2009. To appear in a proceedings. 240 kb - pdf

Unfortunately, there is no public API available for these resources. We asked. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

mitcho.com » Mozilla Planet

Checking mochitest test coverage

Happy Halloween!

Beginning development with Jetpack SDK 0.2

Installing Python

Setting up the Jetpack SDK

The package directory structure

Creating the package

Writing the main code

Testing our package

Using a standard library

Creating a custom library

Using our custom library

Implementing a network status observer

Adding documentation

Exporting an install package

Spring is for Speaking: JSConf, WordCamp SF, IACL

Jetpacking in Boston

Jetpack Ambassadors in MV

Mashing up the browser in Maine

Mozilla By The Numbers

The Aliens Aliases Have Landed

The use case for and importance of CreateAlias

Creating an alias

Bonus: using CmdUtils.previewCommand and CmdUtils.executeCommand

Exploring Command Chaining in Ubiquity: Part 2

Introduction

Simple syntaxes: sequential vs embedding strategies

Sequential and embedding strategies in natural language

The natural syntax of chaining

Parsing the sequential strategy

Conclusion

The Ubiquity Persistence Project: exploring a persistent Ubiquity in the toolbar

Exploring Command Chaining in Ubiquity: Part 1

Technical considerations: hooking the pipes together

Matching inputs and outputs

(A)synchronous composability

Conclusion

Performance vs Responsiveness —or— How I Made the Parser Twice As Fast in One Day

The short story:

The long story: asynchronous parsing

Tests, tests, tests

Cancellability

Let’s talk about how cool our localizers are

HookPress: Webhooks for WordPress

Nountype Quirks: Day 3: Geo Day

noun_type_geolocation

noun_type_async_address

The best laid plans of mice and men…

Bonus: Where’s Jono?

Nountype Quirks: Day 2

noun_type_twitter_user

noun_type_number

noun_type_date and noun_type_time

noun_type_email and noun_type_contact

Same time, same channel

Nountype Quirks: Day 1

noun_type_percentage

noun_type_tag

noun_type_awesomebar

noun_type_url

See you tomorrow~

Judging Noun Types

Introduction

The functions of a noun type

Accepting and suggesting

Scoring

Judging nountypes with the Nountype Tuner

Good nountype scores have roots

the linear model

Conclusion

Report from SIGIR Workshop on Information Access in a Multilingual World

The use case for and importance of `CreateAlias`

Bonus: using `CmdUtils.previewCommand` and `CmdUtils.executeCommand`

`noun_type_geolocation`

`noun_type_async_address`

`noun_type_twitter_user`

`noun_type_number`

`noun_type_date` and `noun_type_time`

`noun_type_email` and `noun_type_contact`

`noun_type_percentage`

`noun_type_tag`

`noun_type_awesomebar`

`noun_type_url`