In order to facilitate this discussion, we created the Persistence tool. With the Persistence tool you can quickly try out new design and interaction ideas, mocking things up with some simple jQuery-powered JavaScript and CSS and see your changes live. The Persistence tool is bundled with our latest Ubiquity beta (install link).

The Ubiquity Persistence Project: exploring a persistent Ubiquity in the toolbar from mitcho on Vimeo.

I just put together a screencast introducing the initiative, demoing the Persistence tool, as well as talking about this project’s relation to the ongoing work on Taskfox. We’ll look forward to your comments and designs!

Related posts:

1. User-Aided Disambiguation: a demo
2. Count command for Ubiquity
3. Exploring Command Chaining in Ubiquity: Part 1

Related posts brought to you by Yet Another Related Posts Plugin.

Command localization in Ubiquity 0.5

A few important design decisions have already been made in implementing command localization in Ubiquity 0.5. The first was the choice of the gettext po (portable object) file format. The po format is a de facto industry standard with many tools built for the format and this design choice hopefully lower the bar for prospective localizers.

Second, in order to simplify the matching of localizations and commands, we require that each command feed have one localization po file, rather than splitting the localizations of different commands across multiple po files.

Finding the appropriate localization po file for a JavaScript command feed is very simple with bundled commands as the files are managed in our main repository so we know exactly where to find them. We just take the command feed’s filename sans extension—the feed key—and look for localizations/lang/feed key.po, where lang is the active language code. This gives us a simple one to one relationship between the JavaScript command feed source and the appropriate localization.

Background

The ability of users to easily write their own Ubiquity commands has always been a huge strength of the Ubiquity platform. Users can also “subscribe” to commands written by other users on other servers. In this case, local copies of those command sources are made.

The herd was developed as a dynamic aggregator of community Ubiquity commands. The herd keeps its own copy of each command source. The herd groups mirrors of command feeds on multiple servers together as well, giving each unique command feed a unique ID.

Localizing distributed resources

This distributed nature of the command feeds is exactly what complicates the localization of community commands. The original sources themselves are distributed, and we also want a community of localizers to be able to localize the commands—i.e., for the localizations to be (in some sense) distributed. Here I will present three options which I believe could be the basis for a winning solution, in order of more distributed to less distributed.

All three of these options have the property that localizations do not need to be “approved” or “managed” by the command author. For example, a possible standard where URL’s of localizations must be in the command feed is not considered. I believe this is a crucial property of any approach we decide on.¹

Option 1: a completely distributed option

In this option, localizers simply put the po files on their own servers (or some code snippet site such as github) and the user must then subscribe to those localizations much as they subscribe to commands now.

Option 2: registration and discovery through the herd

In this option, localizers put po files up on their own servers and then register that po file’s URL with the herd. The herd keeps track of each command feed’s localizations in different languages.

Option 3: localizations on the herd

In this option, po localizations are uploaded right onto the herd. The herd is the centralized repository of all localizations for each command feed.

Users first

In coming up with a criteria for judging different models of community command localization, I think it is helpful to think of the end-user experience. Right now to subscribe to a new command a user must find the command (perhaps via the herd) and click on the subscribe button, then in most cases confirm that they are aware of the possible dangers and confirm subscription.² What work is required for a user to subscribe to a new command and get it in their language?

Under option one, the user would somehow have to find the localization scattered someplace on the internet of their own accord, and then install that localization by themself. In my mind, this is clearly a no go. With options two or three, however, when a user subscribes to a command feed Ubiquity can check with the herd to see whether there are any localizations available. The localizations could be offered to the user or the localization for the currently active language could be automatically installed. There are, on the other hand, disadvantages to requiring a centralized authority, exemplified by the fact that the current iteration of the herd itself has often been down.

Summary and a call for comments

As laid out in this visual guide, I personally have a couple main criteria which I believe we should follow:

1. localization independence: Command authors should not have to manage their commands’ localizations. (See footnote 1.)
2. friendly discovery and subscription: Users should not have to go out and find localizations by themselves. Localizations should be offered to the user.

With these criteria, I’m pretty sure the only logical conclusion is that we need some level of centralization, pointing to options two and three. If you think of another option which satisfies these criteria, or disagree with the criteria above, I would love to know. In addition, have you encountered similar problems of localizing distributed resources elsewhere? What worked there?

If this problem has been solved before, there is no need to reinvent the wheel. As far as I can tell, however, this could be a particularly hairy problem.

Related posts:

1. Ubiquity Localization Update
2. Localizing Commands for Ubiquity 0.5
3. Ubiquity Localization: What’s New, What’s Next

Related posts brought to you by Yet Another Related Posts Plugin.

追伸： ７月３日現在、 Ubiquity 0.5 のリリースを遅らせる方向になったので、残念ながら今日はリリースされません。是非リリース後インストールしてみてください。

Ubiquity 0.5 日本語紹介ビデオ from mitcho on Vimeo.

As Ubiquity 0.5 will be released soon (Thursday morning in Mountain View), I decided it was a good time to put together a screencast in Japanese demoing the use of the new Japanese parser and commands.

Related posts:

1. Changes to Ubiquity Parser 2 and the Playpen
2. Foxkeh demos Ubiquity Parser: The Next Generation
3. A Demonstration of Ubiquity Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.

There is no myriad of buttons and fields to choose from. All the user has to do is directly type in what he wants.

Here are a couple other choice quotes which will ring true for the Ubiquity users and internationalization folks in the audience:

For this to work without driving the user mad, the natural language parser has to be near perfect. The last thing I want is for this to come out smelling like AppleScript.

Problems
This design isn’t perfect as it has two glaring problems. One is that the user has no easy way of discovering how complex the recurrence rules can be. This isn’t such a huge problem, but a way to solve this is to include a help button to show example rules or to include an accompanying iCal style UI to let the user setup the recurrence rule in a more typical fashion. I didn’t include these in the initial implementation though because I wanted to see how users would react to this kind of UI.
Another problem is localization. Even if I write parsers for a few more popular languages, it won’t accommodate the rest of the users in the world. Again, the solution is an accompanying traditional UI, but for now, I’m leaving it the way it is until I get some feedback.

There’s a trend in the wind, my friends: the incorporation of near-natural language for more humane interfaces.

Related posts:

1. How natural should a natural interface be?
2. Friendlier command feed subscription
3. User-Aided Disambiguation: a demo

Related posts brought to you by Yet Another Related Posts Plugin.

A week or two ago while visiting California, Jono and I had a productive charrette, resulting in a new architecture proposal for the Ubiquity parser, as laid out in Ubiquity Parser: The Next Generation. The new architecture is designed to support (1) the use of overlord verbs, (2) writing verbs by semantic roles, and (3) better suggestions for verb-final languages and other argument-first contexts. I’m happy to say that I’ve spent some time putting a proof-of-concept together.

I’ve implemented the basic algorithm of this parser for left-branching languages (like English) and also implemented some fake English verbs, noun types, and semantic roles. This demo should give you a basic sense of how this parser will attempt to identify different types of arguments and check their noun types even without clearly knowing the verb. This should make the suggestion ranking much smarter, particularly for verb-final contexts. (For a good example, try from Tokyo to San Francisco.)

➔ Check out the Ubiquity next-gen parser demo

Clicking on the environment info will give you some information on the specific verbs, noun types, and roles implemented. You can also scroll through the current parse section to see the step by step derivation of how the suggested parses were constructed.

I’ll be flying about 15 hours in the next hour as I make my way back to Japan… hopefully I’ll make some more progress on the plane! I’ll look forward to your comments! For those of you interested in checking out the code yourself, you can find it on BitBucket.

Related posts:

1. Writing commands with semantic roles
2. Ubiquity in Firefox: Focus on Japanese
3. User-Aided Disambiguation: a demo

Related posts brought to you by Yet Another Related Posts Plugin.

The inspiration for this idea came from Aza’s blog post “Solving the ‘it’ problem” which advocates for this type of quick feedback to the user in cases of ambiguity. Such a method would help both the user better understand what is being interpreted by the system, as well as offer an opportunity for the user to correct improper parses. I just tried mocking up such an input box using jQuery.

➔ Try the User-Aided Disambiguation Demo

If you have any bugfixes to submit or want to play around with your own copy, the demo code is up on BitBucket. ^^ Let me know what you think!

Related posts:

1. How natural should a natural interface be?
2. Ubiquity in Firefox: Focus on Japanese
3. Friendlier command feed subscription

Related posts brought to you by Yet Another Related Posts Plugin.

If you’ve ever subscribed to a new Ubiquity command before, you know the red screen of doom. Ubiquity currently takes users to this page every time they wish to subscribe to a new command. The current design is meant to encourage users to be aware of the possible security implications of enabling and executing a command, to avoid getting a trojan horse.

The current screen, however, does not make subscribing to commands foolproof. I personally know I’ve subscribed to a number of commands without reading through the code, defeating the purpose of the anti-trojan horse display. Moreover, the page doesn’t give you any information on how you can use this new command. Especially given the inherent limited discoverability of a natural language interface, taking a moment to help the user actually learn the command becomes key.

Today I did a quick mockup of what a friendlier command feed subscription page might look like. Take a look at this screenshot with some of the features marked:

You can also check out the page itself. If you’d like to visualize it without the “trust” warning, you can also view the trusted version.

This mockup here is but a first iteration. What do you think about this subscription page? What is missing? What should be changed?

Related posts:

1. How natural should a natural interface be?
2. Ubiquity in Firefox: Focus on Japanese
3. selecting from Ubiquity

Related posts brought to you by Yet Another Related Posts Plugin.

今後多様な言語に対応したFirefox内のUbiquityを検討していきますが、その中でも今日は日本語をとりあげます。後日日本語で同じ内容を投稿するつもりです。^^ 日本語でのコメントも大歓迎です！

What commands look like in Japanese

Japanese is not only just a verb-final language but it is strongly head-final, meaning it has postpositions instead of prepositions, direct objects come before verbs, and adjectives precede nouns. In terms of how it identifies its arguments, every argument has a postposition/case marker (called a particle in the Japanese literature) which marks its role in the sentence.

A couple common particles we’ll look at in this example include -を (-o) which marks the direct object (accusative case, you might say) and -に (-ni) which acts like English “to” (dative case). The example sentence we’ll look at today is:

(Note: ʔ is a glottal stop. ACC=accusative, DAT=dative, and IMP=imperative form.)

That final ください is often dropped in very casual speech and, as it adds no new information, we’ll assume today that the user will not enter it. Finally, Japanese doesn’t use spaces in their orthography, so the actual input would be “ケーキをブレアに送って”.

Mockup 1: Particle identification

One of the major hurdles in working with Japanese is that there are no spaces between the words. The natural first step is to split the sentence up into words, but this is a very difficult problem in NLP which big name research groups actively work on.

Fortunately, however, in “Solving the ‘It’ Problem” Aza suggests that, when we encounter ambiguity in our input, we can go ask the user. Great minds think alike, and computer scientist Jean E. Sammet suggested the same idea way back in 1953:

Using English [or any other natural language] definitely involves the requirement for the computer (or more accurately its programming system) to query the user about any possible ambiguity.

Parsing a sentence into words, in the limited context of Ubiquity, is really about identifying the particles which mark the end of each argument. Here’s a mockup of an application of the Sammet-Raskin Method to this problem:

Pros: This completely takes care of the word-breaking problem, with minimal arbitration from the user. The parser knows exactly what arguments it’s dealing with and the visual feedback means the user won’t be surprised by the parse.

Cons: Most of the particles/postpositions we’d have to deal with are a single character, so they may show up pretty often within words, in which case it would be quite annoying to have to press escape after each one.

An even smarter system, when wanting to mark a character as a particle, would first check to see that the argument (before the particle) is a valid argument type for that particle. If the check fails, it doesn’t have to bother with suggesting that character as a particle. This may cut down on the false positives.

Smart suggestions: what works, what doesn’t

One of the key suggestions in Aza’s mockups include a way to choose the prepositions while entering your arguments, based on the current verb.

For example, here, the translate command accepts a direct object, a to-object, and a from-object, so little to and from markers magically show up on the right side, making the appropriate prepositions (and by extension the appropriate arguments) discoverable. I think this line of thinking is a really good one, at least for English.

In a verb-final language, however, you enter the arguments first and then the verb, making this strategy of suggesting appropriate arguments impossible. Note that in the user-contributed spreadsheet of how languages identify their arguments we see that about a quarter of the languages we looked at are verb-final—that is, with Subject-Object-Verb canonical word order.

Instead of seeing this as a disadvantage, however, let’s see what verb-final order allows us to do.

Mockup 2: A different kind of suggestion

Not all verbs allow for every different kind of particle. For example, it doesn’t make sense to have a -に (-ni, “to” or dative) argument for a verb like 検索して (kensaku-shite, “search for”). In English we used this to suggest different types of arguments given a specific verb. In a verb-final language, we could do this backwards.

Pros: This makes verbs highly discoverable, given a certain argument structure. For example, if you enter a few arguments, like a direct object, a “to” argument, and a “from” argument, it’ll suggest verbs that will do something to an object from somewhere to somewhere else. This way, you can easily try out verbs you didn’t even know existed. It’ll only give you verbs appropriate for your arguments, reducing the chance of writing a an infelicitous command.

Cons: Without knowing what kinds of actions are available, it may be difficult to know what kinds of arguments to enter in the first place. If you have a specific verb or service you want to use it may be counterintuitive or downright tricky to start by guessing the right set of arguments.

In addition, from a technical point of view, this requires much of the prediction algorithms in English Ubiquity to run backwards. Ideally, there would be a closed (predetermined) class of particles and a predefined set of noun types. Verbs would not be able to define their own modifiers and noun classes as easily or freely as they can now.

Conclusion

The properties and challenges of Japanese grammar require that we not try to outright copy the English behavior but to think about what really makes sense in that language and that may be an important lesson as we move toward designing a localizable Ubiquity. Please post your questions and criticisms of this design or post your own mockups!

Related posts:

1. Three ways to argue over arguments
2. Contribute: how your language identifies its arguments
3. How natural should a natural interface be?

Related posts brought to you by Yet Another Related Posts Plugin.

Every day, more users are trying out Ubiquity, the Mozilla Labs experiment that lets users accomplish common Internet tasks faster through a natural language interface. As we live more and more of our lives on the web, there is a huge appeal to—and need for—a faster way to access and mashup our information.

But what exactly do we mean by a “natural language interface”? Is it just another programming language with lots of English keywords? Should the final goal be a computer that understands everything we tell it?

As we think about the future directions and possibilities of Ubiquity, we need to go back to our roots and understand the project’s motivations. With that in mind, here are some initial thoughts on the advantages of a natural language interface. The ultimate goal here is to refine the notion of natural language interface and to come up with a set of principles that we can follow in pushing Ubiquity further, into other languages and beyond.

Why language?

In his 2008 article in interactions, Aza describes a clear need for modern UI to move beyond monolithic do-everything apps into efficient, granular commands that can be connected to accomplish tasks. Hierarchical menus with an application’s every possible function are great for discoverability, but slow and inefficient as they grow. Aza advocates for the use of a familiar subset of natural language to this end. In his own words,

Words can capture abstractions that pictures cannot because language has an immense amount of descriptive and differentiating power. Abstract thoughts are exactly represented by the words that give them names. It is this power that comes to the rescue in specifying functionality.

In other words, language gives us the descriptive power to succinctly and creatively express our will, far faster than a series of menus, and with more freedom than a series of shortcuts or gestures. In addition, by tapping into the lexicon of our every day language, we make a direct attack on the learnability problem.¹

The natural syntax test

The ability to string different commands together is not a novel one—indeed, this is what more traditional command lines and programming languages offer. However, these technologies present a huge barrier to the layperson, even for languages with many keywords from English or English-like syntax.

Programming languages can be such teases in this way. Often the first bits of code in a language look remarkably similar to natural language (Python):

print "Hello World"

…but the young coder is quickly disappointed:

print map(lambda x: x*2, [1,2,3])

AppleScript is a language which tries to take this idea further and, indeed, sometimes AppleScript code constitutes readable English.

print pages 1 thru 5 of document 2

Dig a little deeper, though, and AppleScript also fails the “natural syntax” test. In fact, it can be argued that a language that looks like a natural language but differs in some important details can be even more difficult to use than one that is completely novel. Bill Cook, one of the original developers of AppleScript, makes this point in his history of AppleScript: “in hindsight, it is not clear whether it is easier for novice users to work with a scripting language that resembles natural language, with all its special cases and idiosyncrasies.”

If the interface’s syntax is too restrictive or, worse, conflicts with a user’s natural intuitions about their natural language, it immediately fails to be “natural”, no matter how similar the keywords or grammar is.²

Towards a natural (and forgiving) syntax

Aza similarly laments the relegation of text-based interfaces to the higher echelons of geekdom in his 2008 paper: “if commands were memorable, and their syntax forgiving, perhaps we wouldn’t be so scared to reconsider these interface paradigms.”

The key word “forgiving” above (emphasis mine) is two-ways ambiguous, both of which we want a natural language interface to be:

1. Forgiving as in “not difficult to learn and remember”: the syntax must be easy and natural for the user, encouraging experimentation and intuitive application;
2. Forgiving as in “not correcting or prescriptive”: the system should try its darndest to accept the user’s input, even if it’s not the most “well-formed.”

From an implementation point of view, (2) above can also be an advantage. There are many grammatical restrictions in natural language which, as long as the command is unambiguous, Ubiquity need not enforce on the user. Take, for example, the two statements:

1
2

print two copy
print two copies

I feel that Ubiquity should execute both of these statements with equal ease. The numeral “two” makes the user’s intent very clear, even though the plural of “copy” should indeed be “copies.” It need not be the job of the interface to decide whether a sentence is “correct English.” By assuming that the user is trying to communicate a valid and possible task, rather than throwing up an error, the system will be more flexible and more forgiving in the inevitable case of human error. The ultimate goal should be to help the user accomplish their task.

Conclusion

By developing a language interface which truly feels natural to the user, we can successfully bring the power of text-based interfaces to the masses. I feel the key to this “natural-ness” is a less restrictive and in fact forgiving syntax. While this goal akin to natural language programming may be daunting from an implementation angle, and it may indeed prove impossible, as long as the goal is to execute simple imperative commands, the scope of the target syntactic structures is limited.

Ubiquity as it stands is many different things for many people. The natural language guidelines above may feel too restrictive to many current developers for whom Ubiquity is simply a convenient new way to extend Firefox.³ This discussion also seems orthogonal to the mouse-based Ubiquity experiments. As users and developers, how do you feel about the potential benefits and downsides of these natural syntax guidelines? In the coming days I’ll look at some concrete examples of what this “forgiving” syntax would demand of Ubiquity.

Related posts:

1. Using Templates with YARPP 3
2. Testing Google’s Language Detection
3. 回収 vs. 収集 and Better Word Meanings Through Usage

Related posts brought to you by Yet Another Related Posts Plugin.