Today I hope to cover some of the technical considerations required in implementing command chaining in Ubiquity, and I will follow up soon with a blog post on the linguistic considerations required as well.

Technical considerations: hooking the pipes together

I’d first like to lay out some technical challenges and questions. These can be broken into two different categories: (1) how the parse and display of suggestions is affected and (2) how the execution is affected.

Matching inputs and outputs

We’ll first consider how command chaining may affect the parsing. Ubiquity commands each specify the types of argument inputs that it expects using different noun types, such as noun_arb_text which accepts anything, noun_type_number which accepts numbers, or noun_type_language, which takes the name of a language. For example, the translate verb takes maximally three arguments: a noun_arb_text object, a noun_type_language goal (the language to translate into), and a noun_type_language source (the source language). In implementing command chaining, it will be necessary to identify the appropriate noun types for the output of a command.

The first question we must address here is “what is the chaining output of a command”? Is it the preview text? Some text output from the execution?

Big fish eat da lil fish by joemud, CC-SA-NC

To put this question into perspective, we note that Ubiquity commands can be broadly classified into two types: lookup and action execution. Here’s a classification which I believe to be exhaustive:²

In light of this classification I believe we can say that lookup commands are much more likely to be the first verb in a command chain—conversely, chains such as email hello to Blair and then do ... or twitter hello and then ... are quite unlikely. What is much more likely is for the first verb to be a lookup function.

Thus in the same way that not all commands have a useful execution³ perhaps only lookup commands will have a chainable output: the results of the lookup. Even with this restriction, we will most likely need to implement a new “chainable output” method or getter in these commands. This means that commands will need to opt-in to become chainable, but I believe this is a necessary evil.

The second question we must address is “when do we establish the noun type of a command’s chainable output?” One unsung but crucial feature of the way Ubiquity works now is that suggestions’ previews are not computed until that suggestion is selected (except for the first suggestion, which in most skins gets previewed immediately). Should we wait for all of the first verbs’ chainable output to be computed and then run them through the noun type detection system? Or should verbs with chainable output also a priori specify what noun types their output will be?

Both of these approaches have their problems. If we compute the chainable output of the first verb, run a noun type detection on it and then suggest the full combination if it matches what the second verb was expecting, this will have clear performance implications, not to mention that it could greatly complicate our parsing algorithm. While the latter approach doesn’t have these performance implications, it does mean that it will have to list (by name or reference) the noun types that will match its output, meaning that if a command author is unaware of someone else’s noun type, that chain will be impossible, even if the chainable output itself does indeed match that noun type. The a posteriori approach would never have this issue. What other benefits or problems do you forsee with either of these approaches? Is there another approach which avoids these pitfalls?

(A)synchronous composability

Once we have the noun types, parsing, and suggestions down, all that remains is to compute the previews and implement the composite execution. Since the Ubiquity command manager already wraps the preview and execute functions in a wrapper to facilitate localization, among other uses, it would be easy to make the command manager compose asynchronous processes pseudo-synchronously. No major changes should be necessary to do the previews and executions, though, again, there will be a performance cost.

Conclusion

There are a number of technical questions which must be answered, mostly in the parsing/suggesting stage. The key questions to answer are:

1. What is the chaining output of a command?
2. When do we establish the noun type of a command’s chainable output?

I’ll make another post soon on the linguistic considerations necessary in making command chaining happen in a natural fashion.

Related posts:

1. Command Chaining with Oni?
2. Ubiquity Commands by The Numbers
3. Ubiquity Parser: The Next Generation Demo

Related posts brought to you by Yet Another Related Posts Plugin.

As we look to move forward with incorporating the next generation parser into Ubiquity proper, it thus becomes important to take a look at the current command ecosystem to see how possibly disruptive this move will be. To this end last night I wrote a quick perl script to scrape the commands cached on the herd and get some quantitative answers to my questions.

(1577 different verbs were analyzed. None of these computations below are weighted by feed popularity.)

Q: Are there a lot of commands which use more than one argument?

A: The vast majority (>85%) of commands take one or no arguments, requiring no modifiers. Only those remaining 15% will require a switch to refer to different arguments by semantic role.

Q: Do many commands introduce custom noun types?

A: 147 different noun types (lumping anonymous inline objects as one type) were detected. The vast majority of all takes (direct object) arguments were of type noun_arb_text, although many modifiers arguments used custom noun types. The other standard (built-in) noun types are well represented as well, with noun_type_language coming in at second place. Here’s a chart with all the noun types which had more than one use.

Q: Are commands with modifiers using natural-language delimiters?

A: Most of the modifiers detected were English prepositions such as “from”, “to”, “as”, “with”, but other words were also seen such as “title”, “type”, “username”, and “message” and even a handful of commands with symbols such as “@”, “>”, or “#”.

Related posts:

1. Writing commands with semantic roles
2. Ubiquity Parser: The Next Generation Demo
3. Ubiquity in Firefox: Focus on Japanese

Related posts brought to you by Yet Another Related Posts Plugin.

Today I’ll examine highlight some disparate typological data to help us understand these questions: where’s the verb? and what does the verb look like? There are broadly three different verb forms taken in commands in different languages:¹

1. the infinitive,
2. subjunctive mood, or
3. a special verb form such as imperative, participial, or conjunctive (such as Japanese て form)

Let’s give an example of each:

Infinitive (English):²

1

Hit me!

Subjunctive mood (Modern Greek): “Eat it all!”

2
3

Na   to fas olo!
SUBJ it eat all

Imperative form (French): “Eat it!”

4
5

Mange   -le!
eat.IMP it

It’s important to note that some languages have multiple forms available for the same command. For example:

Dutch: three ways to say “watch out!” with the same verb

1. Infinitive: Oppassen!
2. Imperative: Pas op!
3. Participial: Opgepast!

Similarly, I received a great comment by PhiliKON on German and associated data by Robert Kaiser on my blog post yesterday:

German: “search hello with google”

1. Infinitive: hello mit google suchen
2. Imperative: suche hello mit google

In addition, German and Dutch are interesting as they are verb second (V2) languages, so the verb may surface at the beginning or the end of the sentence, depending on the form.

The new parser design (which you can demo) assumes for simplicity that the verb should be found at the beginning or the end of the input, which is consistent with the data I’ve seen (modulo clitics). Multiple verb forms could be accounted for by supporting “synonyms” of the verbs.

What are the different ways verbs are expressed in commands in your language? Is the verb always found at the beginning or the end of the sentence? Is it ever somewhere in the middle?

Related posts:

1. Ubiquity i18n: questions to ask
2. Ubiquity Parser: The Next Generation Demo
3. Contribute: how your language identifies its arguments

Related posts brought to you by Yet Another Related Posts Plugin.

A week or two ago while visiting California, Jono and I had a productive charrette, resulting in a new architecture proposal for the Ubiquity parser, as laid out in Ubiquity Parser: The Next Generation. The new architecture is designed to support (1) the use of overlord verbs, (2) writing verbs by semantic roles, and (3) better suggestions for verb-final languages and other argument-first contexts. I’m happy to say that I’ve spent some time putting a proof-of-concept together.

I’ve implemented the basic algorithm of this parser for left-branching languages (like English) and also implemented some fake English verbs, noun types, and semantic roles. This demo should give you a basic sense of how this parser will attempt to identify different types of arguments and check their noun types even without clearly knowing the verb. This should make the suggestion ranking much smarter, particularly for verb-final contexts. (For a good example, try from Tokyo to San Francisco.)

➔ Check out the Ubiquity next-gen parser demo

Clicking on the environment info will give you some information on the specific verbs, noun types, and roles implemented. You can also scroll through the current parse section to see the step by step derivation of how the suggested parses were constructed.

I’ll be flying about 15 hours in the next hour as I make my way back to Japan… hopefully I’ll make some more progress on the plane! I’ll look forward to your comments! For those of you interested in checking out the code yourself, you can find it on BitBucket.

Related posts:

1. Writing commands with semantic roles
2. Ubiquity in Firefox: Focus on Japanese
3. User-Aided Disambiguation: a demo

Related posts brought to you by Yet Another Related Posts Plugin.

How Ubiquity identifies its arguments

Currently when writing a command in Ubiquity you must specify two properties for each argument: a modifier (the appropriate adposition—the direct object excluded) and the noun type. Here are some quick examples from the standard commands:

email:

• direct object (noun_arb_text)
• to (noun_type_contact)

translate:

• direct object (noun_arb_text)
• to (noun_type_language)
• from (noun_type_language)

This way of specifying arguments has a few shortcomings. First of all, it requires you to identify each type of argument by unique adposition, which does not support languages with case marking nor languages with sets of synonymous adpositions (e.g. French {à la, au, aux}). Second, as we saw in how your language identifies its arguments some languages don’t mark semantic roles on the arguments at all and the current system of specifying arguments is completely incompatible with these languages. Third, the current specification requires command authors to make localized versions of their commands, specifying the language-appropriate modifiers.

In a perfect world the last issue could be solved (at least for languages which mark semantic roles with adpositions) by a mapping of English prepositions to the target language adpositions. Indeed, for some adpositions in some languages this may be possible:

However, some English prepositions do not cleanly map to a particular adpositions. Take, for example, English “with.” This “with” may map to different markings in Chinese and Japanese depending on the sentence:

Note, however, that which set of markings “with” maps to is predictable, as there is a salient semantic difference. The first “with” could be referred to as together-with while the second is a using-with. With this distinction, we can easily predict which paradigm the “with” in “search with Google” should use, because these two “with” arguments represent two different semantic roles.

A proposal: identifying arguments by semantic role¹

Suppose commands could specify their arguments by referring to these semantic roles in lieu of adpositions as they currently do. This way, we would be able to automatically map commands into different languages. For example, you could write a new command called move with the following argument structure:

move:

• role_object (noun_arb_text)
• role_goal (noun_type_geolocation)
• role_source (noun_type_geolocation)

The English mapping of ” => role_object, ‘to’ => role_goal, ‘from’ => role_source could be used to parse the command

move truck from Tokyo to Paris

In addition, with the Japanese mapping of ‘が’ => role_object, ‘に’ => role_goal, ‘から’ => role_source, you could immediately use the command in Japanese as well:

東京からパリにトラックをmoveして

In essence, this proposal would let command authors get their commands localized for free, as long as they stick to a predefined set of semantic roles. For more complex commands and legacy commands, of course, commands could optionally specify particular English modifiers, but then Ubiquity would simply not attempt to localize those commands.

In addition, each language specific parser would determine how to identify its arguments. This would allow languages with case marking or no role marking on arguments at all to handle their own mapping of arguments to semantic roles and still use shared commands. Even parsers such as English would benefit by letting the parser deal with synonymous prepositions and possibly even argument structure alternations (such as English ditransitive alternations).

As a starting point, we could use argument types based on the list of semantic roles given in Fillmore (1971):

• Object: the entity that moves or changes or whose position or existence is in consideration
• Result: the entity that comes into existence as a result of the action
• Instrument: the stimulus or immediate physical cause of an event
• Source: the place from which something moves
• Goal: the place to which something moves
• Experiencer: the entity which receives or accepts or experiences or undergoes the effect of an action …

Comments welcome!

As command authors and Ubiquity users, how do you feel about this proposal? How might this affect, simplify, or complicate the localization of Ubiquity into your language? Thank you in advance! ^^

Related posts:

1. Three ways to argue over arguments
2. Contribute: how your language identifies its arguments
3. Ubiquity in Firefox: Focus on Japanese

Related posts brought to you by Yet Another Related Posts Plugin.

If you’ve ever subscribed to a new Ubiquity command before, you know the red screen of doom. Ubiquity currently takes users to this page every time they wish to subscribe to a new command. The current design is meant to encourage users to be aware of the possible security implications of enabling and executing a command, to avoid getting a trojan horse.

The current screen, however, does not make subscribing to commands foolproof. I personally know I’ve subscribed to a number of commands without reading through the code, defeating the purpose of the anti-trojan horse display. Moreover, the page doesn’t give you any information on how you can use this new command. Especially given the inherent limited discoverability of a natural language interface, taking a moment to help the user actually learn the command becomes key.

Today I did a quick mockup of what a friendlier command feed subscription page might look like. Take a look at this screenshot with some of the features marked:

You can also check out the page itself. If you’d like to visualize it without the “trust” warning, you can also view the trusted version.

This mockup here is but a first iteration. What do you think about this subscription page? What is missing? What should be changed?

Related posts:

1. How natural should a natural interface be?
2. Ubiquity in Firefox: Focus on Japanese
3. selecting from Ubiquity

Related posts brought to you by Yet Another Related Posts Plugin.

今後多様な言語に対応したFirefox内のUbiquityを検討していきますが、その中でも今日は日本語をとりあげます。後日日本語で同じ内容を投稿するつもりです。^^ 日本語でのコメントも大歓迎です！

What commands look like in Japanese

Japanese is not only just a verb-final language but it is strongly head-final, meaning it has postpositions instead of prepositions, direct objects come before verbs, and adjectives precede nouns. In terms of how it identifies its arguments, every argument has a postposition/case marker (called a particle in the Japanese literature) which marks its role in the sentence.

A couple common particles we’ll look at in this example include -を (-o) which marks the direct object (accusative case, you might say) and -に (-ni) which acts like English “to” (dative case). The example sentence we’ll look at today is:

(Note: ʔ is a glottal stop. ACC=accusative, DAT=dative, and IMP=imperative form.)

That final ください is often dropped in very casual speech and, as it adds no new information, we’ll assume today that the user will not enter it. Finally, Japanese doesn’t use spaces in their orthography, so the actual input would be “ケーキをブレアに送って”.

Mockup 1: Particle identification

One of the major hurdles in working with Japanese is that there are no spaces between the words. The natural first step is to split the sentence up into words, but this is a very difficult problem in NLP which big name research groups actively work on.

Fortunately, however, in “Solving the ‘It’ Problem” Aza suggests that, when we encounter ambiguity in our input, we can go ask the user. Great minds think alike, and computer scientist Jean E. Sammet suggested the same idea way back in 1953:

Using English [or any other natural language] definitely involves the requirement for the computer (or more accurately its programming system) to query the user about any possible ambiguity.

Parsing a sentence into words, in the limited context of Ubiquity, is really about identifying the particles which mark the end of each argument. Here’s a mockup of an application of the Sammet-Raskin Method to this problem:

Pros: This completely takes care of the word-breaking problem, with minimal arbitration from the user. The parser knows exactly what arguments it’s dealing with and the visual feedback means the user won’t be surprised by the parse.

Cons: Most of the particles/postpositions we’d have to deal with are a single character, so they may show up pretty often within words, in which case it would be quite annoying to have to press escape after each one.

An even smarter system, when wanting to mark a character as a particle, would first check to see that the argument (before the particle) is a valid argument type for that particle. If the check fails, it doesn’t have to bother with suggesting that character as a particle. This may cut down on the false positives.

Smart suggestions: what works, what doesn’t

One of the key suggestions in Aza’s mockups include a way to choose the prepositions while entering your arguments, based on the current verb.

For example, here, the translate command accepts a direct object, a to-object, and a from-object, so little to and from markers magically show up on the right side, making the appropriate prepositions (and by extension the appropriate arguments) discoverable. I think this line of thinking is a really good one, at least for English.

In a verb-final language, however, you enter the arguments first and then the verb, making this strategy of suggesting appropriate arguments impossible. Note that in the user-contributed spreadsheet of how languages identify their arguments we see that about a quarter of the languages we looked at are verb-final—that is, with Subject-Object-Verb canonical word order.

Instead of seeing this as a disadvantage, however, let’s see what verb-final order allows us to do.

Mockup 2: A different kind of suggestion

Not all verbs allow for every different kind of particle. For example, it doesn’t make sense to have a -に (-ni, “to” or dative) argument for a verb like 検索して (kensaku-shite, “search for”). In English we used this to suggest different types of arguments given a specific verb. In a verb-final language, we could do this backwards.

Pros: This makes verbs highly discoverable, given a certain argument structure. For example, if you enter a few arguments, like a direct object, a “to” argument, and a “from” argument, it’ll suggest verbs that will do something to an object from somewhere to somewhere else. This way, you can easily try out verbs you didn’t even know existed. It’ll only give you verbs appropriate for your arguments, reducing the chance of writing a an infelicitous command.

Cons: Without knowing what kinds of actions are available, it may be difficult to know what kinds of arguments to enter in the first place. If you have a specific verb or service you want to use it may be counterintuitive or downright tricky to start by guessing the right set of arguments.

In addition, from a technical point of view, this requires much of the prediction algorithms in English Ubiquity to run backwards. Ideally, there would be a closed (predetermined) class of particles and a predefined set of noun types. Verbs would not be able to define their own modifiers and noun classes as easily or freely as they can now.

Conclusion

The properties and challenges of Japanese grammar require that we not try to outright copy the English behavior but to think about what really makes sense in that language and that may be an important lesson as we move toward designing a localizable Ubiquity. Please post your questions and criticisms of this design or post your own mockups!

Related posts:

1. Three ways to argue over arguments
2. Contribute: how your language identifies its arguments
3. How natural should a natural interface be?

Related posts brought to you by Yet Another Related Posts Plugin.

When we execute a command in Ubiquity, in very simple terms, we’re hoping to do something (a verb) to some arguments (the nouns). Every sentence in every language uses some method to encode which arguments correspond to which roles of the verb. Here are a couple examples:

1
2

He sees Mary.
彼が Maryを 見る。 (Kare-ga Mary-o miru.)

As speakers of English, you can read sentence (1) above and know exactly who is doing the seeing and who is being seen and speakers of Japanese can get the same information from (2). How do different languages code for arguments in different roles? There are, broadly speaking, three different ways:

We’ll take a brief look today at these three different strategies, all of which a localizeable natural language interface will surely encounter.

Word order

In many languages, the position of the arguments relative to one another and to the verb determine the roles which each argument will play. Mandarin Chinese is a good example of such a language:

3
4

他 喜欢 Mary (Ta xihuan Mary)
Mary 喜欢 他 (Mary xihuan ta)

Here, sentence (3) says “he likes Mary” while sentence (4) says “Mary likes him”. Simply reversing the positions of “he/him” and “Mary” we’re able to flip the roles that they fill in the sentence: that of the person who does the liking and the person who is being liked. Now take a look at sentence (5) which means “John says ‘hello’ to Mary.”

5

John 告诉 Mary "你 好" (John gaosu Mary "ni hao")

We note here that, while in English we used a different strategy of marking one argument (we marked the “hello” argument with “to”), Chinese doesn’t mark either of the arguments. There is, however, a clearly defined order to the arguments, which you might encode this way:

say [who you're speaking to] [what you're saying]

If you swap the order of the two objects in this sentence, it becomes ungrammatical. (Note: the asterisk * here means the sentence is ungrammatical.)

5

* John 告诉 "你 好" Mary (John gaosu "ni hao" Mary)

Here, the word order dictates that “你好” must be “who you’re speaking to” and “Mary” must be “what you’re saying,” but that doesn’t make sense, so the sentence is ungrammatical.

Marking the arguments

Another possible strategy is to mark each argument (or some of the arguments) so that each argument’s role is clear. In many languages this is done with case marking. Take for example this Ancient Greek sentence with its English gloss on line (6). Here, NOM refers to nominative case and ACC refers to accusative case.¹

5
6

ho  didaskal-os  paideuei to  paidi-on  (SVO)
the teacher -NOM teaches  the boy  -ACC

This sentence means “the teacher instructs the boy.” While sentence (5) is in Subject-Verb-Object order, any of the six possible orderings of {subject, verb, object} are also grammatical and mean the same thing:²

7
8
9
10
11

ho didaskalos to paidion paideuei (SOV)
paideuei ho didaskalos to paidion (VSO)
paideuei to paidion ho didaskalos (VOS)
to paidion ho didaskalos paideuei (OSV)
to paidion paideuei ho didaskalos (OVS)

Many languages also use adpositions (prepositions and/or postpositions) to further clarify the role of an argument in addition to case (like English does) or in lieu of case marking altogether. The idea is the same, though: you want to clarify the roles of the arguments so you morphologically mark the arguments with their roles.

Marking the verb

Many languages mark the verb with some information about the argument in a certain role, so that we can properly identify the argument’s roles. This kind of phenomenon is called agreement.

The most common type of verbal agreement is subject agreement, where the verb is marked by a specific form depending on some features of the subject. Anyone who’s taken French 101 will recognize this verb conjugation paradigm:

With this paradigm, if you hear or see “suis” in a French sentence, you immediately know that “je” (I) must be the subject and if you see “sommes,” “nous” (we) is the subject, etc. Standard Average European languages tend to exhibit this sort of subject-verb agreement.

Features of the subject position aren’t the only thing that can be marked on the verb, though. Hungarian, for example, has a type of object agreement. Specifically, the verb marks whether the object is definite or not (in linguistics lingo, “the verb agrees with the object’s definiteness feature”).

12
13
14
15

John lát  egy almát.
John sees an  apple
John látja az  almát.
John sees  the apple

Notice that in sentence (12) (glossed in (13)) the verb for “see” is realized as “lát,” while in (14) it’s “látja.” A speaker can use that agreement to see whether the object is definite or not and thus limit the possible object arguments out of all the nouns in the sentence.

All of the above

Most languages do not use only one of these strategies, but a combination of them. English is a very good example. In a sentence like (12) below the main coding of grammatical roles seems to be word order alone. By reversing the word order into (13), we can effectively swap the argument’s roles.

12
13

John likes Mary.
Mary likes John.

However, this doesn’t work with pronominal arguments. Swapping the arguments in (14) yields (15) which is ungrammatical due to the case marking on the pronouns.

14
15

He likes her.
* Her likes he.

In addition, the verb in English must agree with the subject’s number (singular or plural):

16
17
18

John likes them.
* They likes John.
They like John.

In this way, English exhibits all three strategies: word order, case marking, and agreement, although often only word order is actively used to disambiguate the roles of arguments.

Question: What strategies are used by your language to mark the roles of different arguments?

Related posts:

1. 回収 vs. 収集 and Better Word Meanings Through Usage
2. Testing Google’s Language Detection
3. Gaba, Shame On You

Related posts brought to you by Yet Another Related Posts Plugin.