I recently have begun giving serious thought to what command chaining might look like in Ubiquity and the various considerations which must be made to make it happen. The “command chaining,” or “piping,” described here always involves (at least) two verbs acting sequentially on a passed target—that is, the first command performs some action or lookup and the second command acts on the first command’s output.

A few days ago I penned some initial technical considerations regarding command chaining. In this post I’ll be point out some linguistic considerations involved in supporting a natural syntax for chaining.

Simple syntaxes: sequential vs embedding strategies

When it comes to creating a natural language interface, there’s always a decision to make between requiring a certain kind of input, or working a little harder to understand the user’s natural input. From an implementation point of view, adopting certain programmatic conventions is of course simpler and to this end, there have been a couple different “unnatural” command chaining syntaxes suggested. While these both go against Ubiquity’s basic tenet of natural syntax — that is, to not introduce rules which contradict the user’s natural language — which gives Ubiquity its strengths of usability and memorability, I’ll entertain them here as they illustrate two different structural relationships that we will want to consider.

The first suggestion is to adopt the shell pipe (|), which would lead to input such as

1

translate hello to Spanish | email to Jono

While this itself is pretty unnatural unless you speak shell, note that this syntax is similar to the more natural “, and” syntax, yielding translate hello to Spanish, and email to Jono, which we will consider below. I’ll refer to this strategy as the sequential strategy.

Another very interesting proposal by Alex Fritze is to embed each subordinate computation into an argument position, marked by parentheses. This could also be parsed relatively straightforwardly by writing a noun type which first checks for parentheses and then runs the content of the argument through another ParseQuery.

2

email (translate hello to Spanish) to Jono

I’ll refer to this pattern as the embedding strategy.

Sequential and embedding strategies in natural language

What’s interesting about the two proposals above is that both strategies are seen in natural language. The sequential strategy could correspond to the following linguistic phenomena:

1. coordination: a non-hierarchical joining of two or more clauses, often marked by a conjunction. Here’s an example from English:
   • “[I made a sandwich] and [you will eat it]” where [] represent clause boundaries. Here, “and” is the conjunction.
2. serial verb and converb constructions: a joining of multiple verbs or verb phrases within a single clause, with shared subject and tense/aspect values, with no particular conjugation or delimiter between them. Such constructions are common in many African and east Asian languages. Here are two examples:¹
   

• A converbal construction in Japanese:
  

  3 4 5

  僕は サンドイッチを 作って 食べる boku-wa sandiʔchi-o tsuku-ʔte tabe-ru I-TOP sandwich-ACC make-CON eat

  
  “I (will) make a sandwich and eat [it].” (Here, `TOP` = topic, `ACC` = accusative, `CON` = converbal ending)[^3]
• A serial verb construction in Mandarin Chinese:
  

  6 7 8

  我 作 三明治 吃 wǒ zùo sānmíngzhì chī I make sandwich eat

  
  “I (will) make a sandwich and eat [it]” or “I (will) make a sandwich [in order to] eat [it].”

Note that in both the converb and serial verb construction, the second verb (eat) takes shares its object (sandwich) with the first verb and there is no need for a pronoun such as “it” to introduce that argument as it is with coordination, above.²

The embedding strategy is observed in natural language as well, in the form of the following phenomena:

1. embedded clauses: a sentence is itself the argument of another verb. Example:
   

• “John says [he likes sandwiches].”

Embedded clauses, however, clearly have no relation to command chaining and does not require our attention. 2. relative clauses: a partial sentence³ is attached to a noun in order to describe it or distinguish it from other possible referents. Example:

• “You ate the sandwich that I made” where “sandwich” is called the “head” of the relative clause, and “I made” is what I here call the “partial sentence” (see footnote). The “relative clause” is used here to distinguish “the sandwich that I made” from other sandwiches.

The natural syntax of chaining

So which strategy is used in complex natural language commands: the sequential strategy or the embedding strategy? Both the sequential strategy and embedding strategy can be involved with commands:

9
10

[Make a sandwich] and [eat it]!
Eat (the sandwich that I made)!

These two commands do not mean the same thing, though, and only (9) is the kind of command we would want to give Ubiquity. The problem with relative clauses, as in (10), is that it presupposes the existence of the sandwich in the context. If we both know you just made a sandwich, saying (10) is perfectly appropriate, but out of the blue it doesn’t make sense. For this reason, only the sequential strategy is used in the natural syntax of chaining.

Parsing the sequential strategy

In natural language, unlike the initial simple proposals laid out above, there is often no clear delimiter marking the boundary between the two parts in a sequential relation (e.g. examples (3) and (6) above, particularly given that neither Japanese and Chinese normally break words with spaces). How would we parse a sequential string of commands?

Let’s assume for our purposes here that we can identify find all verbs within the input string.⁴ Parsing a sequential strategy string is not particularly difficult if we can also assume that the verb in any particular language is either always verb-initial or always verb-final. Let’s look at both cases:

• Always verb-initial: Mandarin Chinese:
  

11
12
13

翻譯       hello 到  西班牙語   送    給  Juanito
fānyì     hello dào xībānyáyǔ sòng gěi Juanito
translate hello to  Spanish   send to  Juanito

“Translate hello to Spanish [and] send [it] to Juanito” 1. find every possible verb:
翻譯hello到西班牙語送給Juanito 2. as every verb marks the beginning of a sentence, we now have our two commands: “翻譯hello到西班牙語” (translate hello to Spanish) and “送給Juanito” (send to Juanito). * Always verb-final: Japanese

14
15
16

helloを スペイン語に 訳して Juanitoに 送って
hello-o supeingo-ni yakus-ite Juanito-ni oku-ʔte
hello-ACC Spanish-DAT translate-CON Juanito-DAT send-CON

“Translate hello to Spanish [and] send [it] to Juanito” 1. find every possible verb:
helloをスペイン語に訳してJuanitoに送って 2. as every verb marks the end of a sentence, we now have our two commands: “helloをスペイン語に訳して” (translate hello to Spanish) and “Juanitoに送って” (send to Juanito).

For languages where there is a clear conjunction between the two commands, such as English “and”, we can also use that conjunction as a delimiter as well. We then simply execute the first command and then execute the second with the first command’s output in its interpolation context. This way the output of the first command will be picked up both by an overt pronoun such as “it” in the second command and without it, such as in the Chinese and Japanese examples above.⁵

The only potential problem with this approach is in the case of languages where some commands are verb-initial while others are verb-final. I note that such languages do exist in a previous blog post, Where’s The Verb. In these languages, commands can be expressed by more than one verb form (such as infinitive, imperative, subjunctive, etc.) and some of those verb forms are sentence-initial while others are sentence-final. Here’s one such example from German:

“search hello with google” (German) 1. Infinitive: hello mit google suchen 2. Imperative: suche hello mit google

Here the verb for “search” is “suchen” (infinitive) or “suche” (imperative). I know that this same type of phenomena occurs in other Germanic languages such as Dutch with infinitive and imperative and also other languages such as Modern Greek with infinitive and subjunctive forms. If you are a speaker of one of these lanuages (German, Dutch, Greek, etc.) I would love to know whether you can chain verb-final and verb-initial commands together.

Conclusion

In this blog post I examined command chaining in natural language, focusing on data from English, Mandarin Chinese, and Japanese, which exhibit three linguistically different approaches to chaining. What we found is that the sequential strategy—that of listing the commands one by one, in order of execution—is what is used in natural languages, rather than any sort of embedding. This fact, combined with the fact that our parser can recognize every available verb, offers a simple approach to doing a naive parse of natural command chains in most languages, even without explicit delimiters.

In a final installation of this series on “exploring command chaining,” I hope to consider how the Ubiquity interface itself could present command chains and aid in its input.

Related posts:

1. Exploring Command Chaining in Ubiquity: Part 1
2. Command Chaining with Oni?
3. Ubiquity in Italian

Related posts brought to you by Yet Another Related Posts Plugin.

There is no myriad of buttons and fields to choose from. All the user has to do is directly type in what he wants.

Here are a couple other choice quotes which will ring true for the Ubiquity users and internationalization folks in the audience:

For this to work without driving the user mad, the natural language parser has to be near perfect. The last thing I want is for this to come out smelling like AppleScript.

Problems
This design isn’t perfect as it has two glaring problems. One is that the user has no easy way of discovering how complex the recurrence rules can be. This isn’t such a huge problem, but a way to solve this is to include a help button to show example rules or to include an accompanying iCal style UI to let the user setup the recurrence rule in a more typical fashion. I didn’t include these in the initial implementation though because I wanted to see how users would react to this kind of UI.
Another problem is localization. Even if I write parsers for a few more popular languages, it won’t accommodate the rest of the users in the world. Again, the solution is an accompanying traditional UI, but for now, I’m leaving it the way it is until I get some feedback.

There’s a trend in the wind, my friends: the incorporation of near-natural language for more humane interfaces.

Related posts:

1. How natural should a natural interface be?
2. Friendlier command feed subscription
3. User-Aided Disambiguation: a demo

Related posts brought to you by Yet Another Related Posts Plugin.

The inspiration for this idea came from Aza’s blog post “Solving the ‘it’ problem” which advocates for this type of quick feedback to the user in cases of ambiguity. Such a method would help both the user better understand what is being interpreted by the system, as well as offer an opportunity for the user to correct improper parses. I just tried mocking up such an input box using jQuery.

➔ Try the User-Aided Disambiguation Demo

If you have any bugfixes to submit or want to play around with your own copy, the demo code is up on BitBucket. ^^ Let me know what you think!

Related posts:

1. How natural should a natural interface be?
2. Ubiquity in Firefox: Focus on Japanese
3. Friendlier command feed subscription

Related posts brought to you by Yet Another Related Posts Plugin.

Every day, more users are trying out Ubiquity, the Mozilla Labs experiment that lets users accomplish common Internet tasks faster through a natural language interface. As we live more and more of our lives on the web, there is a huge appeal to—and need for—a faster way to access and mashup our information.

But what exactly do we mean by a “natural language interface”? Is it just another programming language with lots of English keywords? Should the final goal be a computer that understands everything we tell it?

As we think about the future directions and possibilities of Ubiquity, we need to go back to our roots and understand the project’s motivations. With that in mind, here are some initial thoughts on the advantages of a natural language interface. The ultimate goal here is to refine the notion of natural language interface and to come up with a set of principles that we can follow in pushing Ubiquity further, into other languages and beyond.

Why language?

In his 2008 article in interactions, Aza describes a clear need for modern UI to move beyond monolithic do-everything apps into efficient, granular commands that can be connected to accomplish tasks. Hierarchical menus with an application’s every possible function are great for discoverability, but slow and inefficient as they grow. Aza advocates for the use of a familiar subset of natural language to this end. In his own words,

Words can capture abstractions that pictures cannot because language has an immense amount of descriptive and differentiating power. Abstract thoughts are exactly represented by the words that give them names. It is this power that comes to the rescue in specifying functionality.

In other words, language gives us the descriptive power to succinctly and creatively express our will, far faster than a series of menus, and with more freedom than a series of shortcuts or gestures. In addition, by tapping into the lexicon of our every day language, we make a direct attack on the learnability problem.¹

The natural syntax test

The ability to string different commands together is not a novel one—indeed, this is what more traditional command lines and programming languages offer. However, these technologies present a huge barrier to the layperson, even for languages with many keywords from English or English-like syntax.

Programming languages can be such teases in this way. Often the first bits of code in a language look remarkably similar to natural language (Python):

print "Hello World"

…but the young coder is quickly disappointed:

print map(lambda x: x*2, [1,2,3])

AppleScript is a language which tries to take this idea further and, indeed, sometimes AppleScript code constitutes readable English.

print pages 1 thru 5 of document 2

Dig a little deeper, though, and AppleScript also fails the “natural syntax” test. In fact, it can be argued that a language that looks like a natural language but differs in some important details can be even more difficult to use than one that is completely novel. Bill Cook, one of the original developers of AppleScript, makes this point in his history of AppleScript: “in hindsight, it is not clear whether it is easier for novice users to work with a scripting language that resembles natural language, with all its special cases and idiosyncrasies.”

If the interface’s syntax is too restrictive or, worse, conflicts with a user’s natural intuitions about their natural language, it immediately fails to be “natural”, no matter how similar the keywords or grammar is.²

Towards a natural (and forgiving) syntax

Aza similarly laments the relegation of text-based interfaces to the higher echelons of geekdom in his 2008 paper: “if commands were memorable, and their syntax forgiving, perhaps we wouldn’t be so scared to reconsider these interface paradigms.”

The key word “forgiving” above (emphasis mine) is two-ways ambiguous, both of which we want a natural language interface to be:

1. Forgiving as in “not difficult to learn and remember”: the syntax must be easy and natural for the user, encouraging experimentation and intuitive application;
2. Forgiving as in “not correcting or prescriptive”: the system should try its darndest to accept the user’s input, even if it’s not the most “well-formed.”

From an implementation point of view, (2) above can also be an advantage. There are many grammatical restrictions in natural language which, as long as the command is unambiguous, Ubiquity need not enforce on the user. Take, for example, the two statements:

1
2

print two copy
print two copies

I feel that Ubiquity should execute both of these statements with equal ease. The numeral “two” makes the user’s intent very clear, even though the plural of “copy” should indeed be “copies.” It need not be the job of the interface to decide whether a sentence is “correct English.” By assuming that the user is trying to communicate a valid and possible task, rather than throwing up an error, the system will be more flexible and more forgiving in the inevitable case of human error. The ultimate goal should be to help the user accomplish their task.

Conclusion

By developing a language interface which truly feels natural to the user, we can successfully bring the power of text-based interfaces to the masses. I feel the key to this “natural-ness” is a less restrictive and in fact forgiving syntax. While this goal akin to natural language programming may be daunting from an implementation angle, and it may indeed prove impossible, as long as the goal is to execute simple imperative commands, the scope of the target syntactic structures is limited.

Ubiquity as it stands is many different things for many people. The natural language guidelines above may feel too restrictive to many current developers for whom Ubiquity is simply a convenient new way to extend Firefox.³ This discussion also seems orthogonal to the mouse-based Ubiquity experiments. As users and developers, how do you feel about the potential benefits and downsides of these natural syntax guidelines? In the coming days I’ll look at some concrete examples of what this “forgiving” syntax would demand of Ubiquity.

Related posts:

1. Using Templates with YARPP 3
2. Testing Google’s Language Detection
3. 回収 vs. 収集 and Better Word Meanings Through Usage

Related posts brought to you by Yet Another Related Posts Plugin.