As we look to move forward with incorporating the next generation parser into Ubiquity proper, it thus becomes important to take a look at the current command ecosystem to see how possibly disruptive this move will be. To this end last night I wrote a quick perl script to scrape the commands cached on the herd and get some quantitative answers to my questions.

(1577 different verbs were analyzed. None of these computations below are weighted by feed popularity.)

Q: Are there a lot of commands which use more than one argument?

A: The vast majority (>85%) of commands take one or no arguments, requiring no modifiers. Only those remaining 15% will require a switch to refer to different arguments by semantic role.

Q: Do many commands introduce custom noun types?

A: 147 different noun types (lumping anonymous inline objects as one type) were detected. The vast majority of all takes (direct object) arguments were of type noun_arb_text, although many modifiers arguments used custom noun types. The other standard (built-in) noun types are well represented as well, with noun_type_language coming in at second place. Here’s a chart with all the noun types which had more than one use.

Q: Are commands with modifiers using natural-language delimiters?

A: Most of the modifiers detected were English prepositions such as “from”, “to”, “as”, “with”, but other words were also seen such as “title”, “type”, “username”, and “message” and even a handful of commands with symbols such as “@”, “>”, or “#”.

Related posts:

1. Writing commands with semantic roles
2. Ubiquity Parser: The Next Generation Demo
3. Ubiquity in Firefox: Focus on Japanese

Related posts brought to you by Yet Another Related Posts Plugin.

In the future we ideally could build a web-based system to collect these “utterances.” We could … generate parser parameters based on those sentences. That would essentially reduce the parser-construction process to a more run-of-the-mill string translation process.

If we build this type of “command-bank” of common Ubiquity input translated into various languages, we could build a tool to learn various features of each language and generate each parser, essentially learning the language based on data. Today I’ll elaborate on how I believe this could be possible, by analogy to another language learning device: the human.

Step 1: learning words

How does a human learn language? Without getting into any details or theory, we can say that the input for a language learner is always a combination of linguistic input and a referent. In the case of a child, this could be a pairing of linguistic input with real world stimulus:

input	referent
“taiyaki!”	by makitani via creative commons.
“cat!”	by victoriachan via creative commons.

The human child will hear “cat” while looking at the cat and, with time and repetition, learn that that thing is called a “cat,” and some other thing is called “taiyaki.”

Similarly, we could take single-verb data points from our command-bank to match new words with a know referent—in this case, the base English string. Here’s an example from Jan’s comment on yesterday’s sample survey.

Step 2: deduction

Now suppose we know some single words like “taiyaki” and “cat.” Consider the two situations. Given the first sentence and referent “mitcho’s eating a taiyaki,” the child could intuit the appropriate linguistic representation for the latter situation.

input	referent
“mitcho’s eating a taiyaki!”
???

The process is simple. First note that there is only one variable changed between the two situations: the taiyaki has been replaced by a cat head. You can then construct the correct utterance by analogy, replacing “taiyaki” with “cat,” yielding “mitcho’s eating a cat!”¹

Similarly, we could build a tool to analyze the data in a translated command-bank to identify particular features of each language, generating at least basic parsers for each language. Such a task would require a number of minimal pairs in our data set—here’s one such example from yesterday’s survey (with Dutch data from Jan):

Parse {
  verb:      'search',
  arguments: {
    object:  ['HELLO'],
    service: 'Google'
  }
}

Parse {
  verb:      'search',
  arguments: {
    object:  ['this'],
    service: 'Google'
  }
}

A simple string analysis² would tell us that the text HELLO was replaced by dit in the latter Dutch sentence. Meanwhile, since the English reference sentence is chosen manually, we also know the appropriate parses for each of those sentences. An object difference operation would note that the object property was changed from a value of 'HELLO' to 'this'. We could then map dit to the English this. We’ve now learned one (of perhaps many) Dutch deictic pronouns (aka “magic words”).

Given an adequately universal but customizable parser design, we can then develop tests for various parameters by constructing appropriate minimal pairs in the base sentences and having them translated.³ As noted yesterday, such a system could reduce the laborious task of writing individual parsers to a task of string translation, which our community does exceedingly well. I’m eager to hear what others think of this approach. What concerns would you have for this approach? What potential benefits do you see?

Related posts:

1. Ubiquity i18n: questions to ask
2. Localizing Ubiquity: an open letter to linguists
3. Writing commands with semantic roles

Related posts brought to you by Yet Another Related Posts Plugin.

As a linguist, studying a language involves looking at instances of that language as data. To this end, we as Ubiquity internationalizers need to get at some examples of target utterances. Here’s an example survey which could be a good starting point for native speakers who want to contribute information on their language, based on Blair’s list of common Ubiquity verbs.

A survey for Ubiquity localization

Instructions

How would you express the following commands in your language? The words in CAPITAL LETTERS do not need to be translated. Feel free to give multiple possible answers for each command.

Try to express the same command rather than forcing a “literal translation”; for example, if there’s no “map” verb in your language, you could translate example (8) as lookup a map of PLACE. Please keep in mind that the addressee is a computer.

Basic word order / argument structure

1. search HELLO
2. search HELLO with google
3. translate HELLO from English to French
4. lookup the weather for PLACE
5. shop for SHOES with Amazon
6. email HELLO to Bill
7. email HELLO to ADDRESS
8. map PLACE
9. find HELLO
10. tab to HELLO or switch to HELLO tab

…

Pronominal/deictic arguments (aka “magic words”)

1. search this with google
2. translate this to French
3. bookmark this tab

…

How this data is used

Responses to these surveys would be used to identify certain salient features of the language, such as how the language codes for its arguments (for example using adpositions, case marking, or word order), whether the commands tend to be verb-inital or -final. Individual case markings, for example, can be identified by comparing minimal pairs—for example, by comparing item (1) and (2), we can learn how google in an instrumental role is marked, or by comparing example (2) and the “magic word” example (1), we can identify the appropriate “magic word” and determine whether the language uses any clitics or not.

Data collection

In the future we ideally could build a web-based system to collect these “utterances.” We could also use such a system to automatically test our parsers in different languages against the sentences in the command-bank, or ultimately even generate parser parameters based on those sentences. That would essentially reduce the parser-construction process to a more run-of-the-mill string translation process.

Related posts:

1. Contribute: how your language identifies its arguments
2. Writing commands with semantic roles
3. Ubiquity in Firefox: Focus on Japanese

Related posts brought to you by Yet Another Related Posts Plugin.

I gathered some data from the fantastic World Atlas of Language Structures to put together a survey of many of the languages on the Internet. For each of the languages, I got the canonical word order and whether the language marks the role of its argument on the verb and/or the arguments themselves.

As you can see, there are a number of data points that are still missing. Please contribute information on the languages you speak! You can edit the spreadsheet on Google Docs. Thanks!

Related posts:

1. Three ways to argue over arguments
2. Testing Google’s Language Detection
3. Setting Language Research to Music

Related posts brought to you by Yet Another Related Posts Plugin.

• 回収 may take things away from others when collecting while 収集 does not have that implication.
• Things that you 回収 may have been previously distributed by the actor themself while 収集 does not have that implication.¹

Not content with armchair theorizing, however, I decided to take advantage of one of the largest corpora in the world: Google.² To test my hypothesis, I chose two “objects of collection”, one you can take away (and often is distributed first) and one you can’t take away: アンケート (ankēto “survey,” from the French enquête) and 意見 (iken “opinion”). I then took the four resulting collocations³ on Google in quotes (“•”) and recorded how many hits there were.

A better way to organize this data is as follows:

This data clearly supports the hypothesis I laid out above: アンケート, which can be taken away from people and is often distributed first, occurs much more likely with 回収 than 収集. 意見, on the other hand, which crucially cannot be taken away when collected, occurs much more likely with 収集 than 回収.

While this one example doesn’t prove anything in and of itself, it does help clarify with data a nuance between two near synonyms. While my hypothesis was borne out here, native speaker intuitions on word nuances and distinctions can be unreliable.⁴ This type of quick test can be very helpful for language learners and instructors alike.

Languages very often have words which vary in very subtle ways. Just this Tuesday I went to a Tokyo Language Exchange Meetup, a great meetup which brought together various language learners and enthusiasts. A hot topic that night was words with very similar meanings—near synonyms. A few English learners were lamenting sets of words like {see, view, watch} and how difficult they are to learn. I myself have had the same experience studying Mandarin.

I noted that these difficulties in offering contrasting definitions often are due to the fact that word meanings are not just “what the word points to” but also the implication of “what it relates to”.⁵ For example, “unborn baby” and “fetus” may point to the same thing, but are used in different contexts, in contrast to different other terms, for differing effect. Similarly “Death Tax” and “Estate Tax.” “Kneel” and “genuflect.”⁶

The concept of word meanings being “what it points to” and “what it relates to” also helps explain why certain words are difficult to translate. Fillmore uses the Japanese example of ぬるい (nurui) which is the de facto translation of “lukewarm.” However, some Japanese speakers will only use ぬるい in contrast with “hot,” i.e., hot tea can become ぬるい over time but ice water does not become ぬるい. In contrast, English “lukewarm” can be used to describe things that are initially or prototypically hot or cold. “What the words point to” in this case is the same but “what it relates to” or, here, “what it contrasts with” is different, making it an imperfect (though very close) translation.

Every language has near synonyms which vary slightly in nuance but this nuance or “feeling” is borne out objectively in data. Looking at what words certain terms relate to in real usage is often the key to getting a richer understanding of vocabulary.

Related posts:

1. Testing Google’s Language Detection
2. Taipei find: a dictionary of Chinese-Japanese false cognates
3. The Most Beautiful Word

Related posts brought to you by Yet Another Related Posts Plugin.