The problem:

Weak pronouns in romance languages (as well as some other languages) have a special property where they cliticize to the verb, moving from its regular argument position to a position next to the verb. For example, in French, we have an imperative like (1) with gloss as (2):

1
2

Envoyez  le  lettre à  Pierre!
send.IMP the letter to Pierre

If we replace le lettre or à Pierre with a preposition (le, “it”, or lui, “to him”, respectively), those weak pronouns move next to the verb—in particular, (5) exemplifies the change in word order. Replacing both arguments with prepositions creates the stacked clitic form of (7).¹

3
4
5
6
7
8

Envoyez-la à  Pierre!
send   -it to Pierre
Envoyez-lui la  lettre!
send   -him the letter
Envoyez-le-lui!
send   -it-him

The fact that these weak pronouns are attached to the verb and lack separate delimiters mean that we will need a separate mechanism to parse these arguments: indeed, this functionality has been planned in Ubiquity Parser 2 as “step 3”. Here I’ll examine some data and discuss a strategy for the parsing of weak pronouns.

Weak pronouns in Ubiquity

In Ubiquity the only pronoun we currently deal with is the deictic object-role anaphor, like “it,” “this,” etc. in English.² In addition, as these weak pronoun clitics cannot by definition be embedded within a larger noun phrase, its referent would constitute the entire object argument. As such, it is most logical to place clitic handling before argument structure parsing and simply hand the argument parser the argument string without the clitic.

Marking the clitic

We can classify languages with cliticized weak pronouns into two cases based on their processing considerations: languages that overtly mark the clitic and those which do not.

Languages which delimit the clitic

Some languages such as French (see above) clearly mark the boundary between the verb and the clitic. It will be relatively easy to parse weak pronouns in such languages as we can simply insert a no-width space between the verb and the clitic. A list of clitics can then be designated in the parser (much like anaphora are now) and these weak pronouns can be interpreted as the selection (or “this”-referent).³

Portuguese: (from Cysouw 2003)

Come-o
eat -it

Catalan: (from Toni Hermoso Pulido)

Cerca-ho
search-it

Modern Greek: (from Rivero and Terzi 1995; I know, I know, Greek’s not a romance language, but it has weak pronoun clitics too… it’s all good.)

Modern Greek actually inserts a space between the verb and weak pronouns.

Diavase to
read   -it

Languages which do not delimit the clitic

Some languages do not insert any delimiter between the verb and the weak pronoun, essentially entering them as a single word (in the string sense, at least). These cases may be more difficult to parse, especially as there may be sound changes to the verb stem itself.

Italian: (first example from Kayne 1994)

Italian is a case where some verbs actually conjoin with the verb in imperatives, much like their prepositions which I noted yesterday have an elaborate system of portmanteau’ed forms.

Fallo
do-it
Mangialo
eat  -it

Spanish: (first example from Rivero and Terzi 1995, second from Toni Hermoso Pulido)

Spanish is the same way:

Léelo
read-it
Búscalo
search-it

Displaying the suggestion

The current Ubiquity handling of anaphora (aka “magic words”) involves a display of the selection (replacement) text in a stylized way. One problem with clitics may be how to visually present this replacement to the user.

For languages with a delimiter such as French we could simply present the selection as an object right after the verb, without the hyphen.

Things may be more complicated, however, in languages where the clitic is not delimited from the verb, or where the verb form itself has changed due to the attachment of the clitic.

Conclusion

In this blog post I’ve tried to lay out some of the weak pronoun phenomena relevant to Ubiquity with some ideas on how to implement its parsing. I believe parsing weak pronouns should be relatively straightforward in languages with delimiters—for those which do not have delimiters, some creativity may be required in how building regular expressions or rules to detect the clitics and in presenting these suggestions to the user.

Does your language have weak pronoun clitics? What do you think will be the challenges in trying to parse these arguments?

Related posts:

1. Solving a Romantic Problem: Portmanteau’ed Prepositions
2. Where’s The Verb?
3. This week on Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

In many romance languages, prepositions and articles often form portmanteau morphs, combining to form a single word.¹ Some examples include (French) à + le > au, de + le > du, (Catalan) a + el > al, de + les > dels, per + el > pel. Italian has a particularly productive system of portmanteau’ed prepositions and articles… I refer you to the contraction article on Wikipedia.

As I noted a couple weeks ago, however, some combinations do not form portmanteaus.²

French:

1. à + le > au
2. à + la > à la

The problem with this is that if we use both à and au as delimiters, we may end up passing the definite article to the verb as part of the argument in some cases, but not in other cases.

1. “à la table” = “to the table”
2. “au chat” = “to the cat”

The solution:

The solution is a new step in the Parser 2 process which normalizes the form of arguments. Each language’s parser can now optionally define a normalizeArgument() method which takes an argument and returns a list of normalized alternates. Normalized arguments are returned in the form of {prefix: '', newInput: '', suffix: ''}. For example, if you feed “la table” to the French normalizeArgument(), it ought to return

[{prefix: 'la ', newInput: 'table', suffix: ''}]

If there are no possible normalizations, normalizeArgument() should simply return []. Each alternative returned by normalizeArgument() is substituted into a copy of the possible parses just before nountype detection. The prefixes and suffixes are stored in the argument (as inactivePrefix and inactiveSuffix) so they can be incorporated into the suggestion display.

Here, for example, is how the inactive prefix “l’” is displayed in the parser demo. This way the user is told that the “l’” prefix is being ignored, and the nountype detection and verb action can act on the argument “English”.³ (In the future, of course, we could teach this nountype to accept the Catalan “anglès”.)

The easiest way to produce this output is to use the String.match() method. For example normalizeArgument() code, I refer you to the Catalan and French parser files.

I hope that this solution will help make Ubiquity with Parser 2 feel more natural for many romance languages.

Related posts:

1. Where’s The Verb?
2. Adding Your Language to Ubiquity Parser 2
3. Scoring and Ranking Suggestions

Related posts brought to you by Yet Another Related Posts Plugin.

You’ve seen the video. You speak another language. And you’re wondering, “how hard is it to add my language to Ubiquity with Parser 2?” The answer: not that hard. With a little bit of JavaScript and knowledge of and interest in your own language, you’ll be able to get at least rudimentary Ubiquity functionality in your language. Follow along in this step by step guide and please submit your (even incomplete) language files!

As Ubiquity Parser 2 evolves, there is a chance that this specification will change in the future. Keep abreast of such changes on the Ubiquity Planet and/or this blog (RSS).

Set up your environment

If you’re new to Ubiquity core development, you’ll want to first read the Ubiquity 0.1 Development Tutorial to learn how to get a live copy of the Ubiquity repository using Mercurial. Once you’ve set up your Firefox profile to use this development version, make sure to try changing the extensions.ubiquity.parserVersion value to 2 in about:config (as seen in this demo video) to verify that Parser 2 is working for you.

As you read along, you may find it beneficial to follow along in the languages currently included in Parser 2: English, Japanese, Portuguese, and Swedish (and the incomplete Chinese and French).

The structure of the language file

Each language in Parser 2 gets its own file which acts as a JavaScript module. You’ll need to look up the ISO 639-1 code for your language… Here we’ll use English (code en) as an example here and the JavaScript language file would then be called en.js and go in the /ubiquity/modules/parser/new/ directory of the repository.

Here is the basic template for a Ubiquity Parser 2 language file:

1
2
3
4
5
6
7
8
9
10
11
12

var EXPORTED_SYMBOLS = ["makeEnParser"];
 
if ((typeof window) == 'undefined') // kick it chrome style
  Components.utils.import("resource://ubiquity/modules/parser/new/parser.js");
 
function makeEnParser() {
  var en = new Parser('en');
 
...
 
  return en;
};

After lines 1-4 which set up the JavaScript module, everything else is wrapped in a factory function called makeLaParser (for Latin) or makeEnParser (for English, en) or makeFrParser (for French, fr), etc. This function initializes the new Parser object (line 7) with the appropriate language code, sets a bunch of parameters (elided above) and returns it. That’s it!

Now let’s walk through some of the parameters you must set to get your language working. For reference, the properties the language parser object is required to have are: branching, anaphora, and roles.

Identifying your branching parameter

  en.branching = 'right'; // or 'left'

One of the first things you’ll have to set for your parser is the branching parameter. Ubiquity Parser 2 uses the branching parameter to decide which direction to look for an argument after finding a delimiter or “role marker” (most often, these are prepositions or postpositions. For example, in English “from” is a delimiter for the goal role and its argument is on its right.

So “John” is a possible argument for the source role, but “Mary” should not be. Ubiquity can figure this out because English has the property en.branching = 'right'.

In Japanese, on the other hand, the argument of a delimiter like から (“from”) is found on the left of that delimiter, so en.branching = 'left'.

In general, if your language has prepositions, you should use .branching = 'right' and if your language has postpositions, you can use .branching = 'left'.

For more info:

• see branching on Wikipedia.

Defining your roles

  en.roles = [
    {role: 'goal', delimiter: 'to'},
    {role: 'source', delimiter: 'from'},
    {role: 'position', delimiter: 'at'},
    {role: 'position', delimiter: 'on'},
    {role: 'alias', delimiter: 'as'},
    {role: 'instrument', delimiter: 'using'},
    {role: 'instrument', delimiter: 'with'}
  ];

The second required property is the inventory of semantic roles and their corresponding delimiters. Each entry has a role from the inventory of semantic roles and a corresponding delimiter. Note that this mapping can be many-to-many, i.e., each role can have multiple possible delimiters and different roles can have shared delimiters. Try to make sure to cover all of the roles in the inventory of semantic roles.

For more info:

• Writing commands with semantic roles
• the proposed inventory of semantic roles
• Wikipedia entry on thematic relations

Entering your anaphora (“magic words”)

  en.anaphora = ["this", "that", "it", "selection", "him", "her", "them"];

The final required property is the anaphora property which takes a list of “magic words”. Currently there is no distinction between all the different deictic anaphora which might refer to different things.

Special cases

Some special language features can be handled by overriding the default behavior from Parser. Many of these features are still in the works, however, so we’d love to get your comments!

Languages with no spaces

If your language does not delimit arguments (or words, more generally) with spaces, there will be a need to write a custom wordBreaker() function and set usespaces = false and joindelimiter = ''. For an example, please take a look at the Japanese or Chinese.

Case marking languages

If you have a strongly case-marked language, you’ll have to write some rules to identify those different cases in wordBreaker() and then add some extra roles for these case markers, but for a number of languages the current design does not allow an elegant solution for parsing such arguments. Updates to this issue will be posted to this trac ticket.

In the mean time, however, if you could write a parser even with only the prepositions/postpositions in your language, that would be a great benefit in getting started in your language. UPDATE: a proposal on how to deal with strongly case-marked languages has been written here: In Case of Case….

Stripping articles

Some languages have some delimiters which combine with articles. For example, in French, the preposition “à” combines with the masculine definite article “le” but not “la”:

1. à + la = à la
2. à + le = au

You can add both “à” and “au” as delimiters of the goal role, but then you will get feminine arguments back with the determiner (e.g. “la table”) while masculine arguments would be parsed without a determiner (e.g. “chat”).

1. “à la table” = “to the table”
2. “au chat” = “to the cat”

One possible solution to this is to write a custom cleanArgument() method. After arguments have been parsed and placed in their appropriate roles, each argument text (say, “la table” or “chat”) are passed to cleanArgument(). You can simply write a cleanArgument() to strip off any “la ” at the beginning of the input and return it and both example inputs will get normalized arguments: “table” and “chat”, respectively. UPDATE: For more up-to-date information on how to deal with these types of articles, please see Solving a Romance Problem.

Test your parser

Now you can go into about:config and change extensions.ubiquity.language to be your language code and restart. All the verbs and nountypes at this point will remain the same as in the English version, but it should obey the argument structure (the word order and delimiters) of your language.¹ If you run into any trouble, feel free to ask for help on the Ubiquity i18n listhost or find me on the Ubiquity IRC channel (mitcho @ irc.mozilla.org#ubiquity). Of course, once you’re at a good stopping point, please contribute your language file to Ubiquity!

More to come…

At this point, you’ve only localized the argument structure of your language… additional work will be required to localize the nountypes and verb names, which is the subject of ongoing discussion… join the Google Group to get in on the discussion!

Related posts:

1. Ubiquity Parser: The Next Generation Demo
2. Rolling out the Roles
3. Foxkeh demos Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

Foxkeh demos Ubiquity Parser: The Next Generation from mitcho on Vimeo.

There are some details which are not covered in this introductory video, such as how to deal with case marking languages or languages without spaces. Hopefully this’ll inspire some people to play with the demo, though. I’d love to hear your comments! ^^

Related posts:

1. This week on Ubiquity Parser: The Next Generation
2. Ubiquity Parser: The Next Generation Demo
3. Localizing Ubiquity: an open letter to linguists

Related posts brought to you by Yet Another Related Posts Plugin.

Last week I released a proof-of-concept demo of the next generation Ubiquity parser design and it was also the focus of discussion in our weekly internationalization meeting.¹ Christian Sonne even wrote a Danish plugin for it during the meeting—a testament to the pluggability and of the new parser design.

In addition, at the Ubiquity weekly meeting, pushing this new parser into Ubiquity proper was identified as a key goal of Ubiquity 0.2, making frequent iteration and debate over this parser essential.

To that end, I’ll highlight some of the changes made to the parser demo codebase in the past week:

• left-branching support and a Japanese parser
• basic French parser
• a timer display
• Danish parser by Christian Sonne
• synonyms: as an example, you can now use “purchase” or “buy,” both of which point to the same verb.
• verb name localizations: you no longer need to use the English verb names with different languages. (Currently only Japanese has any verb localizations.)
• a number of optimizations and corrections

I encourage you to check out the demo again or check out the source on BitBucket.

➔ Ubiquity Parser: The Next Generation demo

I’d love to get comments, patches, or additional parsers! Thanks! ^^

Related posts:

1. Ubiquity Parser: The Next Generation Demo
2. Ubiquity in Portuguese
3. Localizing Ubiquity: an open letter to linguists

Related posts brought to you by Yet Another Related Posts Plugin.