+ + =

That is all.

Related posts:

1. 北京 Part 2: Summer Palace, bargaining, The Tree, and fried apple pie

Related posts brought to you by Yet Another Related Posts Plugin.

Voicemail from Jesse from mitcho on Vimeo.

Related posts:

1. Foxkeh demos Ubiquity Parser: The Next Generation
2. Changes to Ubiquity Parser 2 and the Playpen
3. HookPress: Webhooks for WordPress

Related posts brought to you by Yet Another Related Posts Plugin.

1. Confusing ids and classes. ids should be unique on the page… but here’s an instance of using multiple instances of the same id in order to format them together.
   

<div id="indent"> <div id="number">4.2.1</div> <div id="page">161</div> <div id="section">Old French (Adams 1987)</div>
</div> <div id="indent"> <div id="number">4.2.2</div> <div id="page">164</div> <div id="section">The evolution of the dialects of northern Italy</div>

1. Putting a class on every instance of something. Everything paragraph should be formatted equivalently. We get the point.
   

<p class=MsoNormal><b>The English Noun Phrase in Its Sentential Aspect</b></p>
<p class=MsoNormal>Steven Paul Abney</p>
<p class=MsoNormal>May 1987</p>

1. Using blank space for formatting.
   

<p class=MsoNormal><o:p>&amp;nbsp;</o:p></p>

1. CSS styles that don’t exist. Browsers just ignore these anyway…
   

<p class=MsoNormal>One factor in determining which worlds a modal quantifies
over is the temporal argument of the modal’s accessibility relation.<span
style='mso-spacerun:yes'>  </span>It is well-known that a higher tense affects
the accessibility relation of modals.<span style='mso-spacerun:yes'> 
</span>What is not well-known is that there are aspectual operators high enough
to affect the accessibility relation of modals.<span style='mso-spacerun:yes'> 
</span&gt

The solution

My solution was to write a perl script which takes care of a number of these issues. It’s not foolproof and doesn’t involve any voodoo—for example, it can’t retypeset things which were formatted using whitespace—but it does a good job as a first pass.

You can run the script by making it executable (chmod +x cleanwordhtml.pl) then specifying a target filename as an argument. For example,

./cleanwordhtml.pl source.html > clean.html

I used this with a simple bash for loop to run over all my files:

for f in */*.html; do ./cleanwordhtml.pl $f > ${f%.html}-clean.html; done;

Hopefully someone else can benefit from my experience.

Related posts:

1. Markdown for WordPress and bbPress
2. 回収 vs. 収集 and Better Word Meanings Through Usage
3. The Most Beautiful Word

Related posts brought to you by Yet Another Related Posts Plugin.

Related posts:

1. Oh Evan
2. ワンセグ TV coming to the iPhone
3. RISK on the iPhone

Related posts brought to you by Yet Another Related Posts Plugin.

Time working for Mozilla: 6.5 months

Mozilla-related blog posts written: 69

Academic papers written on Ubiquity: 1

Ubiquity presentations given: 5

Screencasts made: 8

Most popular video on Vimeo: Ubiquity 0.5 日本語紹介ビデオ, the Japanese Ubiquity 0.5 introduction video: 2252 views

Languages Ubiquity commands and parser now support: 6

Commits to the Ubiquity repository: 492

Other web projects started during this period: 2+ (Ten Grand Is Buried There, HookPress)

TechCrunch references: 2 (1, 2)

Countries worked in: 2

Mythical Kiwis worked with: 1

References to bugs I introduced as “glitcho”s: 1

Extremely disturbing homages to me and Django: 1

Friends made; experience gained; lessons on Open-ness learned; personal growth: priceless enumerable

Thanks to all who made this experience amazing, beginning with Aza, Jono, Atul, Blair and the rest of the Labs team; intern extraordinaire Brandon; the always thoughtful and friendly Mozilla Japan team; and of course the fantastic Ubiquity community! Please visit me in Boston—I should be around for a while.

Related posts:

1. Report from Mozilla Party JP 10!
2. This week on Ubiquity Parser: The Next Generation
3. Notes from BarCamp Tokyo 2009

Related posts brought to you by Yet Another Related Posts Plugin.

This blog post includes a lot of complex CSS-formatted graphs which may be best viewed in — what else? — Firefox. You may also want to access this blog post directly rather than through a planet.

One portion of the problem description above merits clarification: I define “without sacrificing quality” to mean that, if we did not throw out any candidates early and waited until all the scores are computed fully and accurately, we would still yield the same top winners. This already gives us the key insight towards an appropriate solution: we can only throw out candidates when we know that it has no further chance of making it up into top candidates.

Let’s get formal

Let’s call the score of candidate at time in the derivation and we’ll assume that the score derivations are done in parallel with a unique origin ().¹ We’ll use the notation to represent the equilibrium or final score, equal to for all a certain which exists for each candidate. This function thus defines a time series for each candidate.

Given a set of candidates , we want to find the best subset of candidates; that is, such that

.

Approach 1: A Threshold Model

The key insight above would naturally give us what I call the threshold model. Here, we require the score sequences to be non-increasing: . This way, we can naturally throw out candidates which have reached below a certain threshold (or attained a certain level of badness, you might say) which we can then be sure will never recover.

For example, suppose the following diagram represents the scores of five different candidates after the first four time steps of the derivation. (The full gray bar marks the initial score () and the arrows indicate the successive score differentials.) The vertical line marks the threshold, .

We can tell after four steps that candidates 2 and 4, given that the score sequences are non-increasing, have no chance to finish their derivation with a score . What is important to note, however, is that candidate 4 already had no chance of beating the threshold after three steps. There was no need to calculate the fourth derivation of the score of candidate 4 (). In other words, after three steps, we could completely take candidate 4 out of the running and after another step, take candidate 2 out of the running.

C1      C2      C3      C4      C5      …	→	C1       C2       C3       C4       C5       …	→	C1        C2        C3        C4     C5        …	→

This non-decreasing score approach was used in Ubiquity Parser 2 until just recently, and you can in fact still play with it on the online Ubiquity Parser TNG demo. In that version, every parse started with an initial score of 1 and every score factor would be a value between 0 and 1. Every score factor was multiplied onto the previous score throughout the derivation, making it trivially non-increasing.

The problem with this approach is how to choose a smart threshold and that, given a constant threshold, you may get a different number of results for every different candidate set (i.e. parser query). If your score indicates a meaningful value with an a priori specified target of acceptable values, having a threshold makes sense. In the case of Ubiquity, however, the interface expects a certain number of suggestions to be returned.² If we plan to display five suggestions but the parser only returns four, even though there were other candidates, there must be a very good reason and justification for that threshold value.

Approach 2: Raising the Bar

The problem with Approach 1 was that there was no way of guaranteeing that we would yield our predefined winning candidates. Even if at some point in the derivation we are left with candidates still above the threshold, as the only restriction we have is that our score series are non-increasing, there is still a possibility that those remaining candidates’ scores will drop below later in the derivation.

We must instead at some point in the derivation identify (a) a set of at least candidates which will not get “worse” in the derivation and (b) candidates which have no chance of overtaking the (a) candidates. In this situation we can safely throw out the (b) candidates.

One way to do this is to require that all the scores are bounded and non-decreasing. By virtue of being non-decreasing, our top candidates at any point in our derivation will never get “worse” afterwards, satisfying condition (a). If relatively early in the computation we can compute a bound , we can identify candidates which will never surpass the top candidates in group (a) above, satisfying condition (b).

In the example below, and the thin bars mark the upper bounds . At we can identify candidate 2 and 4 as being our top two candidates. Note that there is one candidate, candidate 5, whose upper bound is less than both and . By definition and because the scores are non-decreasing and . Therefore

and

and we can thus throw out candidate 5 at this point. By the same logic, after we can throw candidate 2 out of the running.

C1      C2      C3      C4      C5      …	→	C1       C2       C3       C4       C5     …	→	C1        C2     C3        C4        C5     …	→

Calling this the “raising the bar” method refers to the fact that, at any particular time , the “bar” is and every other candidate must have an upper bound greater than the bar in order to not be thrown out of consideration. This “bar” itself is, together with the component scores, non-decreasing, decreasing the number of surviving candidates over time.

In the case of the Ubiquity parser we could build such a non-decreasing and bounded scoring model by using an additive model. As the main component of parser scoring is how well the parsed arguments match the verbs’ specified nountypes, we could simply add up all the confidence scores of each nountype suggestion, each of which are a value between 0 and 1. This would trivially be non-decreasing. As each parse has a finite and known number of parsed arguments, we could easily determine a bound as well. For example, say a parse has two arguments. Before we check each of the nountypes’ match scores, we already know that .

Unfortunately, there are also other factors which we would like to consider in our parses which may not fit into this non-decreasing model so easily…

Approach 2’: The Rising Sun Model³

One problem with both of the previous approaches is that it requires that the scoring schemes be either non-increasing or non-decreasing across the derivation. There are many situations, however, where you would want different factors to affect the score both positively and negatively. In the case of the Ubiquity parser, here are some different factors which could be good positive and negative score factors in computing the score of each parse.

As we see, there are both positive and negative factors which we hope to consider in scoring our possible Ubiquity parses. They key to making this work is by noting that Approach 2 only requires that the scoring series be bounded and non-decreasing after a certain known time in the derivation. For example, even if a parse involves a number of decreases early in the parse derivation, if after a certain point we can be certain that it is non-decreasing and bounded, we can simply use that bound and start eliminating poor candidates at that time (in this example, after ).

This is very much possible in the Ubiquity parser as, given the Ubiquity Parser 2 design, the negative factors such as whether the parse has a verb from the input or not (step 2), whether multiple arguments are identified with the same semantic role (step 4), and how many of the verb’s arguments are in the input (step 4) can be identified early on in the derivation, all before the very computationally intensive step of nountype detection (step 7) and argument suggestion (step 8). In this way, we can front-load all the negative factors in scoring and continue to use a version of Approach 2 to optimize our parsing.

We can moreover make the effect of the negative factors be felt across the entire derivation by figuring the negative factors into a factor between 0 and 1 and multiplying it onto each of the positive factors being added. In other words, we can compute all the negative factors into a single score multiplier earlier in the derivation and then afterwards when adding up each of the positive factors simply applying that score multiplier to the score derivation:

.

This model is what is going on under the hood in Ubiquity Parser 2. The Parser.Parse class has a property called .scoreMultiplier which contains the score multiplier as described above. A method called .getMaxScore() is implemented in addition to .getScore() so that, even before all of the nountype suggestion scores have been computed (e.g., in the case of asynchronous suggestions) .getMaxScore() can be used as an upper bound and compared to the in-progress scores of other candidates and lower candidates can thus be taken out of consideration earlier in the parse process.

Conclusion

In this blog post I’ve laid out a few different iterations of approaches I’ve thought of on the problem of scoring and ranking Ubiquity suggestions in a smart way. While some of the basic mechanisms of front-loading the negative factors into a scoreMultiplier and the computation of the maxScore (or upper bound) have been implemented, the actual optimization algorithm described here of removing parses from consideration earlier in the parser query has yet to be implemented in Ubiquity Parser 2 and I look forward to seeing it in action. In addition, there are surely factors I haven’t considered in the scoring or further tricks to improve the optimized scoring algorithm. I’d love to get your feedback and ideas on this topic. Thanks!

Related posts:

1. Scoring and Ranking Suggestions
2. Ubiquity Parser: The Next Generation Demo
3. This week on Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

Every day on the way to work I walk by a fine establishment known as Yoshinoya (吉野家), Japan’s largest gyudon (牛丼) chain restaurant. For those of you whose lives have yet to be graced by gyudon, it’s a bowl of rice topped with beef and onions stewed in a sweet-savory soy-based sauce. Loving gyudon and being a cheapskate, I naturally noticed the recent 50 yen off gyudon promotion at Yoshinoya. The above photo is a photo of part of that sign.

Part of this sign, though, made me think about our new Ubiquity parser. In particular, it was the attachment ambiguity in the end date of the promotion. The text in the photo above literally is “April 15th (Wed.) 8PM until”. (Note that Japanese is a strongly head-final language, and that the “until” is a postposition.) There are two possible readings for this expression, as illustrated by the two composition trees below.

The first tree, on the left, represents the reading “until (April 15th 8PM)”, while the second represents two arguments: “on April 15th” and “until 8PM”. In other words, in the first reading, the promotion begins at some earlier date and extends until April 15th at 8PM while, in the second reading, the promotion is one day only, on April 15th, until 8pm. Such syntactic ambiguities are called “attachment ambiguities” in linguistics as it is an ambiguity of where different arguments “attach” in a tree representation.

This attachment ambiguity was possible because there was no clear marker on “April 15th,” which may have disambiguated it as “on April 15th”. In fact, in many languages this time position argument comes with no case marker or preposition, or it’s optional, making parsing for them difficult. If such a sentence is entered with spaces, the Ubiquity Parser: The Next Generation would try a parse where “8PM” is the “until” or goal argument and “April 15th” is an object argument, but it will only check its noun type, not put it in the correct semantic role (position). Perhaps this is something to think about in the future.

These types of situations will surely come up as we continue work on the Ubiquity parser, making it essential to look at different languages. Are there certain kinds of arguments in your language that do not have any word-external markers such as case or prepositions/postpositions?

Related posts:

1. User-Aided Disambiguation: a demo
2. Ubiquity in Firefox: Focus on Japanese
3. Talking Ubiquity in Japan: 拡張機能勉強会にて発表

Related posts brought to you by Yet Another Related Posts Plugin.

The issue of ranking Ubiquity suggestions can be restated as predicting an optimal output given a certain input and various conflicting considerations. Ubiquity (1.8, as of this writing) computes four “scores” for each suggestion:

1. duplicateDefaultMatchScore: 100 by default—lowered if an unused argument gets multiple suggestions (in the words of the code: “reduce the match score so that multiple entries with the same verb are only shown if there are no other verbs.”)
2. frequencyMatchScore: a score from the suggestion memory of the frequency of the suggestion’s verb, given the input verb (currently the first word) or nothing, in the case of noun-first suggestions
3. verbMatchScore: float in [0,1]: (as described here)
   • 0.75 is returned in case there it is a noun-first suggestion (by virtue of the fact that String.indexOf('')==0)
   • 1 if the verb name is equivalent across input-output
   • in [0.75,1) if the input is a prefix of the suggestion verb name
   • in [0.5,0.75) if the input is a non-prefix substring of the suggestion verb
   • in [0.25,0.5] if the input is a prefix of one of the synonyms
   • in [0,0.25) if the input is a non-prefix substring of one of the synonyms
4. argMatchScore: the number of arguments with matching “specific” nountypes, where “specific” is designated by the nountype having property rankLast=false.

With the numeric scores for each of these criteria, a partial order of suggestions is constructed using a lexicographic order: that is, compare candidates first using duplicateDefaultMatchScore, break ties using frequencyMatchScore, if still tied break using verbMatchScore, and if still tied break using argMatchScore. This paradigm of constraints is called “strictly ranked” and a corollary of this is that lower constraints, no matter how well you score on them, can never overcome a loss at a higher constraint. A crucial corollary of this system is that lower constraints’ scores need not be computed if a higher constraint already dooms it to a lower position.¹

Ranking in The Next Generation

One of the goals of Parser The Next Generation is to make noun/argument-first input first-class citizens of Ubiquity, improving their suggestions in particular to the benefit of verb-final languages. Arguments will be split up and tested against different noun types before a verb is even entered into the input, in which case target verbs can be ranked according to the appropriateness of the input’s arguments. As such, I believe the argMatchScore criteria above should either be ranked higher in a strictly ranked model or be allowed to overtake lower scores for the higher constraints in a non-strictly ranked model.

The Parser The Next Generation proposal and demo currently orders using a product of various criteria’s scores, rather than a lexicographic order of strictly ranked constraints. The component factors are:

1. 0.5 for parses where the verb was suggested
2. 0.5 for each extra (>1) object argument (essentially “unused words” in the previous parser)
3. the score of each argument against that semantic role’s target noun type
4. 0.8 for each unset argument of that verb

Each component score is a value in [0,1], so the score is always non-decreasing across the derivation. This offers a natural way to optimize the candidate set creation: if a possible parse ever gets a score below a magic “threshold” value, it is immediately thrown away.

A possible problem with the current Parser TNG scoring model is that it will implicitly hinder verbs and parses with more arguments as it could have more sub-1 noun type score factors—this consideration may be great enough that a weighted additive model should be considered over a multiplicative one.

How do you think we can make Ubiquity’s suggestion ranking smarter? What other factors should be considered, and what factors could be left out?

Related posts:

1. Ubiquity Parser: The Next Generation Demo
2. Ubiquity in Firefox: Focus on Japanese
3. Ubiquity Commands by The Numbers

Related posts brought to you by Yet Another Related Posts Plugin.

Today I’ll examine highlight some disparate typological data to help us understand these questions: where’s the verb? and what does the verb look like? There are broadly three different verb forms taken in commands in different languages:¹

1. the infinitive,
2. subjunctive mood, or
3. a special verb form such as imperative, participial, or conjunctive (such as Japanese て form)

Let’s give an example of each:

Infinitive (English):²

1

Hit me!

Subjunctive mood (Modern Greek): “Eat it all!”

2
3

Na   to fas olo!
SUBJ it eat all

Imperative form (French): “Eat it!”

4
5

Mange   -le!
eat.IMP it

It’s important to note that some languages have multiple forms available for the same command. For example:

Dutch: three ways to say “watch out!” with the same verb

1. Infinitive: Oppassen!
2. Imperative: Pas op!
3. Participial: Opgepast!

Similarly, I received a great comment by PhiliKON on German and associated data by Robert Kaiser on my blog post yesterday:

German: “search hello with google”

1. Infinitive: hello mit google suchen
2. Imperative: suche hello mit google

In addition, German and Dutch are interesting as they are verb second (V2) languages, so the verb may surface at the beginning or the end of the sentence, depending on the form.

The new parser design (which you can demo) assumes for simplicity that the verb should be found at the beginning or the end of the input, which is consistent with the data I’ve seen (modulo clitics). Multiple verb forms could be accounted for by supporting “synonyms” of the verbs.

What are the different ways verbs are expressed in commands in your language? Is the verb always found at the beginning or the end of the sentence? Is it ever somewhere in the middle?

Related posts:

1. Ubiquity i18n: questions to ask
2. Ubiquity Parser: The Next Generation Demo
3. Contribute: how your language identifies its arguments

Related posts brought to you by Yet Another Related Posts Plugin.

What caught my eye is the line “served with ご一緒に 配,” meant to be read as part of “Beef in BBQ sauce… served with Pepsi…”. The Chinese 配 (pèi) is fine here, meaning “with,” but the Japanese “ご一緒に” (goissho-ni) seemed awkward to me.

The issue is that this adverbial meaning “together” normally comes after the “what it’s with” in an order like (1) (glossed in (2)):

1
2

A B-と       ご一緒に
A B-and/with together

In other words, where English and Chinese both would say “A with B”, it is most natural in Japanese to say the equivalent of “A B with (together)”.¹ This is the reason why it seems unnatural to have anything between the “Beef in BBQ sauce…” line and “Pepsi…” line.

Looking at the rest of the menu, it’s clear that this isn’t a case where a native speaker wasn’t involved with the writing of the menu—the rest of the Japanese is perfect. The Japanese modifier was inserted there just for the sake of parallel design, to the detriment of the text’s naturalness. When have you seen design conflict with the structure of your language?

Related posts:

1. Three ways to argue over arguments
2. Testing Google’s Language Detection
3. Ubiquity in Firefox: Focus on Japanese

Related posts brought to you by Yet Another Related Posts Plugin.

Here’s a picture of an ad for Gaba, a big English conversation school in Japan, I snapped on a train recently. I felt the English sentence about Gaba’s satisfaction was extremely awkward, so I put it up on twitter to check with some other native speakers. My friends concurred. What do you think?

I personally think the sentence would be improved by removing the “the” in “the satisfaction.” Others offered “continues to rise” as possibly preferable to “continually rise.” English articles, especially the definiteness of abstract nouns, is very difficult for many non-native speakers. That being said, it’s sad for a sentence of such questionable acceptability to come from a company which, in theory, prides itself in its English ability and surely hires many native speakers. Gaba, shame on you.

Related posts:

1. 回収 vs. 収集 and Better Word Meanings Through Usage
2. Krashen The Party
3. White Protestants and Catholics don’t frequently attend religious services

Related posts brought to you by Yet Another Related Posts Plugin.

It’s always interesting to see these release spikes in download traffic. Note that this release was on the Wednesday but that was during the day, so Wednesday’s traffic is still higher than the normal ~300/day level, while the big peak (by day) is on Thursday. Too bad wordpress.org doesn’t give me hourly stats, though I guess that would be a little ridiculous.

YARPP is just about at that 35k download mark. I’m looking forward to the next release. ^^

Related posts:

1. Yet Another Related Posts Plugin 2.0
2. Keep up with Yet Another Related Posts Plugin with RSS!
3. Modifiying WordPress plugin activation behavior

Related posts brought to you by Yet Another Related Posts Plugin.

I am convinced that this bald approach will cost American families far less than the alternative—a continuing series of financial institution failures and frozen credit markets unable to fund economic expansion.

Unfortunately, the key phrase in this passage was widely mistranscribed in the media as a “bold approach.” But now that more details of the new Troubled Asset Relief Program have being released, Secretary Paulson’s true intentions are clear.

Chris Carey of Bailout Sleuth writes:

The Treasury Department tapped James H. Lambright [above center], head of the Export-Import Bank, as the interim chief investment officer for the $700 billion Troubled Asset Relief Program… The bailout program is being directed by Neel Kashkari [above left], who had been senior advisor to Treasury Secretary Henry M. Paulson Jr [above right].

Will this new program stem the global credit crisis? Maybe. But at least we can all agree… it’s a bald move.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

• 回収 may take things away from others when collecting while 収集 does not have that implication.
• Things that you 回収 may have been previously distributed by the actor themself while 収集 does not have that implication.¹

Not content with armchair theorizing, however, I decided to take advantage of one of the largest corpora in the world: Google.² To test my hypothesis, I chose two “objects of collection”, one you can take away (and often is distributed first) and one you can’t take away: アンケート (ankēto “survey,” from the French enquête) and 意見 (iken “opinion”). I then took the four resulting collocations³ on Google in quotes (“•”) and recorded how many hits there were.

A better way to organize this data is as follows:

This data clearly supports the hypothesis I laid out above: アンケート, which can be taken away from people and is often distributed first, occurs much more likely with 回収 than 収集. 意見, on the other hand, which crucially cannot be taken away when collected, occurs much more likely with 収集 than 回収.

While this one example doesn’t prove anything in and of itself, it does help clarify with data a nuance between two near synonyms. While my hypothesis was borne out here, native speaker intuitions on word nuances and distinctions can be unreliable.⁴ This type of quick test can be very helpful for language learners and instructors alike.

Languages very often have words which vary in very subtle ways. Just this Tuesday I went to a Tokyo Language Exchange Meetup, a great meetup which brought together various language learners and enthusiasts. A hot topic that night was words with very similar meanings—near synonyms. A few English learners were lamenting sets of words like {see, view, watch} and how difficult they are to learn. I myself have had the same experience studying Mandarin.

I noted that these difficulties in offering contrasting definitions often are due to the fact that word meanings are not just “what the word points to” but also the implication of “what it relates to”.⁵ For example, “unborn baby” and “fetus” may point to the same thing, but are used in different contexts, in contrast to different other terms, for differing effect. Similarly “Death Tax” and “Estate Tax.” “Kneel” and “genuflect.”⁶

The concept of word meanings being “what it points to” and “what it relates to” also helps explain why certain words are difficult to translate. Fillmore uses the Japanese example of ぬるい (nurui) which is the de facto translation of “lukewarm.” However, some Japanese speakers will only use ぬるい in contrast with “hot,” i.e., hot tea can become ぬるい over time but ice water does not become ぬるい. In contrast, English “lukewarm” can be used to describe things that are initially or prototypically hot or cold. “What the words point to” in this case is the same but “what it relates to” or, here, “what it contrasts with” is different, making it an imperfect (though very close) translation.

Every language has near synonyms which vary slightly in nuance but this nuance or “feeling” is borne out objectively in data. Looking at what words certain terms relate to in real usage is often the key to getting a richer understanding of vocabulary.

Related posts:

1. Testing Google’s Language Detection
2. Taipei find: a dictionary of Chinese-Japanese false cognates
3. The Most Beautiful Word

Related posts brought to you by Yet Another Related Posts Plugin.

I don’t think those two were related.

Related posts:

1. Oh Amazon, you’re so funny

Related posts brought to you by Yet Another Related Posts Plugin.

I’ve been a fan of the SNL Digital Shorts since Lazy Sunday, but this is absolutely something else. It’s a brilliant piece of cross-cultural parody. Many on the associated Hulu page had some questions, however, so I decided to write up a little explanation of what’s actually going on in this short, and why I love it so.¹

The Digital Short begins with the Japanese version of the intro sequence, including a shrine, a 700 series bullet train, and the Scranton city sign now showing Amagasaki (尼崎市), a similarly industrial city near Osaka. Dwight shredding paper with Japanese text and Jim eating noodles are nice touches. All the names, in case you were wondering, are possible Japanese names (modulo Jim’s actor’s name being in katakana, and thus exclusively foreign). After Michael Scott with extra black hair readjusts his lucky cat (manekineko, 招き猫), we get to the brilliant title card.

As Japanese lacks definite and indefinite articles, the word “The” is replaced with 「その」 sono, the demonstrative ‘that’, making the title literally “That Office.” Lacking a straightforward replacement for “The,” however, I feel that this is a very cute artifact of overly-faithful translation.

The short itself runs through the Japanese versions of a few key scenes from the first episode of The Office. In the first, Pam is answering the phone and Michael interrupts her in his signature way, repeating her name (or, the Japanese equivalent of “Pam”, pamu パム) and then dropping -san, a personal name suffix—the equivalent of Mister or Miss—and smiling into the camera, content with his own cleverness. Pam says something indiscernible to Michael, referring to him as Tanaka-san (the “Mr. Smith” of Japan—even though his name plate accurately said “Michael Scott” マイケル・スコット), to which Michael mumbles 「そういうことです」, a phrase meaning “and that’s that,” or “and that is the case.” My guess is that this was the attempted translation of “that’s what she said.”

Michael then goes on to say 「日本で一番面白いボスです」 (nihon-de ichiban omoshiroi bosu desu, ‘[I am] the most interesting (=funniest) boss in Japan’). Steve Carell’s snicker halfway through that line, in response to his trying really hard at producing it, is very cute. The mug itself says 「世界中で一番面白い社長」(“world’s funniest company president”). This reminds me of my dad when he speaks Japanese, in the best way possible. ^^

Next is of course the jello scene. Dwight picks up the phone with the phone-appropriate 「もしもし」 (moshimoshi) and Jim asks where the stapler is. Dwight yells back 「バカ！」 (baka, ‘stupid!’) and Pam laughs, though in the stereotypical Japanese female’s high pitch manner, appropriately covering her mouth (though Pam also actually does this in the original). Michael walks in and they all apologize, 「ごめんなさい」 gomennasai. Although the bowing is a bit excessive in a classical SNL parody way, the traditionally hierarchical status quo of Japanese offices is very succinctly reflected here.

The next scene is also from the first episode of The Office, where Michael introduces himself, 「僕は君たちのリージョナル・マネージャ」 (boku-wa kimitachi-no regional manager, ‘I am your Regional Manager’). Dwight states that he is the “Assistant Regional Manager” (アシスタント・リージョナル・マネージャ) and then is corrected, 「リージョナル・マネージャのアシスタントです」 (regional manager-no assistant desu, ‘[you are] Assistant to the Regional Manager’). It’s very cool to see how this back and forth translates beautifully, and to see these actors execute it with the right timing and effect in a foreign language. Michael asks 「どうしてここにいるのだ」 (doushite koko-ni irunoda, ‘why are [you] here?’) and leads them in the radio exercises (ラジオ体操), saying 「じゃ、ラジオ体操をしましょう」 (jya, rajio taisou-wo shimashou, ‘well then, let’s do the radio exercises’).

Here’s another beautiful cultural point. These “radio exercises” are real, as every Mori no Ike villager knows, broadcasted over public radio and TV every morning, and are often done en masse at schools and some businesses. Pam’s then notes 「田中さんはみんなの健康を心配しています」 (Tanaka-san-wa minna-no kenkou-wo shinpai-shite-imasu, ‘Mr. Tanaka (=Michael) is concerned for everyone’s health’) in a conference-room reflection, and we see Stanley doing the crossword again.

After hurting his shoulder and cooling it with some oolong tea—the same bottle that was on Pam’s counter in the first scene—Michael reflects:

「今日はいい日でした。」 (kyou-ha ii hi deshita, ‘Today was a good day.’)
「いい仕事をした。」 (ii shigoto-wo shita, ‘[I] did good work.’)
「そう思う…かな？ はいはいはい！」 (sou omou… kana? hai hai hai!, ‘I think this way…? Yes yes yes!’)

The last line there is beautifully translated, capturing the essence of Michael in Japanese. As Japanese is a verb-final language, you literally say “blah blah blah I think” to mean “I think blah blah blah”, which may help explain the last phrase, 「そう思う」. Finally, the 「かな」 thrown in at the end turns the entire sentence, which was declarative up till then, into a question, which the bobble-head then answers. Brilliant!

In the final scene, Michael’s singing karaoke and Dwight yells 「かんぱい！」 (kanpai!, ‘bottoms up!’). The final credits list Sarah Sawyer and Hanna(h) Sawyer as producers… I wonder if they were actually involved with this Short or if they are also made up as well.

The details in the Short are great: the little Hello Kitties and origami, orchid plant on the reception desk (Japanese love orchids—or wait, maybe that’s just my grandfather), and all the copy paper that had 「コピー用紙」 (kopii-youshi, ‘copy paper’) pasted on. There’s a bonsai tree on Jim’s desk and his spoon is replaced with chopsticks.

If you want to get picky, of course, there are many rough edges… the incorrect use of 「ステープラー」 (how you would say “stapler” in Japanese) in lieu of 「ホッチキス」,² some text being poorly typeset, etc. But overall, this SNL Digital Short was obviously written by someone with a solid (albeit stereotypical) understanding of Japanese culture and strong intermediate Japanese skills. If the goal was simply only to play off of Japanese stereotypes, accurate Japanese wouldn’t have even been necessary, and so I really appreciate the effort that went into this. In addition, Steve Carell et al’s delivery in a language they don’t speak, in my opinion, is commendable.

“It’s funny because it’s racist,” in the best way possible. Bravo!

Related posts:

1. Taipei find: a dictionary of Chinese-Japanese false cognates
2. Bailey won the Japanese Language Speech Contest
3. Mailplane Japanese localization available!

Related posts brought to you by Yet Another Related Posts Plugin.

As Google adds ten more languages to its machine translation service, it seems to be on its way to becoming the most convenient universal translator of the world’s popular languages. Google’s handling of languages of course isn’t perfect, however—in particular, I’ve been complaining to friends for a while about the weaknesses of Google’s handling of queries in Chinese character (漢字/汉字) scripts. In this post, I run some tests using Google’s Language Detection service to try to better understand its handling of Chinese character queries.

Background

Chinese characters have been used all across East Asia, most notably in Chinese, Japanese, Korean, and Vietnamese (the “CJKV”). Prescriptivist writing reforms in Communist China and Japan have simplified many characters, though. Some characters were simplified in the same way, some in different ways, and some in only one country but not the other. For more information, there’s Wikipedia or Ken Lunde’s CJKV Information Processing.

The problem

The issue comes up when you try to search for a word in Chinese characters which clearly came from one Chinese character-using language. From my experience, Google doesn’t consider which language you are a user of, based on the query, and returns many results in other Chinese character-using languages as well.¹

Take, for example, a query like “七面鳥”, meaning ‘turkey’ in Japanese. While all characters are very common in traditional Chinese (鳥 is simplified to 鸟 in China), the combination “七面鳥” is quite rare in Chinese. However, when you search for “七面鳥,” many of the first results are in Chinese and only two of the first ten results are in Japanese.

Does Google’s corpus not identify “七面鳥” as a primarily Japanese word? Google does indeed attest to this fact: searching for “七面鳥” and limiting to a certain language yields the following number of hits. A similar effect can be seen with Japanese words such as “芝生” (‘grass’) or “泥棒” (‘burglar’). The “Japanese on first page” column gives the number of results that are in Japanese which come up in a language-unspecified search from the US.

In a perfect world, I would like Google to identify the language that the query is in, and then weigh results that are in that language higher in the results list. So the issue comes down to one of language detection.

There are broadly two different approaches to language detection and, indeed, all natural language processing problems: parsing and counting. In this case, parsing involves trying to break apart the query into words and then computing how likely such a string of words is in each given language. Counting simply takes an inventory of the characters given and compares them to their frequencies in each language, computing how likely such a string of characters is in each language. Parsing is the “smarter” approach, but more difficult and computationally intensive.

Google was kind enough to give us an language detection AJAX service so we can get a sense for how their language detection works. This service also gives a “confidence” value on the detection result. For the rest of this entry, we’ll test some hypotheses against this service and conclude at the end.

Do spaces matter?

No. While spaces are sometimes used in Japanese and Chinese writing to represent word boundaries, especially around numbers and roman letters, they also are seen on the web to encourage line breaks. It would make sense for Google’s language detection service to ignore spaces in Chinese character queries and that does seem to be the case. All tests I ran with Chinese character queries gave the same result with same confidence with and without spaces in random places.

Does order matter?

No. This was slightly disappointing to see. I took the Japanese string “骨粗鬆症” (‘osteoporosis’, if you’re curious) and ran every permutation against the language detector and got the same results, including the same confidence values. This is a clear indicator that Google uses only counting, not parsing, in their parser.

Does repetition matter?

Yes. Now that it seems that Google does not use any parsing and only uses character frequencies in identifying the source language, let’s see how repetition can affect the detection service.

First, I took some Chinese character strings and ran them through the detection service with different numbers of repetitions, e.g. “参加”, “参加参加”, “参加参加参加”, “参加参加参加参加”… The queries I used were the following:

For each token type, the detection service made up its mind quite quickly. Its confidence, however, was more interesting.

Each of the confidence values dips sharply after three, five, or ten repetitions. Note, however, the length of the tokens which dipped at each of those points. I interpret this to mean that there is a different parser for less than ten characters and ten or more characters. However, the detection service did not change its answer after this point on any of the tokens.

Second, I took two characters, “簡” and “体,” and crossed different numbers of them together to see how that would affect the language detected. Note that “簡” is used in traditional Chinese and Japanese, while “体” is used in simplified Chinese and Japanese.

Conclusion

For Chinese character-based languages, Google’s language detection algorithm uses simple counting rather than parsing, identifying languages by looking at the frequency of characters rather than the frequency of words. As such, the algorithm simply acts as a script detector, not a language detector. Moreover, as a simple counting method is used, duplicating characters used in one language but not another can very easily skew the resulting output.

As a trivial aside, it seems that Google’s algorithm is slightly different for strings less than ten characters, as can be seen in a dip and then rise of confidence values after ten characters.

Related posts:

1. Taipei find: a dictionary of Chinese-Japanese false cognates
2. Bailey won the Japanese Language Speech Contest
3. Setting Language Research to Music

Related posts brought to you by Yet Another Related Posts Plugin.

White Protestants and Catholics backed Mrs. Clinton, but Mr. Obama was strongly supported by voters who frequently attend religious services.

Seeing as backing Mrs. Clinton and supporting Mr. Obama are, in terms of votes, mutually exclusive, this sentence entails that white Protestants and Catholics (the majority of ) are not a part of “voters who frequently attend religious services”, as is demonstrated by the infelicity of the following sentence:

“Group A did A, and Group B did not do A — but Group A is part of Group B.”

Well, that just settles it then.

Related posts:

1. Obama for Taiwan 2008

Related posts brought to you by Yet Another Related Posts Plugin.

So why rental? Thus spoke Steve: “Your favorite movie… most of us watch movies once… maybe a few times.”¹ Currently number eight on the top rentals is one of Paul Sally’s favorites, The Usual Suspects . From the iTunes Store description:

There are a handful of movies that demand a second viewing—because they’re so good, or because a surprise ending gives every scene a new meaning when it’s watched a second time. The Usual Suspects is both.

Related posts:

1. Great news! You can opt-out from Omniture’s 192.168.112.2o7.net

Related posts brought to you by Yet Another Related Posts Plugin.

Wikipedia notes: “Norton grew up in the Midwest, but considers the Jersey Shore his home… He currently lives in San Francisco, California.” So, is this a Jersey Shore or California thing? I have no idea.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.