mitcho.com » Firefox

Checking mochitest test coverage

mitcho — Tue, 22 Mar 2011 23:05:19 +0000

One of the last bugs for Firefox Panorama was bug 625818: “Check Panorama mochitest test suite coverage”. Our automated tests ensure that we do not regress on existing functionality, but it’s only as good as its coverage: how much of the Panorama code base is actually being “hit” through the process of running the test suite.

Panorama went through a pretty rapid development cycle, making it into Firefox 4 which was released today (yay!). Moreover, for a while we were developing outside of mozilla-central, without the regular “patches require tests” requirement. This makes checking its test coverage particularly important.

Check out the final result, the Panorama test coverage report. The good news: our code coverage is 86%! (Some notes on what improvements can be made are in the bug.)

PhiliKON had previously worked on hooking into the JS Debugger service’s interruptHook to test xpcshell tests. I modified this code to run instead in the Mochitest browser chrome tests. This code can be found on the bug.

With this patch applied, I invoked the test suite with the following code: TEST_PATH=browser/base/content/tests/tabview COVERAGE_FILTER="*tabview*" COVERAGE=true make -C obj-ff-dbg mochitest-browser-chrome . That’s a regular mochitest-browser-chrome invocation with the COVERAGE=true flag which turns on code coverage checking, and COVERAGE_FILTER=*tabview* which filters out results from files which don’t have “tabview” in their paths. This creates a file called coverage.json in the working directory of the test suite, meaning, for me, obj-ff-dbg/_tests/testing/mochitest/.

This JSON file is a multidimensional array, with file paths and then line numbers as keys. The file paths here, as best as possible, have been converted into local filesystem paths. PhiliKON built a script which produces beautiful reports based on this output.

A word of warning: running with this JSD interruptHook is ridiculously slow. A number of tests for Panorama are timing-dependent (drag-drop tests, for example), making some of them fail, but that’s okay… as long as it completed not via a timeout, it actually did run through all the code. In order to get this to run through everything with some degree of control, I split up the mochitest tabview suite in to a few chunks. I then took the multiple resulting coverage.json files and passed them into another script, in tools/coverage/aggregate.py, which takes multiple JSON results like this and puts them together into a single JSON file. I then passed this aggregate JSON file to PhiliKON’s wonderful report script and—voila—the Panorama test coverage report! Easy as pie.

Related posts brought to you by Yet Another Related Posts Plugin.

Happy Halloween!

mitcho — Fri, 29 Oct 2010 20:22:51 +0000

Happy Halloween from the Firefox Panorama team!

We carved some pumpkins a couple days ago in my department. I carved the Panorama logo above, but also one of the Stata Center.

More Jack-O-Lantern photos, including great ones of Chomsky and Norvin Richards, are up on Flickr.

Related posts brought to you by Yet Another Related Posts Plugin.

After the Deadline for Firefox

mitcho — Tue, 02 Feb 2010 01:02:54 +0000

After the Deadline is a powerful and intelligent proofreading tool which checks for spelling errors, misused words, some grammatical gaffes, and even some stylistic issues. For the past month, I’ve been working for Automattic, the company behind AtD and the makers of WordPress.com, to create a Firefox add-on which enables this superior technology everywhere on the web. Words can’t do justice to the magic that is AtD, so here’s a video we put together:

I invite you all to give it a spin:

Working on After the Deadline for Firefox gave me my first experience creating an add-on from the ground up and I’ve learned a lot. After working on Ubiquity and dabbling with Jetpack, it’s given me another perspective on extensibility on the web and I look forward to thinking and writing more about these experiences in the near future.

In the mean time, happy proofreading!

Related posts brought to you by Yet Another Related Posts Plugin.

Light of Firefox (tomoshibi 灯) from Mozilla Japan

mitcho — Tue, 30 Jun 2009 14:58:54 +0000

Here at Mozilla Japan Firefox 3.5 Headquarters,¹ we just launched the new and improved Light of Firefox (in Japanese, tomoshibi (灯)) for Firefox 3.5. The Light of Firefox is a real-time, interactive website which shows sparks on a map of Japan for every manual download of the new Firefox from mozilla.jp.

The name tomoshibi means “torch” in Japanese. As a new Firefox brings new technologies and possibilities to all corners of the web, so too will the tomoshibi light up the night in Japan!

The site was designed and coded by Daisuke Akatsuka of the Keio Kakehi Lab (xlab), the same fine folks who brought you interFORest.

Related posts brought to you by Yet Another Related Posts Plugin.

Ten Grand Is Buried There

mitcho — Wed, 17 Jun 2009 18:58:07 +0000

By now many of you have probably seen this new Microsoft Australia campaign, “Ten Grand Is Buried Here.com,”¹ which calls Firefox “old” and Safari “boring”:

I’m not sure what this is saying about me, but my immediate reaction was to go check whether tengrandisburiedthere.com was available. To my surprise, Microsoft had yet to snatch it up! A few hours later, here’s the result:

Go ahead, visit the site and give it a try!

Note: Not being a marketing guy, I just threw some text together to introduce Firefox. If someone has some better copy for this display, please let me know.

As of this writing, this domain actually has yet to serve anything. ↩

Related posts brought to you by Yet Another Related Posts Plugin.

Ubiquity in Firefox: Focus on Japanese

mitcho — Fri, 20 Feb 2009 11:08:14 +0000

One of the eventual goals of the Ubiquity project is to bring some of its functionality and ideas to Firefox proper. To this end, Aza has been exploring some possible options for what that would look like (round 1, round 2). All of his mockups, however, use English examples. I’m going to start exploring what Ubiquity in Firefox might look like in different kinds of languages. Let’s kick this off with my mother tongue, Japanese.¹

今後多様な言語に対応したFirefox内のUbiquityを検討していきますが、その中でも今日は日本語をとりあげます。後日日本語で同じ内容を投稿するつもりです。^^ 日本語でのコメントも大歓迎です！

What commands look like in Japanese

Japanese is not only just a verb-final language but it is strongly head-final, meaning it has postpositions instead of prepositions, direct objects come before verbs, and adjectives precede nouns. In terms of how it identifies its arguments, every argument has a postposition/case marker (called a particle in the Japanese literature) which marks its role in the sentence.

A couple common particles we’ll look at in this example include -を (-o) which marks the direct object (accusative case, you might say) and -に (-ni) which acts like English “to” (dative case). The example sentence we’ll look at today is:

ケーキを	ブレアに	送って	(ください)
kēki-o	burea-ni	okuʔte	kudasai
cake.ACC	Blair.DAT	send.IMP	“please”
“Please send a cake to Blair.”

(Note: ʔ is a glottal stop. ACC=accusative, DAT=dative, and IMP=imperative form.)

That final ください is often dropped in very casual speech and, as it adds no new information, we’ll assume today that the user will not enter it. Finally, Japanese doesn’t use spaces in their orthography, so the actual input would be “ケーキをブレアに送って”.

Mockup 1: Particle identification

One of the major hurdles in working with Japanese is that there are no spaces between the words. The natural first step is to split the sentence up into words, but this is a very difficult problem in NLP which big name research groups actively work on.

Fortunately, however, in “Solving the ‘It’ Problem” Aza suggests that, when we encounter ambiguity in our input, we can go ask the user. Great minds think alike, and computer scientist Jean E. Sammet suggested the same idea way back in 1953:

Using English [or any other natural language] definitely involves the requirement for the computer (or more accurately its programming system) to query the user about any possible ambiguity.

Parsing a sentence into words, in the limited context of Ubiquity, is really about identifying the particles which mark the end of each argument. Here’s a mockup of an application of the Sammet-Raskin Method to this problem:

Pros: This completely takes care of the word-breaking problem, with minimal arbitration from the user. The parser knows exactly what arguments it’s dealing with and the visual feedback means the user won’t be surprised by the parse.

Cons: Most of the particles/postpositions we’d have to deal with are a single character, so they may show up pretty often within words, in which case it would be quite annoying to have to press escape after each one.

An even smarter system, when wanting to mark a character as a particle, would first check to see that the argument (before the particle) is a valid argument type for that particle. If the check fails, it doesn’t have to bother with suggesting that character as a particle. This may cut down on the false positives.

Smart suggestions: what works, what doesn’t

One of the key suggestions in Aza’s mockups include a way to choose the prepositions while entering your arguments, based on the current verb.

For example, here, the translate command accepts a direct object, a to-object, and a from-object, so little to and from markers magically show up on the right side, making the appropriate prepositions (and by extension the appropriate arguments) discoverable. I think this line of thinking is a really good one, at least for English.

In a verb-final language, however, you enter the arguments first and then the verb, making this strategy of suggesting appropriate arguments impossible. Note that in the user-contributed spreadsheet of how languages identify their arguments we see that about a quarter of the languages we looked at are verb-final—that is, with Subject-Object-Verb canonical word order.

Instead of seeing this as a disadvantage, however, let’s see what verb-final order allows us to do.

Mockup 2: A different kind of suggestion

Not all verbs allow for every different kind of particle. For example, it doesn’t make sense to have a -に (-ni, “to” or dative) argument for a verb like 検索して (kensaku-shite, “search for”). In English we used this to suggest different types of arguments given a specific verb. In a verb-final language, we could do this backwards.

Pros: This makes verbs highly discoverable, given a certain argument structure. For example, if you enter a few arguments, like a direct object, a “to” argument, and a “from” argument, it’ll suggest verbs that will do something to an object from somewhere to somewhere else. This way, you can easily try out verbs you didn’t even know existed. It’ll only give you verbs appropriate for your arguments, reducing the chance of writing a an infelicitous command.

Cons: Without knowing what kinds of actions are available, it may be difficult to know what kinds of arguments to enter in the first place. If you have a specific verb or service you want to use it may be counterintuitive or downright tricky to start by guessing the right set of arguments.

In addition, from a technical point of view, this requires much of the prediction algorithms in English Ubiquity to run backwards. Ideally, there would be a closed (predetermined) class of particles and a predefined set of noun types. Verbs would not be able to define their own modifiers and noun classes as easily or freely as they can now.

Conclusion

The properties and challenges of Japanese grammar require that we not try to outright copy the English behavior but to think about what really makes sense in that language and that may be an important lesson as we move toward designing a localizable Ubiquity. Please post your questions and criticisms of this design or post your own mockups!

Happy International Mother Language Day! ^^ ↩

Related posts brought to you by Yet Another Related Posts Plugin.

selecting from Ubiquity

mitcho — Wed, 07 Jan 2009 15:50:13 +0000

I created a new Ubiquity command that gives you super fast access to your MySQL database’s data: select.

select Command for Ubiquity from mitcho on Vimeo.

Get the command and set it up here.

External orders in WordPress queries

Related posts brought to you by Yet Another Related Posts Plugin.