mitcho Michael 芳貴 Erlewine

Postdoctoral fellow, McGill Linguistics.


Checking mochitest test coverage

Firefox Download ButtonOne of the last bugs for Firefox Panorama was bug 625818: “Check Panorama mochitest test suite coverage”. Our automated tests ensure that we do not regress on existing functionality, but it’s only as good as its coverage: how much of the Panorama code base is actually being “hit” through the process of running the test suite.

Panorama went through a pretty rapid development cycle, making it into Firefox 4 which was released today (yay!). Moreover, for a while we were developing outside of mozilla-central, without the regular “patches require tests” requirement. This makes checking its test coverage particularly important.

Check out the final result, the Panorama test coverage report. The good news: our code coverage is 86%! (Some notes on what improvements can be made are in the bug.)

code coverage report

PhiliKON had previously worked on hooking into the JS Debugger service’s interruptHook to test xpcshell tests. I modified this code to run instead in the Mochitest browser chrome tests. This code can be found on the bug.

With this patch applied, I invoked the test suite with the following code: TEST_PATH=browser/base/content/tests/tabview COVERAGE_FILTER="*tabview*" COVERAGE=true make -C obj-ff-dbg mochitest-browser-chrome . That’s a regular mochitest-browser-chrome invocation with the COVERAGE=true flag which turns on code coverage checking, and COVERAGE_FILTER=*tabview* which filters out results from files which don’t have “tabview” in their paths. This creates a file called coverage.json in the working directory of the test suite, meaning, for me, obj-ff-dbg/_tests/testing/mochitest/.

This JSON file is a multidimensional array, with file paths and then line numbers as keys. The file paths here, as best as possible, have been converted into local filesystem paths. PhiliKON built a script which produces beautiful reports based on this output.

A word of warning: running with this JSD interruptHook is ridiculously slow. A number of tests for Panorama are timing-dependent (drag-drop tests, for example), making some of them fail, but that’s okay… as long as it completed not via a timeout, it actually did run through all the code. In order to get this to run through everything with some degree of control, I split up the mochitest tabview suite in to a few chunks. I then took the multiple resulting coverage.json files and passed them into another script, in tools/coverage/, which takes multiple JSON results like this and puts them together into a single JSON file. I then passed this aggregate JSON file to PhiliKON’s wonderful report script and—voila—the Panorama test coverage report! Easy as pie.

Tags: , , , ,

If you enjoyed this post, make sure you subscribe to my RSS feed (optionally with tweets from my Twitter)!

  • sfink

    The instrumentation in might help with the speed problem, though it’s not done yet and the output format isn’t really what you want. It accumulates the execution count for every opcode, broken down by which engine executed it (interpreter vs tracejit vs methodjit). The interruptHook disables the tracing JIT and is expensive besides; this instrumentation slows down all execution but doesn’t disable anything and should be much less overhead overall. (interruptHook is doing a full function call per op; bug 637393 executes a couple of additional machine code instructions per op.)

    The patch dumps out a disassembly of every JSScript (function or toplevel) on JSContext teardown, together with per-op execution counts and line numbers. So you could massage that into a coverage report. (Sum of all ops executed in a line > 0?) Or maybe better, dump out your own condensed report, though that’ll require trawling through src notes which isn’t all that fun.

    I’ve never tried it on a full browser build. You’d need to arrange for the JSOPTION_PCPROFILE bit to be set on all contexts you care about.

    It’s probably not baked enough to use for this yet, but any comments/suggestions in the bug will be taken seriously. (Or if you want to take it over, go ahead!)

  • Philipp von Weitershausen

    Wow, I’m happy you were able to use my code so smoothly. This is awesome, I think I’ll be using this patch to mochitests in our upcoming projects :D OMG THIS OPEN SOURCE THING WORKS

    On a side note, I’m still working on a general way of instrumenting SpiderMonkey to provide us with coverage. Since the approach here doesn’t work with xpcshell-tests (there’ll always be JS on the stack), we need a generic solution anyway.