blog

On January 2nd, 2008, from the rural town in eastern Taiwan where I was then working, I committed version 1.0 of the Yet Another Related Posts Plugin (YARPP) to the WordPress plugins repository. Since then, YARPP has been downloaded almost one million times and has evolved greatly. It also has been the impetus for my further involvement in the WordPress open source community.

I believe the success of YARPP comes from its simple premise: display related content to visitors and they will stay on your site longer. Your visitors learn more about you and they in turn find more compelling content that they’re interested in.

One oft requested feature for YARPP has been the ability to display related posts from across multiple sites. I’ve thought about what this would involve technically—the infrastructure, the algorithms, the queries—and have chosen not to go down this route. To do so would require a serious investment of time and resources, which doesn’t make sense for me as I continue to be in grad school.

In January, though, I met the team at BlogGlue while in Phoenix for WordCamp Phoenix. It was refreshing to meet a team who has really thought through the technical aspects of calculating “relatedness” of content, as well as the potential social and economic impact that this type of related content suggestion can have. They’ve been working in this space for a couple years and have a patented, scalable algorithm for computing “relatedness” of content from across thousands of different sites. When you sign up with BlogGlue, it displays “related links” on your blog post, just like YARPP does, but from your own site as well as from other content sources that you recommend. In turn, your blog posts may show up on your partners’ sites as well.

What surprised me during that visit was some of the statistics they had gathered from their users: not only does BlogGlue bring in traffic to your site, visitors who come in via the BlogGlue network stay on your site 3–4 times longer than visitors from Google and view 2–4 times as many pages. This makes complete sense to me based on my experience with YARPP: people want to see content that is relevant to them. A search query is one way to find content you’re interested in, but it’s imperfect. With related posts across a network, the system can find related content based on all the content on the current page, which is a much richer “cue” than a simple search query would ever be.

Today I’m pleased to announce that YARPP has officially partnered with BlogGlue. Think of BlogGlue as “YARPP Pro”: it’s the multi-site, social, next generation of YARPP. YARPP helps your visitors discover more of your own content once they’re on your site—BlogGlue additionally helps more people discover your content in the first place, in a rich, meaningful way.

YARPP will of course continue to exist and be supported by me. BlogGlue and I will be advising each other on technical aspects of our offerings, pushing both products forward, as well as cross-promoting our offerings. The latest version of YARPP, 3.3, already has some UI simplifications which came out of a conversation with the BlogGlue team. I’m looking forward to working with them more in the future and I urge you to try out BlogGlue on your own site as well.

One of the last bugs for Firefox Panorama was bug 625818: “Check Panorama mochitest test suite coverage”. Our automated tests ensure that we do not regress on existing functionality, but it’s only as good as its coverage: how much of the Panorama code base is actually being “hit” through the process of running the test suite.

Panorama went through a pretty rapid development cycle, making it into Firefox 4 which was released today (yay!). Moreover, for a while we were developing outside of mozilla-central, without the regular “patches require tests” requirement. This makes checking its test coverage particularly important.

Check out the final result, the Panorama test coverage report. The good news: our code coverage is 86%! (Some notes on what improvements can be made are in the bug.)

PhiliKON had previously worked on hooking into the JS Debugger service’s interruptHook to test xpcshell tests. I modified this code to run instead in the Mochitest browser chrome tests. This code can be found on the bug.

With this patch applied, I invoked the test suite with the following code: TEST_PATH=browser/base/content/tests/tabview COVERAGE_FILTER="*tabview*" COVERAGE=true make -C obj-ff-dbg mochitest-browser-chrome . That’s a regular mochitest-browser-chrome invocation with the COVERAGE=true flag which turns on code coverage checking, and COVERAGE_FILTER=*tabview* which filters out results from files which don’t have “tabview” in their paths. This creates a file called coverage.json in the working directory of the test suite, meaning, for me, obj-ff-dbg/_tests/testing/mochitest/.

This JSON file is a multidimensional array, with file paths and then line numbers as keys. The file paths here, as best as possible, have been converted into local filesystem paths. PhiliKON built a script which produces beautiful reports based on this output.

A word of warning: running with this JSD interruptHook is ridiculously slow. A number of tests for Panorama are timing-dependent (drag-drop tests, for example), making some of them fail, but that’s okay… as long as it completed not via a timeout, it actually did run through all the code. In order to get this to run through everything with some degree of control, I split up the mochitest tabview suite in to a few chunks. I then took the multiple resulting coverage.json files and passed them into another script, in tools/coverage/aggregate.py, which takes multiple JSON results like this and puts them together into a single JSON file. I then passed this aggregate JSON file to PhiliKON’s wonderful report script and—voila—the Panorama test coverage report! Easy as pie.

Happy Halloween from the Firefox Panorama team!

We carved some pumpkins a couple days ago in my department. I carved the Panorama logo above, but also one of the Stata Center.

More Jack-O-Lantern photos, including great ones of Chomsky and Norvin Richards, are up on Flickr.

Every website has a purpose. Maybe you want people to buy a product, donate to your cause, download your app, or subscribe to your mailing list. How can you confidently modify your site to make it more effective with respect to this goal?

A/B testing is a process by which multiple variants of a website are presented to different users randomly and statistical tools are used to see whether any variant is more effective, according to an overall goal metric such as conversions or revenue.

While various A/B testing products—many free—exist, none are made from the ground up to work within the WordPress ecosystem. I believe a solution made particularly for WordPress could make A/B testing so much easier and more straightforward, and that such a solution could be greatly beneficial to the platform as a whole.

I’m happy to announce my new project, code-named ShrimpTest,¹ which is directly aimed at filling this void. I’ll be working on this project this summer together with the fantastic folks at Automattic.

The best way to keep up with development is on the project’s development blog, the ShrimpTest P2. Most updates will most likely be much shorter than this initial post. You can get less frequent, milestone-like updates by following ShrimpTest on twitter. Development will be open so feel free to check it out (haha) and submit patches as well. As I go along, I’ll also look forward to your feedback.

Everyone I know in linguistics uses the LINGUIST List website to a greater or lesser degree. Linguist List began as a mailing list in the 90’s, with book, job, and dissertation announcements, call-for-papers, and general academic discussions.

Nowadays many people follow the various announcements on Linguist List using an RSS feed reader such as Google Reader or my personal favorite NetNewsWire.

Unfortunately, the Linguist List RSS feeds (at least recently) don’t include the full text of the articles and have a few other quirks as well. It’s often hard to judge based on the title whether it’s really something I’m interested or not, so I’ve spent a lot of time frustratedly opening any possibly interesting-looking entry in a separate NetNewsWire tab. Today I decided enough was enough: I just wrote a script which parses each of the Linguist List RSS feeds, finds the actual descriptions and interleaves them.¹ It’s working remarkably well so far:

(more…)

I recently confirmed three different very exciting speaking gigs which I’ll be doing this spring:

(more…)

A couple weeks ago I gave a talk at the Boston Javascript meetup introducing Jetpack and filling people in in the latest developments in the project, including the Reboot. Between 20 to 30 people came to the talk which was at Microsoft Cambridge. Here are the slides from the talk:¹

Extend the Browser with Jetpack

(more…)

A couple weeks ago I went out to Mozilla HQ in Mountain View for a Jetpack Ambassador meetup. Jetpack is a project at Mozilla labs intended to make writing Firefox add-ons easier, and shares some ancestry with the Ubiquity project dear to my heart. The Jetpack Ambassadors are a team of Mozilla community members who will be involved with Jetpack marketing, evangelizing Jetpack and writing about our own experiences working with the exciting new Jetpack architecture.

We spent a good chunk of time with a team from Invisible Elephant who came in to give us some training on making technical presentations, and then dug into the code on Day 2. It was great to have the geniuses at Mozilla Labs like Atul and Myk show us the latest Jetpack code as well as get the latest project direction from Daniel, Aza, and Nick, from which we could see the amount of careful consideration and effort that’s gone into the Jetpack reboot.

The best part of the whole experience, though, has to be the fellowship with the other Jetpack Ambassadors. The Ambassadors came from all over the world, encompassing Europe, Asia, S. America, and of course N. America. Each are involved with some really exciting projects and have each made a name for themselves in their respective communities. I’ve put together a twitter list of all the Jetpack Ambassadors and the core team members and invite you to follow them.

We also had the greatest number of Ubiquity core developers to have ever been in the same place at the same time, which of course had to be documented.

(More photos can be seen in my gallery.)

I had a fantastic time in MV and it was a shame I could only be there for such a short time. I feel honored to be a part of this group and am looking forward to speaking on Jetpack soon at an event near you!

After the Deadline is a powerful and intelligent proofreading tool which checks for spelling errors, misused words, some grammatical gaffes, and even some stylistic issues. For the past month, I’ve been working for Automattic, the company behind AtD and the makers of WordPress.com, to create a Firefox add-on which enables this superior technology everywhere on the web. Words can’t do justice to the magic that is AtD, so here’s a video we put together:

I invite you all to give it a spin:

Working on After the Deadline for Firefox gave me my first experience creating an add-on from the ground up and I’ve learned a lot. After working on Ubiquity and dabbling with Jetpack, it’s given me another perspective on extensibility on the web and I look forward to thinking and writing more about these experiences in the near future.

In the mean time, happy proofreading!

In working on a new website for the MIT Working Papers in Linguistics, I recently inherited a collection of HTML files with all of our books’ abstracts. To my dismay (but not surprise) the markup in these files were horrendous. Here are some of the cardinal sins of markup that I saw committed in these files:

1. Confusing ids and classes. ids should be unique on the page… but here’s an instance of using multiple instances of the same id in order to format them together.
   

<div id="indent"> <div id="number">4.2.1</div> <div id="page">161</div> <div id="section">Old French (Adams 1987)</div>
</div> <div id="indent"> <div id="number">4.2.2</div> <div id="page">164</div> <div id="section">The evolution of the dialects of northern Italy</div>

1. Putting a class on every instance of something. Everything paragraph should be formatted equivalently. We get the point.
   

<p class=MsoNormal><b>The English Noun Phrase in Its Sentential Aspect</b></p>
<p class=MsoNormal>Steven Paul Abney</p>
<p class=MsoNormal>May 1987</p>

1. Using blank space for formatting.
   

<p class=MsoNormal><o:p>&amp;nbsp;</o:p></p>

1. CSS styles that don’t exist. Browsers just ignore these anyway…
   

<p class=MsoNormal>One factor in determining which worlds a modal quantifies
over is the temporal argument of the modal’s accessibility relation.<span
style='mso-spacerun:yes'>  </span>It is well-known that a higher tense affects
the accessibility relation of modals.<span style='mso-spacerun:yes'> 
</span>What is not well-known is that there are aspectual operators high enough
to affect the accessibility relation of modals.<span style='mso-spacerun:yes'> 
</span&gt

The solution

My solution was to write a perl script which takes care of a number of these issues. It’s not foolproof and doesn’t involve any voodoo—for example, it can’t retypeset things which were formatted using whitespace—but it does a good job as a first pass.

You can run the script by making it executable (chmod +x cleanwordhtml.pl) then specifying a target filename as an argument. For example,

./cleanwordhtml.pl source.html > clean.html

I used this with a simple bash for loop to run over all my files:

for f in */*.html; do ./cleanwordhtml.pl $f > ${f%.html}-clean.html; done;

Hopefully someone else can benefit from my experience.

Last week I was invited to give a talk at the TechMaine annual conference in Portland, Maine.

Being a longer time slot than I previously have used to talk about Ubiquity, I decided to dedicate a good portion of the talk to Jetpack. Being outside of Mozilla for the past few months, this gave me an opportunity to get reacquainted with the Jetpack APIs. I myself was impressed by how easy it was to develop a quick Jetpack. I ended up preparing two to live-code during the talk: one called Helvetica which, with one click, replaces all fonts on the current page with Helvetica; and You Are Here which uses an open API from IPinfoDB to display the physical location of the domain you are currently visiting in the status bar. Both are now on the Jetpack Gallery.

Unfortunately there was a bit of a snowstorm leading up to the event, but there was still a nice turnout and I got to meet some fantastic people there. Ken Shoemake of slerp and quaternion fame came up to me after my talk and said “the Ubiquity parser reminded me of the dancing bear… it’s less surprising that it works well as that it works at all.” I also enjoyed the other great presentations in the technology track, covering the virtues of REST and basic iPhone development.

Mashup the Browser with Ubiquity and Jetpack

Yesterday I gave a talk at the Boston WordPress Meetup. The Boston WordPress Meetup meets monthly at the Microsoft Cambridge Research Center which is a fantastic venue right on the Charles river. Last night we got to be up on the 10th floor which has a great view of Boston right over the river. There was pretty good turnout, with about thirty or fourty people there.

My talk was a general introduction to WordPress plugin development, beginning with the concepts of actions and filters, and concluding with a description of HookPress, my new plugin which enables webhooks in WordPress. Here are the slides:

(more…)

About six months ago I started working for Mozilla Labs full-time, focusing on Ubiquity, the multilingual natural language interface for the browser. This week marked my last week on contract as I go back to grad school next week. While the work will go on and I hope to continue to stay involved as time allows, here’s a quick bird’s eye view of my activities in my Mozilla tenure:

Time working for Mozilla: 6.5 months

Mozilla-related blog posts written: 69

Academic papers written on Ubiquity: 1

Ubiquity presentations given: 5

Screencasts made: 8

Most popular video on Vimeo: Ubiquity 0.5 日本語紹介ビデオ, the Japanese Ubiquity 0.5 introduction video: 2252 views

Languages Ubiquity commands and parser now support: 6

Commits to the Ubiquity repository: 492

Other web projects started during this period: 2+ (Ten Grand Is Buried There, HookPress)

TechCrunch references: 2 (1, 2)

Countries worked in: 2

Mythical Kiwis worked with: 1

References to bugs I introduced as “glitcho”s: 1

Extremely disturbing homages to me and Django: 1

Friends made; experience gained; lessons on Open-ness learned; personal growth: priceless enumerable

Thanks to all who made this experience amazing, beginning with Aza, Jono, Atul, Blair and the rest of the Labs team; intern extraordinaire Brandon; the always thoughtful and friendly Mozilla Japan team; and of course the fantastic Ubiquity community! Please visit me in Boston—I should be around for a while.

This week I implemented a new way to customize and extend Ubiquity commands: CmdUtils.CreateAlias.

The use case for and importance of CreateAlias

CreateAlias lets you easily create a special-case alias of another, more generic verb. Ubiquity comes bundled with useful verbs like translate and search which can be used for a number of different uses based on their arguments. In some cases, and in some languages, though, typing out translate to English or search with Google is unnatural, though, as there is a more succinct and direct way to make that request. For example, in English one could say “anglicize” or “google”, respectively, for the verbs and arguments above. Indeed, in order to support both search with Google and google, Ubiquity traditionally has implemented two different verbs, search and google, which duplicate functionality and code.

CreateAlias lets us create such natural aliases without repeating ourselves. We can easily create an anglicize verb which, in one word, does the work of translate to English, or google which is semantically equivalent to search with Google.

These sorts of aliases become particularly important in our perpetual quest to internationalize Ubiquity. One discussion that came up early on on our Ubiquity-i18n list is the fact that not all languages have the verb “Google”: in many languages it is necessary to explicitly say “search with Google”. Moreover, other languages may have other domain-specific verbs which English doesn’t have either. Maybe some language has a special verb for “email with Hotmail” or “map Denmark”. Who knows? With CreateAlias we can easily enable such localizations based on the more generic commands bundled with Ubiquity.

Creating an alias

CreateAlias was designed to be incredibly simple to use. Here’s an example that will be bundled (but not installed by default) in Ubiquity:

CmdUtils.CreateAlias({
  names: ["anglicize"],
  verb: "translate",
  givenArgs: { goal: "English" }
});

As you see, this syntax is incredibly straightforward. There are two required properties, names, an array of names for the alias, and verb, a reference to the target verb that this alias should use.¹

The alias can also have a givenArgs property which is a hash of pre-specified arguments with their semantic roles. Because translate takes three arguments (an object text, a goal language, and a source language) but we have pre-specified a goal in the givenArgs, the new anglicize command will only take two arguments: the object text and a source language. Of course, if you specify no givenArgs, you’ll get a simple synonym without having access to the original verb’s code.

As you see, the preview of this command is simply the preview of the translate verb. Its preview and execution is just as if you had entered translate こんにちは to English.

Just like other commands created with CreateCommand, the object specifying the alias can also have properties like help, description, author information, and so on. I used the icon property to add a Union Jack to it so that it was easily identifiable.

Bonus: using CmdUtils.previewCommand and CmdUtils.executeCommand

On the road to implementing CreateAlias, I also implemented the CmdUtils.previewCommand and CmdUtils.executeCommand functions. The majority of this code comes from previous work by Louis-Rémi Babé, though I adapted it to the modern Ubiquity system. Using previewCommand and executeCommand you can take advantage of the preview or execute functionality of another command. In the new alias-commands feed I included a command called germanize which essentially is a straightforward analogy to anglicize, seen above, but using these functions within a CreateCommand. While CreateAlias is much more straightforward for simple aliases, for more complex subcommands where you would like to adapt another verb’s execution or preview, or only take one of those but re-implement the other part, previewCommand and executeCommand are the way to go.

Introduction

I recently have begun giving serious thought to what command chaining might look like in Ubiquity and the various considerations which must be made to make it happen. The “command chaining,” or “piping,” described here always involves (at least) two verbs acting sequentially on a passed target—that is, the first command performs some action or lookup and the second command acts on the first command’s output.

A few days ago I penned some initial technical considerations regarding command chaining. In this post I’ll be point out some linguistic considerations involved in supporting a natural syntax for chaining.

(more…)