Today I hope to cover some of the technical considerations required in implementing command chaining in Ubiquity, and I will follow up soon with a blog post on the linguistic considerations required as well.

Technical considerations: hooking the pipes together

I’d first like to lay out some technical challenges and questions. These can be broken into two different categories: (1) how the parse and display of suggestions is affected and (2) how the execution is affected.

Matching inputs and outputs

We’ll first consider how command chaining may affect the parsing. Ubiquity commands each specify the types of argument inputs that it expects using different noun types, such as noun_arb_text which accepts anything, noun_type_number which accepts numbers, or noun_type_language, which takes the name of a language. For example, the translate verb takes maximally three arguments: a noun_arb_text object, a noun_type_language goal (the language to translate into), and a noun_type_language source (the source language). In implementing command chaining, it will be necessary to identify the appropriate noun types for the output of a command.

The first question we must address here is “what is the chaining output of a command”? Is it the preview text? Some text output from the execution?

Big fish eat da lil fish by joemud, CC-SA-NC

To put this question into perspective, we note that Ubiquity commands can be broadly classified into two types: lookup and action execution. Here’s a classification which I believe to be exhaustive:²

In light of this classification I believe we can say that lookup commands are much more likely to be the first verb in a command chain—conversely, chains such as email hello to Blair and then do ... or twitter hello and then ... are quite unlikely. What is much more likely is for the first verb to be a lookup function.

Thus in the same way that not all commands have a useful execution³ perhaps only lookup commands will have a chainable output: the results of the lookup. Even with this restriction, we will most likely need to implement a new “chainable output” method or getter in these commands. This means that commands will need to opt-in to become chainable, but I believe this is a necessary evil.

The second question we must address is “when do we establish the noun type of a command’s chainable output?” One unsung but crucial feature of the way Ubiquity works now is that suggestions’ previews are not computed until that suggestion is selected (except for the first suggestion, which in most skins gets previewed immediately). Should we wait for all of the first verbs’ chainable output to be computed and then run them through the noun type detection system? Or should verbs with chainable output also a priori specify what noun types their output will be?

Both of these approaches have their problems. If we compute the chainable output of the first verb, run a noun type detection on it and then suggest the full combination if it matches what the second verb was expecting, this will have clear performance implications, not to mention that it could greatly complicate our parsing algorithm. While the latter approach doesn’t have these performance implications, it does mean that it will have to list (by name or reference) the noun types that will match its output, meaning that if a command author is unaware of someone else’s noun type, that chain will be impossible, even if the chainable output itself does indeed match that noun type. The a posteriori approach would never have this issue. What other benefits or problems do you forsee with either of these approaches? Is there another approach which avoids these pitfalls?

(A)synchronous composability

Once we have the noun types, parsing, and suggestions down, all that remains is to compute the previews and implement the composite execution. Since the Ubiquity command manager already wraps the preview and execute functions in a wrapper to facilitate localization, among other uses, it would be easy to make the command manager compose asynchronous processes pseudo-synchronously. No major changes should be necessary to do the previews and executions, though, again, there will be a performance cost.

Conclusion

There are a number of technical questions which must be answered, mostly in the parsing/suggesting stage. The key questions to answer are:

1. What is the chaining output of a command?
2. When do we establish the noun type of a command’s chainable output?

I’ll make another post soon on the linguistic considerations necessary in making command chaining happen in a natural fashion.

Related posts:

1. Command Chaining with Oni?
2. Ubiquity Commands by The Numbers
3. Ubiquity Parser: The Next Generation Demo

Related posts brought to you by Yet Another Related Posts Plugin.

It’s time for one more installment of Nountype Quirks, where I review and tweak Ubiquity’s built-in nountypes. For an introduction to this effort, please read Judging Noun Types and my updates from Day 1 and Day 2.

Today I ended up spending most of the day attempting to implement (but not yet completing) major improvements to the geolocation-related nountypes whose plans I lay out here.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

noun_type_geolocation

noun_type_geolocation is the nountype used by the weather command for its location argument in input like “weather near Chicago”. The neat feature of noun_type_geolocation is that it has a smart default value which uses Firefox’s geolocation system to give you your current location by default, so I can enter “weather” and get the suggestion “weather near Broomfield, Colorado” (not completely correct, but close enough for the weather). Otherwise, however, noun_type_geolocation does not do too hot… for any input you give it, it’ll just accept it with a score of 0.3, much like noun_arb_text. We could do better.

One issue with this noun_type_geolocation is a conceptual one. Is this nountype supposed to accept only municipalities? Countries? Or should it accept landmarks or addresses as well? Part of the issue is that it’s only used by one built-in command in Ubiquity now, weather. But to be called a general “geolocation” nountype, its output should not be specific to weather’s usage, which is to throw the result at the Weather Underground API.

I propose that we change this to be something like noun_type_geo_town and also make similar nountypes like noun_type_geo_country, noun_type_geo_region, going all the way down to noun_type_address (which already exists—see below). All of the nountypes in this family could use a geocoding API such as Google’s or Yahoo’s. Their data properties could include all of this geocoded geographic data (in English) and also the latitude/longitude coordinate data.

The weather command could then accept noun_type_geo_town but, as some municipalities are not in Weather Underground or, for some countries, it is only as granular as administrative districts, we could just display the results of the geocoding API but then give Weather Underground the geocoded latitude/longitude data.

noun_type_async_address

noun_type_async_address attempts to do exactly what I’ve laid out above for the most granular level: that of geolocations with data all the way down to the street level. This is the nountype which is used for the built-in map command and uses the Yahoo geocoding service to accomplish this. Let’s see what kinds of results it returns:

input	suggestion
mitcho	mitcho	0.5
grenada	grenada	0.9
jono	jono	0.9
mountain view	mountain view	0.9

Let’s lay out some immediate quirks:

1. All scores are either 0.5 or 0.9. In general, if the Yahoo API returns some geocoded interpretation, it gets 0.9, but otherwise it accepts everything with 0.5.
2. The results that came back from the Yahoo service doesn’t add any useful information like the country or administrative region. Even the case stays lowercase.
3. Since when is Jono a location!? I’ll get back to this later.

For starters, the Yahoo! Maps API terms of service dictate that we can’t use its geocoding service if we’re not also displaying Yahoo maps, so I rewrote it using the Google API which also had the advantage of offering JSON output.

One quirk of the Google Geocoding API, though, is that all of the resulting municipality names are only in English. Try for example queries for Wien or 東京 (Tokyo). Since we want our suggestions to only add information to our input, not replace the input entirely (and especially not in another language), we’ll then only take results which have the input as an initial substring. On the other hand, if none of the results have the input as a proper prefix of the return value, we will take the geocoding information from the first result but with the original input as the display text. Such results will have a markedly lower score.¹

As this is the address nountype, we’ll penalize results which do not have detailed information such as street address or town-level information. All of this is very easy to judge as every result from the API has a geocoding accuracy value.

The best laid plans of mice and men…

I spent a good few hours this afternoon and evening attempting to implement this new family of nountypes, including this new nountype_geo_address, but also nountype_geo_subregion, nountype_geo_region, and nountype_geo_country. Some of the quirks of the weather and map commands, however, have prevented me from completely replacing the legacy noun_type_address and noun_type_geolocation described above. I hope to continue this work again soon and actually make this transition, ideally before 0.5.2.

Look forward to one (or maybe two?) more episode(s) of Nountype Quirks where I hope to definitively explain, analyze, and tweak matchScore, the scoring algorithm which underlies the majority of the nountypes in Ubiquity. As always, I look forward to your comments and feedback.

Bonus: Where’s Jono?

It turns out that noun_type_async_address was recognizing “Jono” as an address because Jono is actually a location afterall! Not only that, but Jono is in Japan!!

You clearly can’t take Japan out of Jono, but it turns out you can’t take Jono out of Japan either.

Related posts:

1. Nountype Quirks: Day 2
2. Nountype Quirks: Day 1
3. Judging Noun Types

Related posts brought to you by Yet Another Related Posts Plugin.

Today I’m continuing the process of reviewing and tweaking all of the nountypes built-in to Ubiquity. For a more respectable introduction to this endeavor, please read my blog post from a couple days ago, Judging Noun Types and my status update from yesterday, Nountype Quirks: Day 1.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

noun_type_twitter_user

Let’s begin again by considering the suggestions and scores that a variety of different inputs to this nountype return and see what quirks we find.

To test this nountype, I made sure I had logged into Twitter once with the login mitchoyoshitaka.

input	suggestion
mitcho	mitchoyoshitaka	0.85
	mitcho	0.5
mitchoyoshi	mitchoyoshitaka	0.94
	mitcho	0.5
test	test	0.5
テスト	none
hello world	none
@test	none

As nountypes go, this is looking pretty good. For usernames which look like logins we’ve saved before, we’re using matchScore to get decent differential scores.¹ It’s even ruling out impossible twitter username strings, according to Twitter’s own restriction:

One possible improvement we could make is to let @ strings be accepted. I went ahead and made this improvement. The initial @ will be stripped off and then will be checked as normal, but the final score will receive a slight boost using an nth root formula. The twitter command was also updated to deal with inputs with and without the initial @.

input	suggestion
mitcho	mitchoyoshitaka	0.85
	mitcho	0.5
@mitcho	@mitchoyoshitaka	0.88
	@mitcho	0.57
test	test	0.5
@test	@test	0.57

Although the noun_type_twitter_user nountype is currently most used by the built-in twitter command to specify the user’s username, in theory it could also be used for example in a command which pulls up another user’s tweets. With that in mind, perhaps in the future we could check the browser history and/or bookmarks for entries of the form http://twitter.com/... and suggest those as well (trac #846).

noun_type_number

input	suggestion
text	none
0.5	0.5	1
0.5.1	none

This nountype has an incredibly simple job and does it with ease. I’m going to leave it alone.

noun_type_date and noun_type_time

noun_type_date and noun_type_time both use the magical Date.parse method to parse date- and time-like strings. Let’s first take a look at some of its suggestions:

input	date suggestion	time suggestion
June 8th 5pm	2009-06-08	05:00 PM	1
5pm	2009-07-30	05:00 PM	1
5	2009-07-05	12:00 AM	1
June 8th	2009-06-08	12:00 AM	1
today	2009-07-30	12:00 AM	1
now	2009-07-30	02:40 PM	1
5pm is a good time	none	none

The quirks in these outputs can be summed up into these two factors:

1. There is no differential scoring at all.
2. Both nountypes parse the input with Date.parse and then just spit out the date or time components of the result. Thus time-only inputs get the default date and date-only inputs get the default time with equal scores.

I just rewrote both nountypes and also added a new noun_type_date_time. Here are some of the features of the new implementation:

1. If the input only contains digits and spaces, it is marked down.
2. With the exception of the outputs ‘today’ and ‘now’, if the resulting Date object’s date is today, its date suggestion is scored lower; equivalently for time being the default value, “12:00 AM”.
3. Scores (with the exception of ‘today’ and ‘now’) which are shorter than the output string get a slight penalty. This factor reflects the intuition that a longer output than input means some generic information was added and thus there is less confidence in the output.

Here’s what some of the inputs give now:

input	suggestion
June 8th 5pm	date: 2009-06-08	0.7
	time: 05:00 PM	0.7
	date_time: 2009-06-08 05:00 PM	0.86
5pm	date: 2009-07-30	0.27
	time: 05:00 PM	0.81
	date_time: 2009-07-30 05:00 PM	0.49
5	date: 2009-07-05	0.53
	time: 12:00 AM	0.19
	date_time: 2009-07-05 12:00 AM	0.34
June 8th	date: 2009-06-08	0.95
	time: 12:00 AM	0.35
	date_time: 2009-06-08 12:00 AM	0.58
today	date: 2009-07-30	1
	time: 12:00 AM	0.45
	date_time: 2009-06-08 12:00 AM	0.7
now	date: 2009-07-30	0.7
	time: 12:00 AM	1
	date_time: 2009-06-08 04:34 PM	1

In addition, looking to the future we’d like to make nountypes localizable as well, and these two nountypes in particular will surely require some good thinking and planning to make localizable.

noun_type_email and noun_type_contact

noun_type_email and noun_type_contact are two closely related nountypes. noun_type_email simply validates email address-looking strings, while noun_type_contact will return the noun_type_email suggestions and additionally return contacts from GMail if available.

The first thing to note is that I’ve often found the GMail contact lookup to be finicky in my own use. Reading through the code, I discovered the solution: GMail must either be open in a tab or you must use the “stay signed in” option and close the GMail tab.² With this mystery solved, and some code cleanup done to this contact fetching, let’s take a look at some example suggestions: (suggestions overlapping with noun_type_email are not listed here)

input	suggestion
aza@m	aza@mozilla.com	0.42
jono	jdicarlo@mozilla.com	0.28
jdicarlo	jdicarlo@mozilla.com	0.19

In general, we see that these scores all look pretty poor. In particular, though, note that the “jono” input yielded a higher score for the same suggestion than “jdicarlo”, even though “jdicarlo” is longer and thus, intuitively, has more informational content and should maybe do better. Digging into the code I realized why this is. It was computing the scores by comparing “jono” and “jdicarlo” not simply to “Jono DiCarlo” and “jdicarlo@mozilla.com” respectively, but to the combined string “Jono DiCarlo <jdicarlo@mozilla.com>”. Now with this change in place, both the email address and name are analyzed individually and, due to the way nountype detection works in Parser 2, no duplicates are returned. Here are the updated results:

input	suggestion
jono	jdicarlo@mozilla.com	0.83
jdicarlo	jdicarlo@mozilla.com	0.85

That’s much better!

Now let’s consider the suggestions from noun_type_email. Here are what they originally looked like:

input	suggestion
bpung	none
bpung@m	bpung@m	1
bpung@mozilla.com	bpung@mozilla.com	1

noun_type_email is based on a very robust regular expression for RFC 2822. Unfortunately this means that it completely rules out strings such as “bpung” which could be a proper prefix of an email address—something that I’ve advocated for avoiding before (see footnote 2 of Judging Noun Types). Moreover, due to a quirk of how nountypes based on regular expressions are scored, all results are given the score of 1.

I just committed a change so that this behavior is improved. The new version accepts strings which match the username part of the email address spec sans @ and domain, but with a great score penalty.³ Moreover, domains which do not have a final label (the top level domain) with more than one letter (unless it’s an IP address) or do not have any periods (.) in the domain will be penalized as well. Here’s what the same inputs produce now:

input	suggestion
bpung	bpung	0.3
bpung@m	bpung@m	0.8
bpung@mozilla.com	bpung@mozilla.com	1

Same time, same channel

I hope this post sheds light on the many changes I made together as well as the underlying thought process. If you don’t agree with any particular fix or analysis, please comment! I’ll be back again tomorrow with another installment of Nountype Quirks. Stay tuned!

Related posts:

1. Nountype Quirks: Day 1
2. Judging Noun Types
3. Big Issues and Small Issues with Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.

Today I began the process of going through all of the nountypes built-in to Ubiquity using the principles and criteria I laid out yesterday—a task I’ve had in planning for a while now. As I explained yesterday, improved suggestions and scoring from the built-in nountypes could directly translate to better and smarter suggestions, resulting in a better experience for all users. Here I’ll document some of the nountype quirks I’ve discovered so far and what remedy has been implemented or is planned.

Note: this blog post includes a number of graphs using HTML/CSS formatting. If you are reading this article through a feed reader or planet, I invite you to read it on my site.

noun_type_percentage

Here’s what a few different inputs originally returned:

input	suggestion
20	20%	1
20%	20%	1
0.2	20%	1
0.2%	20%	1
20.0	2000%	1
2 hens in the garden	2%	1

Let me highlight a couple obvious quirks:

1. In certain cases, where the numerical expression includes a decimal and is less than one, it is interpreted as a proportional, rather than percent, value, e.g. “0.2” → “20%”. “0.2%” is not even an option. This is the case even when explicitly adding a % sign.
2. All suggestions, including those where the numeral was extracted from a long string of text (e.g. “2 hens in the garden”), get the same score of 1.

I just committed a fix so noun_type_percentage now…

1. Counts the number of characters in the input which match [\d.%] and caps the score by (number of acceptable characters)/(length of input).
2. Strings which do not include “%” get a 10% penalty.
3. In the case of decimals less than 1 without a % sign, the proportion interpretation is also suggested (e.g. “0.2” → “20%”) in addition to the original suggestion (“0.2%”), but with a slight penalty.

Here is what they now return:

input	suggestion
20	20%	0.9
20%	20%	1
0.2	0.2%	0.9
	20%	0.81
0.2%	0.2%	1
20.0	20%	0.9
2 hens in the garden	2%	0.05

noun_type_tag

Here’s what a few different inputs originally returned. Keep in mind that currently in this test profile, the preexisting tags are “animal”, “help”, “test”, and “ubiquity”.

input	suggestion
animal	animal	0.3
mineral	mineral	0.3
anim	animal	0.7
	anim	0.3
help, test, ubiq	help,test,ubiquity	0.7
	help,test,ubiq	0.3
google, yahoo, ubiq	google,yahoo,ubiquity	0.7
	google,yahoo,ubiq	0.3
google, , yahoo	google,yahoo	0.3

Here are a few of noun_type_tag’s quirks:

1. There are only two scores ever given out: 0.3 and 0.7.
2. Only the last tag in the list and whether it exists or not is taken into account.
3. When the last tag is incomplete, the completion is suggested with a higher score, but if the last tag is exactly equal to an existing tag, it gets the lower score.

Ideally, we want noun_type_tag to look at each of the tags given to it, with higher scores for when there are more preexisting tags and fewer new ones. Keep in mind, though, that we only have to suggest the completion of the very last tag as that may be one where the user hasn’t completed typing yet… for earlier tags, we can assume (safely or not) that the user placed the comma where they meant to. We can’t teach Ubiquity to read minds, after all.¹

With this in mind, I just made a change to noun_type_tag which aims to follow these principles. The basic idea is that we start with a base score of 0.3 but then raise it via nth root for every tag in the sequence which is preexisting. Here’s what the same inputs return now. Recall that the preexisting tags are “animal”, “help”, “test”, and “ubiquity”.

input	suggestion
animal	animal	0.55
mineral	mineral	0.3
anim	animal	0.55
	anim	0.3
help, test, ubiq	help,test,ubiquity	0.86
	help,test,ubiq	0.74
google, yahoo, ubiq	google,yahoo,ubiquity	0.55
	google,yahoo,ubiq	0.3
google, , yahoo	google,yahoo	0.3

noun_type_awesomebar

input	suggestion
moz	http://www.mozilla.com/	0.8
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.8
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.8
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.8

There are a couple quirks here:

1. All suggestions are returned with the same scores.
2. The nountype returns the URL of the entry as the HTML-formatted result and the title as the text-formatted result, which clearly does not make sense. However, it’s not clear to me whether the title, URL, or some combination of both is what we should be returning as the suggestion text presented to the user.²

I just rewrote noun_type_awesomebar to actually do some differential scoring. This new version also presents the URL or title depending on whichever had a better match using the matchScore function.³

input	suggestion
moz	www.mozilla.com	0.7
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.63
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.61
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.6

noun_type_url

The purpose of noun_type_url’s suggest function is two-fold: first, to accept strings which may look like a URL and, second, to suggest URL’s from the history just like noun_type_url, but only based on URL matches and not title matches.⁴ Here are a few sample inputs:

input	suggestion
moz	http://www.mozilla.com/	0.9
	http://moz	0.5
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.9
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.9
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.9
test	http://test	0.5
http://	http://	0.5
http:	http:	0.5
http	http	0.5
_test	http://_test	0.5
hello world!	http://hello world!	0.5

Oh, where to begin!? Here are some initial quirks… it’s possible that you could think of more!

1. There is no differential scoring… only 0.9 for suggestions from history and 0.5 for URL-like strings.
2. A number of invalid domain names are being accepted and turned into suggestions (“hello world!”, “_test”, etc.).
3. It’s trying to be smart by suggesting “http://” as a default URI scheme but doing so even for prefixes (initial substrings) of the word “http” itself.

With these thoughts in mind, I just took a first stab at improving this situation. Here are some features of the new implementation:

1. History entries are scored in the same way as in noun_type_awesomebar, using matchScore.
2. URLs without an explicit URI scheme (like “http://”) get a 10% penalty.
3. “http://” is only suggested if one of a long list of common URI schemes are not detected.
4. It repairs schemes which are missing a slash or two, suggesting for example “http:hello.com” → “http://hello.com”.
5. It actually uses Firefox’s own IDNService to check if the domain name is a valid internationalized domain name. If it’s an IDN as opposed to LDH (“letters, digits, and hyphens”), it gets a 10% penalty. If it’s not even a valid IDN, it is ruled out (see last two example inputs below).
6. There are also penalties for only being a domain name with no path and for the domain not having any periods (.) in it.

Here is what our suggestions now look like:

input	suggestion
moz	http://www.mozilla.com/	0.6
	http://moz	0.65
	https://wiki.mozilla.org/Labs/Ubiquity/ Parser_2_API_Conversion_Tutorial	0.63
	http://en-us.start3.mozilla.com/ firefox?client=firefox-a&rls= org.mozilla:en-US:official	0.61
	http://en-us.www.mozilla.com/en-US/firefox/about/	0.6
test	http://test	0.65
http://	http://	1
	shttp://	0.75
http:	http://	0.9
	shttp://	0.7
http	http://	0.72
	https://	0.71
	shttp://	0.68
	http://http	0.65
_test	none
hello world!	none

See you tomorrow~

Alright, enough nountype wrangling for one day. I’ll be back again tomorrow for another installment.

Related posts:

1. Localizing Commands for Ubiquity 0.5
2. Judging Noun Types
3. A Demonstration of Ubiquity Parser 2

Related posts brought to you by Yet Another Related Posts Plugin.

This blog post includes a lot of complex CSS-formatted graphs which may be best viewed in — what else? — Firefox. You may also want to access this blog post directly rather than through a planet.

One portion of the problem description above merits clarification: I define “without sacrificing quality” to mean that, if we did not throw out any candidates early and waited until all the scores are computed fully and accurately, we would still yield the same top winners. This already gives us the key insight towards an appropriate solution: we can only throw out candidates when we know that it has no further chance of making it up into top candidates.

Let’s get formal

Let’s call the score of candidate at time in the derivation and we’ll assume that the score derivations are done in parallel with a unique origin ().¹ We’ll use the notation to represent the equilibrium or final score, equal to for all a certain which exists for each candidate. This function thus defines a time series for each candidate.

Given a set of candidates , we want to find the best subset of candidates; that is, such that

.

Approach 1: A Threshold Model

The key insight above would naturally give us what I call the threshold model. Here, we require the score sequences to be non-increasing: . This way, we can naturally throw out candidates which have reached below a certain threshold (or attained a certain level of badness, you might say) which we can then be sure will never recover.

For example, suppose the following diagram represents the scores of five different candidates after the first four time steps of the derivation. (The full gray bar marks the initial score () and the arrows indicate the successive score differentials.) The vertical line marks the threshold, .

We can tell after four steps that candidates 2 and 4, given that the score sequences are non-increasing, have no chance to finish their derivation with a score . What is important to note, however, is that candidate 4 already had no chance of beating the threshold after three steps. There was no need to calculate the fourth derivation of the score of candidate 4 (). In other words, after three steps, we could completely take candidate 4 out of the running and after another step, take candidate 2 out of the running.

C1      C2      C3      C4      C5      …	→	C1       C2       C3       C4       C5       …	→	C1        C2        C3        C4     C5        …	→

This non-decreasing score approach was used in Ubiquity Parser 2 until just recently, and you can in fact still play with it on the online Ubiquity Parser TNG demo. In that version, every parse started with an initial score of 1 and every score factor would be a value between 0 and 1. Every score factor was multiplied onto the previous score throughout the derivation, making it trivially non-increasing.

The problem with this approach is how to choose a smart threshold and that, given a constant threshold, you may get a different number of results for every different candidate set (i.e. parser query). If your score indicates a meaningful value with an a priori specified target of acceptable values, having a threshold makes sense. In the case of Ubiquity, however, the interface expects a certain number of suggestions to be returned.² If we plan to display five suggestions but the parser only returns four, even though there were other candidates, there must be a very good reason and justification for that threshold value.

Approach 2: Raising the Bar

The problem with Approach 1 was that there was no way of guaranteeing that we would yield our predefined winning candidates. Even if at some point in the derivation we are left with candidates still above the threshold, as the only restriction we have is that our score series are non-increasing, there is still a possibility that those remaining candidates’ scores will drop below later in the derivation.

We must instead at some point in the derivation identify (a) a set of at least candidates which will not get “worse” in the derivation and (b) candidates which have no chance of overtaking the (a) candidates. In this situation we can safely throw out the (b) candidates.

One way to do this is to require that all the scores are bounded and non-decreasing. By virtue of being non-decreasing, our top candidates at any point in our derivation will never get “worse” afterwards, satisfying condition (a). If relatively early in the computation we can compute a bound , we can identify candidates which will never surpass the top candidates in group (a) above, satisfying condition (b).

In the example below, and the thin bars mark the upper bounds . At we can identify candidate 2 and 4 as being our top two candidates. Note that there is one candidate, candidate 5, whose upper bound is less than both and . By definition and because the scores are non-decreasing and . Therefore

and

and we can thus throw out candidate 5 at this point. By the same logic, after we can throw candidate 2 out of the running.

C1      C2      C3      C4      C5      …	→	C1       C2       C3       C4       C5     …	→	C1        C2     C3        C4        C5     …	→

Calling this the “raising the bar” method refers to the fact that, at any particular time , the “bar” is and every other candidate must have an upper bound greater than the bar in order to not be thrown out of consideration. This “bar” itself is, together with the component scores, non-decreasing, decreasing the number of surviving candidates over time.

In the case of the Ubiquity parser we could build such a non-decreasing and bounded scoring model by using an additive model. As the main component of parser scoring is how well the parsed arguments match the verbs’ specified nountypes, we could simply add up all the confidence scores of each nountype suggestion, each of which are a value between 0 and 1. This would trivially be non-decreasing. As each parse has a finite and known number of parsed arguments, we could easily determine a bound as well. For example, say a parse has two arguments. Before we check each of the nountypes’ match scores, we already know that .

Unfortunately, there are also other factors which we would like to consider in our parses which may not fit into this non-decreasing model so easily…

Approach 2’: The Rising Sun Model³

One problem with both of the previous approaches is that it requires that the scoring schemes be either non-increasing or non-decreasing across the derivation. There are many situations, however, where you would want different factors to affect the score both positively and negatively. In the case of the Ubiquity parser, here are some different factors which could be good positive and negative score factors in computing the score of each parse.

As we see, there are both positive and negative factors which we hope to consider in scoring our possible Ubiquity parses. They key to making this work is by noting that Approach 2 only requires that the scoring series be bounded and non-decreasing after a certain known time in the derivation. For example, even if a parse involves a number of decreases early in the parse derivation, if after a certain point we can be certain that it is non-decreasing and bounded, we can simply use that bound and start eliminating poor candidates at that time (in this example, after ).

This is very much possible in the Ubiquity parser as, given the Ubiquity Parser 2 design, the negative factors such as whether the parse has a verb from the input or not (step 2), whether multiple arguments are identified with the same semantic role (step 4), and how many of the verb’s arguments are in the input (step 4) can be identified early on in the derivation, all before the very computationally intensive step of nountype detection (step 7) and argument suggestion (step 8). In this way, we can front-load all the negative factors in scoring and continue to use a version of Approach 2 to optimize our parsing.

We can moreover make the effect of the negative factors be felt across the entire derivation by figuring the negative factors into a factor between 0 and 1 and multiplying it onto each of the positive factors being added. In other words, we can compute all the negative factors into a single score multiplier earlier in the derivation and then afterwards when adding up each of the positive factors simply applying that score multiplier to the score derivation:

.

This model is what is going on under the hood in Ubiquity Parser 2. The Parser.Parse class has a property called .scoreMultiplier which contains the score multiplier as described above. A method called .getMaxScore() is implemented in addition to .getScore() so that, even before all of the nountype suggestion scores have been computed (e.g., in the case of asynchronous suggestions) .getMaxScore() can be used as an upper bound and compared to the in-progress scores of other candidates and lower candidates can thus be taken out of consideration earlier in the parse process.

Conclusion

In this blog post I’ve laid out a few different iterations of approaches I’ve thought of on the problem of scoring and ranking Ubiquity suggestions in a smart way. While some of the basic mechanisms of front-loading the negative factors into a scoreMultiplier and the computation of the maxScore (or upper bound) have been implemented, the actual optimization algorithm described here of removing parses from consideration earlier in the parser query has yet to be implemented in Ubiquity Parser 2 and I look forward to seeing it in action. In addition, there are surely factors I haven’t considered in the scoring or further tricks to improve the optimized scoring algorithm. I’d love to get your feedback and ideas on this topic. Thanks!

Related posts:

1. Scoring and Ranking Suggestions
2. Ubiquity Parser: The Next Generation Demo
3. This week on Ubiquity Parser: The Next Generation

Related posts brought to you by Yet Another Related Posts Plugin.

The issue of ranking Ubiquity suggestions can be restated as predicting an optimal output given a certain input and various conflicting considerations. Ubiquity (1.8, as of this writing) computes four “scores” for each suggestion:

1. duplicateDefaultMatchScore: 100 by default—lowered if an unused argument gets multiple suggestions (in the words of the code: “reduce the match score so that multiple entries with the same verb are only shown if there are no other verbs.”)
2. frequencyMatchScore: a score from the suggestion memory of the frequency of the suggestion’s verb, given the input verb (currently the first word) or nothing, in the case of noun-first suggestions
3. verbMatchScore: float in [0,1]: (as described here)
   • 0.75 is returned in case there it is a noun-first suggestion (by virtue of the fact that String.indexOf('')==0)
   • 1 if the verb name is equivalent across input-output
   • in [0.75,1) if the input is a prefix of the suggestion verb name
   • in [0.5,0.75) if the input is a non-prefix substring of the suggestion verb
   • in [0.25,0.5] if the input is a prefix of one of the synonyms
   • in [0,0.25) if the input is a non-prefix substring of one of the synonyms
4. argMatchScore: the number of arguments with matching “specific” nountypes, where “specific” is designated by the nountype having property rankLast=false.

With the numeric scores for each of these criteria, a partial order of suggestions is constructed using a lexicographic order: that is, compare candidates first using duplicateDefaultMatchScore, break ties using frequencyMatchScore, if still tied break using verbMatchScore, and if still tied break using argMatchScore. This paradigm of constraints is called “strictly ranked” and a corollary of this is that lower constraints, no matter how well you score on them, can never overcome a loss at a higher constraint. A crucial corollary of this system is that lower constraints’ scores need not be computed if a higher constraint already dooms it to a lower position.¹

Ranking in The Next Generation

One of the goals of Parser The Next Generation is to make noun/argument-first input first-class citizens of Ubiquity, improving their suggestions in particular to the benefit of verb-final languages. Arguments will be split up and tested against different noun types before a verb is even entered into the input, in which case target verbs can be ranked according to the appropriateness of the input’s arguments. As such, I believe the argMatchScore criteria above should either be ranked higher in a strictly ranked model or be allowed to overtake lower scores for the higher constraints in a non-strictly ranked model.

The Parser The Next Generation proposal and demo currently orders using a product of various criteria’s scores, rather than a lexicographic order of strictly ranked constraints. The component factors are:

1. 0.5 for parses where the verb was suggested
2. 0.5 for each extra (>1) object argument (essentially “unused words” in the previous parser)
3. the score of each argument against that semantic role’s target noun type
4. 0.8 for each unset argument of that verb

Each component score is a value in [0,1], so the score is always non-decreasing across the derivation. This offers a natural way to optimize the candidate set creation: if a possible parse ever gets a score below a magic “threshold” value, it is immediately thrown away.

A possible problem with the current Parser TNG scoring model is that it will implicitly hinder verbs and parses with more arguments as it could have more sub-1 noun type score factors—this consideration may be great enough that a weighted additive model should be considered over a multiplicative one.

How do you think we can make Ubiquity’s suggestion ranking smarter? What other factors should be considered, and what factors could be left out?

Related posts:

1. Ubiquity Parser: The Next Generation Demo
2. Ubiquity in Firefox: Focus on Japanese
3. Ubiquity Commands by The Numbers

Related posts brought to you by Yet Another Related Posts Plugin.

A week or two ago while visiting California, Jono and I had a productive charrette, resulting in a new architecture proposal for the Ubiquity parser, as laid out in Ubiquity Parser: The Next Generation. The new architecture is designed to support (1) the use of overlord verbs, (2) writing verbs by semantic roles, and (3) better suggestions for verb-final languages and other argument-first contexts. I’m happy to say that I’ve spent some time putting a proof-of-concept together.

I’ve implemented the basic algorithm of this parser for left-branching languages (like English) and also implemented some fake English verbs, noun types, and semantic roles. This demo should give you a basic sense of how this parser will attempt to identify different types of arguments and check their noun types even without clearly knowing the verb. This should make the suggestion ranking much smarter, particularly for verb-final contexts. (For a good example, try from Tokyo to San Francisco.)

➔ Check out the Ubiquity next-gen parser demo

Clicking on the environment info will give you some information on the specific verbs, noun types, and roles implemented. You can also scroll through the current parse section to see the step by step derivation of how the suggested parses were constructed.

I’ll be flying about 15 hours in the next hour as I make my way back to Japan… hopefully I’ll make some more progress on the plane! I’ll look forward to your comments! For those of you interested in checking out the code yourself, you can find it on BitBucket.

Related posts:

1. Writing commands with semantic roles
2. Ubiquity in Firefox: Focus on Japanese
3. User-Aided Disambiguation: a demo

Related posts brought to you by Yet Another Related Posts Plugin.

In order to introduce an external ordering source, we need to do four things: 1. create the external ordering source, 2. hook up (read “join”) the external ordering source 3. make sure we use that order, and 4. make it play nice. ^^

By the way, I’m going to assume you, dear reader, are PHP-savvy, proficient in MySQL, and already know a little about WordPress. This how-to is not for the PHPhobic.

The ordering source

For this example, suppose we want to display posts by order of “interestingness.” We’ll just create a table called wp_interestingness with two columns, ID and interestingness and populate it with some data. We’ll even be nice to our database by making sure the ID is the primary key. Easy.

Hook up the external ordering source

When you run a query through query_posts() (or use WP_Query’s query method²), what it’s doing is taking your special request and translating it into a MySQL statement. This means a query like "monthnum=1" is turned into SELECT ... wp_posts.* FROM wp_posts WHERE 1=1 AND MONTH(wp_posts.post_date)='1' .... Every different query introduces something new to the basic SELECT command—in this case, the AND MONTH(wp_posts.post_date)='1'.

We first want to introduce the interestingness for each post and that means joining the new table into the query. We’ll do this using the posts_join filter. This filter lets you add a join statement to the MySQL request.

1
2
3
4
5

add_filter('posts_join','my_join_filter');
function my_join_filter($arg) {
	$arg .= " natural join wp_interestingness ";
	return $arg;
}

Note that here we’re using natural join as wp_posts and wp_interestingness have only one key in common, ID, and that’s exactly the column we want to join them on.

Use the new order

Now that we’ve joined wp_interestingness in, we can refer to wp_interestingness.interestingness in our query. Note now that, by default, an $wpdb->posts.post_date will be used to order the posts. We’ll use another filter here; this time posts_orderby, to patch this part of the query. We’ll search for the default ORDER BY value and replace it with our own interestingness.

1
2
3
4
5
6

add_filter('posts_orderby','my_orderby_filter');
function my_orderby_filter($arg) {
	global $wpdb;
	$arg = str_replace("$wpdb->posts.post_date","wp_interestingness.interestingness",$arg);
	return $arg;
}

By the way, you can now check the resulting MySQL query by echoing $wp_query->request. (If you’re using the WP_Query method I advocated below in footnote (2), you’ll of course have to change $wp_query to the WP_Query object you’re using.)

Learn to play nice ^^

The instructions above do indeed work, but they also cause some major breakdowns in other functions of your blog. Why? That’s because the current code will edit your queries for every instance of The Loop: your index page, your archives, and your RSS feeds. You probably only want to search by interestingness in certain situations. What we need is a way to tell our (admittedly stupid) my_join_filter and my_orderby_filter when they should apply their interestingness magic and when they shouldn’t. There are several ways to set up such a system but here I’ll lay out one that I feel is particularly elegant. We’ll set it up so you can actually use query_posts("orderby=interestingness") and it’ll know what you’re talking about.

One of the first things that happens in query_posts—indeed, way before even the posts_join and posts_orderby filters—is an action hook called parse_query. This lets us look at the initial state of the WP_Query object as it starts to run. In particular, we can look at the orderby query variable and see if we want to order by interestingness. If we do, we’ll set a global variable called $use_interestingness_flag to be true.

1
2
3
4
5
6
7
8

add_action('parse_query','set_use_interestingness_flag');
function set_use_interestingness_flag($query) {
	global $use_interestingness_flag;
	if ($query->query_vars['orderby'] == 'interestingness')
		$yarpp_score_override = true;
	else
		$yarpp_score_override = false;
}

Now we just have to edit our filters so they only run when $use_interestingness_flag == true. We also will make sure to turn the flag back off in my_orderby_filter, as it’s our last filter to run during each query. It’s just like putting the seat back down after using a unisex bathroom.³

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

add_filter('posts_join','my_join_filter');
function my_join_filter($arg) {
	global $use_interestingness_flag;
	if ($use_interestingness_flag)
		$arg .= " natural join wp_interestingness ";
	return $arg;
}
add_filter('posts_orderby','my_orderby_filter');
function my_orderby_filter($arg) {
	global $wpdb, $use_interestingness_flag;
	if ($use_interestingness_flag)
		$arg = str_replace("$wpdb->posts.post_date","wp_interestingness.interestingness",$arg);
	$use_interestingness_flag = false;
	return $arg;
}

This method has a great advantage as you can just set it up once and invoke it whenever you want, even together with other parameters, without any additional code. For example, you can try query_posts("monthnum=1&orderby=interestingness") or query_posts("cat=-529&orderby=interestingness").

Conclusion

Adding an external ordering source to your WordPress post queries can be relatively straightforward if you understand what query_posts does and take advantage of its hooks. This tutorial can also serve as the basis for many other patches to WP_Query, not just the orderby parameter. To better understand the way WordPress builds its MySQL queries and the many posts_* filters which you can take advantage of, go to the source: wp-includes/query.php. Finally, you can use the special parse_query hook and global variables as flags to only apply the filters when necessary.

Related posts:

1. Yet Another Related Posts Plugin 2.0
2. Markdown for WordPress and bbPress
3. Yet Another Related Posts Plugin

Related posts brought to you by Yet Another Related Posts Plugin.

Well, it’s been a while since I updated my plugin YARPP—in my humble opinion the best related posts plugin for WordPress. ^^ Today I release version 2.0, incorporating a number of important requests and bug fixes:

• New algorithm which considers tags and categories, by frequent request
• Order by score, date, or title, by request
• Excluding certain tags or categories, by request
• Sample output displayed in the options screen
• Bugfix: an excerpt length bug
• Bugfix: now compatible with the following plugins:
  • diggZEt
  • WP-Syntax
  • Viper’s Video Quicktags
  • WP-CodeBox
  • WP shortcodes

Check out the Yet Another Related Posts Plugin page on this site, the page on wordpress.org, or download it directly now!

Related posts:

1. Yet Another Related Posts Plugin
2. Modifiying WordPress plugin activation behavior
3. Markdown for WordPress and bbPress

Related posts brought to you by Yet Another Related Posts Plugin.

This posting is now outdated… for the latest information on YARPP, please visit YARPP’s very own page on my site, or its page on wordpress.org. If you have questions, please submit on the wordpress.org forum. Thanks!

Description

Today I’m releasing Yet Another Related Posts Plugin (YARPP¹) 1.0 for WordPress. It’s the result of some tinkering with Peter Bowyer’s version of Alexander Malov & Mike Lu’s Related Entries plugin. Modifications made include:

1. Limiting by a threshold: Peter Bowyer did the great work of making the algorithm use mysql’s fulltext search score to identify related posts. But it currently just displayed, for example, the top 5 most “relevant” entries, even if some of them weren’t at all similar. Now you can set a threshold limit² for relevance, and you get more related posts if there are more related posts and less if there are less. Ha!
2. Being a better plugin citizen: now it doesn’t require the user to click some sketchy button to alter the database and enable a fulltext key. Using register_activation_hook, it does it automagically on plugin activation. Just install and go!
3. Miscellany: a nicer options screen, displaying the fulltext match score on output for admins, an option to allow related posts from the future, a couple bug fixes, etc.

Installation

Just put it in your /wp-content/plugins/ directory, activate, and then drop the related_posts function in your WP loop. Change any options in the Related Posts (YARPP) Options pane in Admin > Plugins.

You can override any options in an individual instance of related_posts using the following syntax:

`related_posts(limit, threshold, before title, after title, show excerpt, len, before excerpt, after excerpt, show pass posts, past only, show score);

Most of these should be self-explanatory. They’re also in the same order as the options on the YARPP Options pane.

Example: related_posts(10, null, 'title: ') changes the maximum related posts number to 10, keeps the default threshold from the Options pane, and adds title: to the beginning of every title.

There’s also a related_posts_exist) function. It has three optional arguments to override the defaults: a threshold, the past only boolean, and the show password-protected posts boolean.

Examples

For a barebones setup, just drop <?php related_posts(); ?> right after <?php the_content() ?>.

On my own blog I use the following code with <li> and </li> as the before/after entry options:

<?php if (related_posts_exist()): ?>
<p>Related posts:
<ol>
<?php related_posts(); ?>
</ol>
</p>
<?php else: ?>
<p>No related posts.</p>
<?php endif; ?>

Coming soon (probably)

1. Incorporation of tags and categories in the algorithm. I’ve gotten the code working, but I still need to think about what the most natural algorithm would be for weighing these factors against the mysql fulltext score currently used (and works pretty well, I must say).
2. Um, something else! Let me know if you have any suggestions for improvement. ^^

Version log

1.0 Initial upload (20071229)

1.0.1 Bugfix: 1.0 assumed you had Markdown installed (20070105)

Related posts:

1. Introducing Smartdate

Related posts brought to you by Yet Another Related Posts Plugin.