This blog post includes a lot of complex CSS-formatted graphs which may be best viewed in — what else? — Firefox. You may also want to access this blog post directly rather than through a planet.

One portion of the problem description above merits clarification: I define “without sacrificing quality” to mean that, if we did not throw out any candidates early and waited until all the scores are computed fully and accurately, we would still yield the same top winners. This already gives us the key insight towards an appropriate solution: we can only throw out candidates when we know that it has no further chance of making it up into top candidates.

Let’s get formal

Let’s call the score of candidate at time in the derivation and we’ll assume that the score derivations are done in parallel with a unique origin ().¹ We’ll use the notation to represent the equilibrium or final score, equal to for all a certain which exists for each candidate. This function thus defines a time series for each candidate.

Given a set of candidates , we want to find the best subset of candidates; that is, such that

.

Approach 1: A Threshold Model

The key insight above would naturally give us what I call the threshold model. Here, we require the score sequences to be non-increasing: . This way, we can naturally throw out candidates which have reached below a certain threshold (or attained a certain level of badness, you might say) which we can then be sure will never recover.

For example, suppose the following diagram represents the scores of five different candidates after the first four time steps of the derivation. (The full gray bar marks the initial score () and the arrows indicate the successive score differentials.) The vertical line marks the threshold, .

We can tell after four steps that candidates 2 and 4, given that the score sequences are non-increasing, have no chance to finish their derivation with a score . What is important to note, however, is that candidate 4 already had no chance of beating the threshold after three steps. There was no need to calculate the fourth derivation of the score of candidate 4 (). In other words, after three steps, we could completely take candidate 4 out of the running and after another step, take candidate 2 out of the running.

C1      C2      C3      C4      C5      …	→	C1       C2       C3       C4       C5       …	→	C1        C2        C3        C4     C5        …	→

This non-decreasing score approach was used in Ubiquity Parser 2 until just recently, and you can in fact still play with it on the online Ubiquity Parser TNG demo. In that version, every parse started with an initial score of 1 and every score factor would be a value between 0 and 1. Every score factor was multiplied onto the previous score throughout the derivation, making it trivially non-increasing.

The problem with this approach is how to choose a smart threshold and that, given a constant threshold, you may get a different number of results for every different candidate set (i.e. parser query). If your score indicates a meaningful value with an a priori specified target of acceptable values, having a threshold makes sense. In the case of Ubiquity, however, the interface expects a certain number of suggestions to be returned.² If we plan to display five suggestions but the parser only returns four, even though there were other candidates, there must be a very good reason and justification for that threshold value.

Approach 2: Raising the Bar

The problem with Approach 1 was that there was no way of guaranteeing that we would yield our predefined winning candidates. Even if at some point in the derivation we are left with candidates still above the threshold, as the only restriction we have is that our score series are non-increasing, there is still a possibility that those remaining candidates’ scores will drop below later in the derivation.

We must instead at some point in the derivation identify (a) a set of at least candidates which will not get “worse” in the derivation and (b) candidates which have no chance of overtaking the (a) candidates. In this situation we can safely throw out the (b) candidates.

One way to do this is to require that all the scores are bounded and non-decreasing. By virtue of being non-decreasing, our top candidates at any point in our derivation will never get “worse” afterwards, satisfying condition (a). If relatively early in the computation we can compute a bound , we can identify candidates which will never surpass the top candidates in group (a) above, satisfying condition (b).

In the example below, and the thin bars mark the upper bounds . At we can identify candidate 2 and 4 as being our top two candidates. Note that there is one candidate, candidate 5, whose upper bound is less than both and . By definition and because the scores are non-decreasing and . Therefore

and

and we can thus throw out candidate 5 at this point. By the same logic, after we can throw candidate 2 out of the running.

C1      C2      C3      C4      C5      …	→	C1       C2       C3       C4       C5     …	→	C1        C2     C3        C4        C5     …	→

Calling this the “raising the bar” method refers to the fact that, at any particular time , the “bar” is and every other candidate must have an upper bound greater than the bar in order to not be thrown out of consideration. This “bar” itself is, together with the component scores, non-decreasing, decreasing the number of surviving candidates over time.

In the case of the Ubiquity parser we could build such a non-decreasing and bounded scoring model by using an additive model. As the main component of parser scoring is how well the parsed arguments match the verbs’ specified nountypes, we could simply add up all the confidence scores of each nountype suggestion, each of which are a value between 0 and 1. This would trivially be non-decreasing. As each parse has a finite and known number of parsed arguments, we could easily determine a bound as well. For example, say a parse has two arguments. Before we check each of the nountypes’ match scores, we already know that .

Unfortunately, there are also other factors which we would like to consider in our parses which may not fit into this non-decreasing model so easily…

Approach 2’: The Rising Sun Model³

One problem with both of the previous approaches is that it requires that the scoring schemes be either non-increasing or non-decreasing across the derivation. There are many situations, however, where you would want different factors to affect the score both positively and negatively. In the case of the Ubiquity parser, here are some different factors which could be good positive and negative score factors in computing the score of each parse.

As we see, there are both positive and negative factors which we hope to consider in scoring our possible Ubiquity parses. They key to making this work is by noting that Approach 2 only requires that the scoring series be bounded and non-decreasing after a certain known time in the derivation. For example, even if a parse involves a number of decreases early in the parse derivation, if after a certain point we can be certain that it is non-decreasing and bounded, we can simply use that bound and start eliminating poor candidates at that time (in this example, after ).

This is very much possible in the Ubiquity parser as, given the Ubiquity Parser 2 design, the negative factors such as whether the parse has a verb from the input or not (step 2), whether multiple arguments are identified with the same semantic role (step 4), and how many of the verb’s arguments are in the input (step 4) can be identified early on in the derivation, all before the very computationally intensive step of nountype detection (step 7) and argument suggestion (step 8). In this way, we can front-load all the negative factors in scoring and continue to use a version of Approach 2 to optimize our parsing.

We can moreover make the effect of the negative factors be felt across the entire derivation by figuring the negative factors into a factor between 0 and 1 and multiplying it onto each of the positive factors being added. In other words, we can compute all the negative factors into a single score multiplier earlier in the derivation and then afterwards when adding up each of the positive factors simply applying that score multiplier to the score derivation:

.

This model is what is going on under the hood in Ubiquity Parser 2. The Parser.Parse class has a property called .scoreMultiplier which contains the score multiplier as described above. A method called .getMaxScore() is implemented in addition to .getScore() so that, even before all of the nountype suggestion scores have been computed (e.g., in the case of asynchronous suggestions) .getMaxScore() can be used as an upper bound and compared to the in-progress scores of other candidates and lower candidates can thus be taken out of consideration earlier in the parse process.

Conclusion

In this blog post I’ve laid out a few different iterations of approaches I’ve thought of on the problem of scoring and ranking Ubiquity suggestions in a smart way. While some of the basic mechanisms of front-loading the negative factors into a scoreMultiplier and the computation of the maxScore (or upper bound) have been implemented, the actual optimization algorithm described here of removing parses from consideration earlier in the parser query has yet to be implemented in Ubiquity Parser 2 and I look forward to seeing it in action. In addition, there are surely factors I haven’t considered in the scoring or further tricks to improve the optimized scoring algorithm. I’d love to get your feedback and ideas on this topic. Thanks!

Related posts:

1. Scoring and Ranking Suggestions
2. Judging Noun Types
3. Ubiquity Parser: The Next Generation Demo

Related posts brought to you by Yet Another Related Posts Plugin.

The issue of ranking Ubiquity suggestions can be restated as predicting an optimal output given a certain input and various conflicting considerations. Ubiquity (1.8, as of this writing) computes four “scores” for each suggestion:

1. duplicateDefaultMatchScore: 100 by default—lowered if an unused argument gets multiple suggestions (in the words of the code: “reduce the match score so that multiple entries with the same verb are only shown if there are no other verbs.”)
2. frequencyMatchScore: a score from the suggestion memory of the frequency of the suggestion’s verb, given the input verb (currently the first word) or nothing, in the case of noun-first suggestions
3. verbMatchScore: float in [0,1]: (as described here)
   • 0.75 is returned in case there it is a noun-first suggestion (by virtue of the fact that String.indexOf('')==0)
   • 1 if the verb name is equivalent across input-output
   • in [0.75,1) if the input is a prefix of the suggestion verb name
   • in [0.5,0.75) if the input is a non-prefix substring of the suggestion verb
   • in [0.25,0.5] if the input is a prefix of one of the synonyms
   • in [0,0.25) if the input is a non-prefix substring of one of the synonyms
4. argMatchScore: the number of arguments with matching “specific” nountypes, where “specific” is designated by the nountype having property rankLast=false.

With the numeric scores for each of these criteria, a partial order of suggestions is constructed using a lexicographic order: that is, compare candidates first using duplicateDefaultMatchScore, break ties using frequencyMatchScore, if still tied break using verbMatchScore, and if still tied break using argMatchScore. This paradigm of constraints is called “strictly ranked” and a corollary of this is that lower constraints, no matter how well you score on them, can never overcome a loss at a higher constraint. A crucial corollary of this system is that lower constraints’ scores need not be computed if a higher constraint already dooms it to a lower position.¹

Ranking in The Next Generation

One of the goals of Parser The Next Generation is to make noun/argument-first input first-class citizens of Ubiquity, improving their suggestions in particular to the benefit of verb-final languages. Arguments will be split up and tested against different noun types before a verb is even entered into the input, in which case target verbs can be ranked according to the appropriateness of the input’s arguments. As such, I believe the argMatchScore criteria above should either be ranked higher in a strictly ranked model or be allowed to overtake lower scores for the higher constraints in a non-strictly ranked model.

The Parser The Next Generation proposal and demo currently orders using a product of various criteria’s scores, rather than a lexicographic order of strictly ranked constraints. The component factors are:

1. 0.5 for parses where the verb was suggested
2. 0.5 for each extra (>1) object argument (essentially “unused words” in the previous parser)
3. the score of each argument against that semantic role’s target noun type
4. 0.8 for each unset argument of that verb

Each component score is a value in [0,1], so the score is always non-decreasing across the derivation. This offers a natural way to optimize the candidate set creation: if a possible parse ever gets a score below a magic “threshold” value, it is immediately thrown away.

A possible problem with the current Parser TNG scoring model is that it will implicitly hinder verbs and parses with more arguments as it could have more sub-1 noun type score factors—this consideration may be great enough that a weighted additive model should be considered over a multiplicative one.

How do you think we can make Ubiquity’s suggestion ranking smarter? What other factors should be considered, and what factors could be left out?

Related posts:

1. Scoring for Optimization
2. Judging Noun Types
3. Ubiquity Parser: The Next Generation Demo

Related posts brought to you by Yet Another Related Posts Plugin.

In order to introduce an external ordering source, we need to do four things: 1. create the external ordering source, 2. hook up (read “join”) the external ordering source 3. make sure we use that order, and 4. make it play nice. ^^

By the way, I’m going to assume you, dear reader, are PHP-savvy, proficient in MySQL, and already know a little about WordPress. This how-to is not for the PHPhobic.

The ordering source

For this example, suppose we want to display posts by order of “interestingness.” We’ll just create a table called wp_interestingness with two columns, ID and interestingness and populate it with some data. We’ll even be nice to our database by making sure the ID is the primary key. Easy.

Hook up the external ordering source

When you run a query through query_posts() (or use WP_Query’s query method²), what it’s doing is taking your special request and translating it into a MySQL statement. This means a query like "monthnum=1" is turned into SELECT ... wp_posts.* FROM wp_posts WHERE 1=1 AND MONTH(wp_posts.post_date)='1' .... Every different query introduces something new to the basic SELECT command—in this case, the AND MONTH(wp_posts.post_date)='1'.

We first want to introduce the interestingness for each post and that means joining the new table into the query. We’ll do this using the posts_join filter. This filter lets you add a join statement to the MySQL request.

1
2
3
4
5

add_filter('posts_join','my_join_filter');
function my_join_filter($arg) {
	$arg .= " natural join wp_interestingness ";
	return $arg;
}

Note that here we’re using natural join as wp_posts and wp_interestingness have only one key in common, ID, and that’s exactly the column we want to join them on.

Use the new order

Now that we’ve joined wp_interestingness in, we can refer to wp_interestingness.interestingness in our query. Note now that, by default, an $wpdb->posts.post_date will be used to order the posts. We’ll use another filter here; this time posts_orderby, to patch this part of the query. We’ll search for the default ORDER BY value and replace it with our own interestingness.

1
2
3
4
5
6

add_filter('posts_orderby','my_orderby_filter');
function my_orderby_filter($arg) {
	global $wpdb;
	$arg = str_replace("$wpdb->posts.post_date","wp_interestingness.interestingness",$arg);
	return $arg;
}

By the way, you can now check the resulting MySQL query by echoing $wp_query->request. (If you’re using the WP_Query method I advocated below in footnote (2), you’ll of course have to change $wp_query to the WP_Query object you’re using.)

Learn to play nice ^^

The instructions above do indeed work, but they also cause some major breakdowns in other functions of your blog. Why? That’s because the current code will edit your queries for every instance of The Loop: your index page, your archives, and your RSS feeds. You probably only want to search by interestingness in certain situations. What we need is a way to tell our (admittedly stupid) my_join_filter and my_orderby_filter when they should apply their interestingness magic and when they shouldn’t. There are several ways to set up such a system but here I’ll lay out one that I feel is particularly elegant. We’ll set it up so you can actually use query_posts("orderby=interestingness") and it’ll know what you’re talking about.

One of the first things that happens in query_posts—indeed, way before even the posts_join and posts_orderby filters—is an action hook called parse_query. This lets us look at the initial state of the WP_Query object as it starts to run. In particular, we can look at the orderby query variable and see if we want to order by interestingness. If we do, we’ll set a global variable called $use_interestingness_flag to be true.

1
2
3
4
5
6
7
8

add_action('parse_query','set_use_interestingness_flag');
function set_use_interestingness_flag($query) {
	global $use_interestingness_flag;
	if ($query->query_vars['orderby'] == 'interestingness')
		$yarpp_score_override = true;
	else
		$yarpp_score_override = false;
}

Now we just have to edit our filters so they only run when $use_interestingness_flag == true. We also will make sure to turn the flag back off in my_orderby_filter, as it’s our last filter to run during each query. It’s just like putting the seat back down after using a unisex bathroom.³

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

add_filter('posts_join','my_join_filter');
function my_join_filter($arg) {
	global $use_interestingness_flag;
	if ($use_interestingness_flag)
		$arg .= " natural join wp_interestingness ";
	return $arg;
}
add_filter('posts_orderby','my_orderby_filter');
function my_orderby_filter($arg) {
	global $wpdb, $use_interestingness_flag;
	if ($use_interestingness_flag)
		$arg = str_replace("$wpdb->posts.post_date","wp_interestingness.interestingness",$arg);
	$use_interestingness_flag = false;
	return $arg;
}

This method has a great advantage as you can just set it up once and invoke it whenever you want, even together with other parameters, without any additional code. For example, you can try query_posts("monthnum=1&orderby=interestingness") or query_posts("cat=-529&orderby=interestingness").

Conclusion

Adding an external ordering source to your WordPress post queries can be relatively straightforward if you understand what query_posts does and take advantage of its hooks. This tutorial can also serve as the basis for many other patches to WP_Query, not just the orderby parameter. To better understand the way WordPress builds its MySQL queries and the many posts_* filters which you can take advantage of, go to the source: wp-includes/query.php. Finally, you can use the special parse_query hook and global variables as flags to only apply the filters when necessary.

Related posts:

1. Yet Another Related Posts Plugin
2. Yet Another Related Posts Plugin 2.0
3. Modifiying WordPress plugin activation behavior

Related posts brought to you by Yet Another Related Posts Plugin.