"Best" Picture?

Between reading all that politics-stuff out there, you might have heard that the Academy of Motion Pictures has decided to expand the number of nominees in their "best picture" category from five to ten. This is, of course, the perfect time to talk about voting methods!

Okay, I'd probably say that about almost anything... but really this is a great example. And liberal blogger Matthew Yglesias almost puts his finger on why: "[B]ut if they want to make this switch they also need to reform the voting procedure to something with ordered preferences or something." But here's the thing, Matt: they already do. Kind of.

The Method

That article explains the current (until just now) method used to vote for best picture. According to that article, it goes like this:

1. All Academy members can submit an ordered ballot of up to 5 movies
2. Any movie which gets at lest 20% of 1st-place choices becomes a nominee
3. The movie with the fewest 1st-place preferences is eliminated, and those ballots are re-allocated to their next choice
4. Repeat until there are 5 nominees
5. Then, a simple plurality vote on the nominees determines the actual winner

The first half is a (a very-poorly described version) of the single transferable vote method for multi-winner elections. And then, of course, we follow this up with a round of the worst voting method known: plurality.

Now, there's a funny thing about plurality; as shown by this graph from Brian Olsen's voting and elections page, the average performance of plurality (assuming honest voters) increases the more candidates there are... until you get beyond five candidates. So increasing the number of nominees to 10 might not actually help much.

But back to STV: it's not a terrible method, but it's not exactly great, particularly for this use: it's not designed to pick the best set of options out of a group, it's designed to pick a set of options that (more or less) evenly covers the voter's opinions. What I mean by that is, if there are, for instance (assuming we're picking 5 nominees) two very good movies that a very well received by a particular niche of voters, but those voters account for less than 40% of the electorate, then they can only get one of them to be nominated; STV will tend towards picking options strongly supported by other niches after one of these two is selected. STV tries to cover all the extremes, not pick 5 films that are all broadly well-liked. This is why it's used in political elections; if a region is 60% Democrat and 40% Republican, it's fairer to pick 3 Democrats and 2 Republicans than it is to pick 5 democrats (which is what would happen in 5 separate plurality elections for those same 5 seats.) This might explain why, sometimes, you get some really bizarre picks: the 20% of voters who hate everything popular randomly settle on some out-of-the-mainstream film. Addi tonally, it doesn't necessarily pick the favorite of each niche, either; like instant runoff voting (which is just STV with a single winner), if there are more than two strong competitors, you can run into problems with spoilers.

Still, STV-to-plurality is certain a better way to pick a winner than a simple plurality vote over a couple hundred candidates. But can we do better? You know what I'm going to say...

Score Voting!

That's right, the Academy should move to score voting. It's going to be a hard sell though. Hollywood (if any sci-fi film is evidence) absolutely hate math, logic, and science. So how do you go about convincing people that one method is better than another, when they have almost 80 years of examples for the old way and there's nothing to support the superiority of your new idea. Well, there's good news for us, because there is.

The Internet Movie Database allows its members to score any and every movie in their system, on 1 to 10 point scale and (barring some algorithmic scrubbing that we'll discuss in a minute), they sort them by average score. That's score voting. And IMDB has at least 430,000 "voters", who have voted on who knows how many hundreds (thousands?) of films. This is all the data we could ever ask for!

Now, about the scrubbing: IMDB doesn't used a simple arithmetic average. First, they work very hard to try to eliminate ballot stuffing (one person, one vote!), but that's probably impossible to do completely. Furthermore, Academy voting all takes place within a single year, whereas IMDB lets members vote years and years after the fact, and that may screw results as tastes change over the years. We'll just have to deal with all that in this analysis. Secondly, they realize that, since most people won't vote on most films, some films will have very few votes, and it'd be foolish to, for example, let a film with only 1 vote of 10 stars be listed as the greatest film of all time. So they do some math to correct for that, not unlike the quorum rule that scorevoting.net suggests. Their quorum is around one-third of one percent of the number of votes received by the most-voted on film; not a large number, but enough to quell the "lunatic with a fanatic following that no one else has heard of wins" fears.

So, how does score voting stack up against STV+Plurality? Compare and contrast for 2000-2008.

Year	Academy	IMDB
2008	Slumdog Millionaire	The Dark Knight1
2007	No Country for Old Men	No Country for Old Men
2006	The Departed	The Lives of Others1,3
2005	Crash2	Sin City1
2004	Million Dollar Baby	Eternal Sunshine of the Spotless Mind1
2003	The Lord of the Rings: The Return of the King	The Lord of the Rings: The Return of the King
2002	Chicago2	City of God1,3
2001	A Beautiful Mind2	The Lord of the Rings: The Fellowship of the Ring
2000	Gladiator	Memento1

1. Not nominated by Academy
2. Not contained in IMDB's "Best/Worst '2000s' Titles"
3. Listed in IMDB under original foreign title

Okay, so what does this tell us? I'll tell you my thoughts in the next post, but for now, I'd like to know your thoughts? Which method do you agree with more often? What films are you favorites that don't make either list? What do you think this years best picture is and–different question!–what film do you think the Academy will pick?