"Best Visual Effects" To Be Determined by Re-weighted Range Voting

We've occasionally talked about the voting procedures of the Academy Awards for Best Picture. In short, they recently changed to using instant runoff voting, and I think that was a pretty poor choice; they should use range (AKA score) voting. So imagine my delight when it was pointed out to me that the nominees for this year's award for best visual effects will be determined by re-weighted range voting! (Rule 7b.)

I'm not entirely sure that a method of proportional representation in the best way to pick nominees for a single-winner election, but this is sill fantastic news. I wish I knew how the idea of RRV made its way into Hollywood, but regardless of how it happened, I'm ecstatic that it has.

Re-weighted Range Voting uses the same ballot as normal range voting, namely, each voter can give each candidate any score within a given range. The first winner is determined the same as well, it's simply the candidate with the highest score. The next step—and the way in which the method becomes proportional—is that each voter who voted for the winner has all of the scores on their ballot re-weighted downward, based on how highly they scored the winner. If you gave the winner the maximum score, the rest of your ballot's weight is cut in half, and the lower the score you gave them, the more strength your ballot retains. And then the second winner is whichever has the highest re-weighted score. Each additional winner further re-weights every ballot (if you gave the maximum score to the first two winners, the rest of your ballot is cut to a third, and so on) so that each time a voter gets their way, they have less say on the following rounds.

RRV is, of course, perfectly compatible with an approval-style ballot, since approval is equivalent to range, with a "range" of just 0 and 1. Re-weighted approval could therefore be a very easy and effective way to implement proportional representation.

Oscar Update

As we discussed last summer, the Oscar for best picture is moving from a field of 5 to a field of 10, and the vote is done by instant runoff. Here's the 2009 best picture nominees, vs. IMDB's "Top Rated Titles".

Academy	IMDB
Avatar The Blind Side District 9 An Education The Hurt Locker Inglorious Basterds Precious: Based on the Novel ‘Push’ by Sapphire A Serious Man Up Up in the Air	Avatar Up Inglorious Basterds District 9 Fantastic Mr. Fox Star Trek

The IMDB list is a little truncated, since they only list by decade, and only the top 50 of the decade, so its hard to find IMDB's best 10 films for any one year (maybe they don't want to step on the Academy's toes). You'll note, there's a good amount of overlap this year, which isn't too surprising since the nomination step uses a "name five" approval-style ballot, in which each voter names up to five films, without ranking them. The ten most-often-named films become the nominees. Top-5 is mathematically similar to IMDB's score-based voting system; at least, the two are more similar to each other than either is to the instant runoff voting (IRV) used for the final round.

But which picture will win? It's tempting to say Avatar, since it's IMDB's top rated film of the year, but we know that IRV has somewhere between a 5 and 15% chance to exhibit non-monotonic behavior, in which the true first-choice is eliminated before the final round. And that's with three strong contenders; the odds get worse the more there are, and I think at least four of these can be considered "strong contenders".

But what does "true first-choice" mean? For me, it means consensus. If you put all the Academy voters in a room and told them to watch a movie, which movie would engender the greatest total happiness for the crowd? That's the question that score voting (and IMDB) asks. But supporters of IRV instead speak of something they call "core support"; they say that it's most important for a winner to have a large following of fanatics, voters who like them best of all. They don't use the word "fanatic", but it's appropriate. Instant runoff repeatedly removes the choice with the fewest fanatic followers and shuffles them off to their next favorite group; it's fair to call them fanatics because other options that you may also like are never considered until your number-one is eliminated. In other words, IRV doesn't offer you much of an opportunity to offer a compromise, because its almost always the compromise that gets eliminated by non-monotonic behavior. So the question is, is Avatar a compromise? Does it make everyone mostly happy, or does it make only a few people very happy? Luckily, IMDB records that data too! Maybe we'll examine that next time.

Also, remember that the Academy hates science fiction, and animation. In which case, maybe Inglorious Basterds has a good chance...

"Better" Picture

Which method is superior for choosing the best movie: single transferable vote to plurality, or score voting? Really, last post didn't give enough data. So first: more data. A note though; I moved foreign films into the year of their U.S. release (since that's what determines eligibility for Best Picture). Films that appear in both columns are in bold.

Year	Academy	IMDB
2009	not yet determined	partial result Up Star Trek The Hangover
2008	Slumdog Millionaire The Curious Case of Benjamin Button Frost/Nixon Milk The Reader	The Dark Knight WALL·E Slumdog Millionaire Gran Torino The Wrestler Lat den ratte komma in (Let the Right One In) The Curious Case of Benjamin Button Vals Im Bashir (Waltz with Bashir) Ip Man
2007	No Country for Old Men Atonement Juno Michael Clayton There Will Be Blood	No Country for Old Men There Will Be Blood Into the Wild The Bourne Ultimatum Ratatouille Le scaphandre et le papillon
2006	The Departed Babel Letters from Iwo Jima Little Miss Sunshine The Queen	Das Leben der Anderen (The Lives of Others) The Departed El Laberinto del Fauno (Pan's Labrinth) The Prestige Children of Men Letters from Iwo Jima
2005	Crash Brokeback Mountain Capote Good Night, and Good Luck Munich	Der Untergang (Downfall) Sin City Batman Begins V for Vendetta
2004	Million Dollar Baby The Aviator Finding Neverland Ray Sideways	Eternal Sunshine of the Spotless Mind Hotel Rwanda Million Dollar Baby Tasogara Seibei (The Twilight Samurai) The Incredibles
2003	The Lord of the Rings: The Return of the King Lost in Translation Master and Commander: The Far Side of the World Mystic River Seabiscuit	The Lord of the Rings: The Return of the King Oldboy Kill Bill: Vol. 1 Finding Nemo
2002	Chicago Gangs of New York The Hours The Lord of the Rings: The Two Towers The Pianist	Cidade de Deus (City of God) The Lord of the Rings: The Two Towers The Pianist Sen to Chihiro no Kamikakushi (Spirited Away)
2001	A Beautiful Mind Gosford Park In the Bedroom The Lord of the Rings: The Fellowship of the Ring Moulin Rouge!	The Lord of the Rings: The Fellowship of the Ring Le fabuleux destin d'Amelie Poulain (Amelie) Donnie Darko Amores Perros (Love's a Bitch) Dil Chahta Hai (Do Your Thing)
2000	Gladiator Chocolat Crouching Tiger, Hidden Dragon Erin Brockovich Traffic	Memento Requiem for a Dream Gladiator Snatch

So, take some time, look at, and see if it tells you anything. (Note: there are additional categories for "Best Animated Feature" and "Best Foreign Feature", which might explain some of that disparity. But there isn't a "Best Adapted Comic Book Feature".)

Our lone commentary from the last post mentioned that they were upset that Shawshank Redemption didn't win best picture in 1994; you'll be happy to know then that, according to IMDB, it's the greatest movie of all time.

Now, if you like IMDB's results, please ask yourself: if it works so well for movies, why wouldn't it work for elections? If the best method for picking a winner out of dozens and dozens of choices is score voting, why wouldn't it work for picking between two; or, do we only get to decide between two because we don't use a better system to decide?

"Best" Picture?

Between reading all that politics-stuff out there, you might have heard that the Academy of Motion Pictures has decided to expand the number of nominees in their "best picture" category from five to ten. This is, of course, the perfect time to talk about voting methods!

Okay, I'd probably say that about almost anything... but really this is a great example. And liberal blogger Matthew Yglesias almost puts his finger on why: "[B]ut if they want to make this switch they also need to reform the voting procedure to something with ordered preferences or something." But here's the thing, Matt: they already do. Kind of.

The Method

That article explains the current (until just now) method used to vote for best picture. According to that article, it goes like this:

1. All Academy members can submit an ordered ballot of up to 5 movies
2. Any movie which gets at lest 20% of 1st-place choices becomes a nominee
3. The movie with the fewest 1st-place preferences is eliminated, and those ballots are re-allocated to their next choice
4. Repeat until there are 5 nominees
5. Then, a simple plurality vote on the nominees determines the actual winner

The first half is a (a very-poorly described version) of the single transferable vote method for multi-winner elections. And then, of course, we follow this up with a round of the worst voting method known: plurality.

Now, there's a funny thing about plurality; as shown by this graph from Brian Olsen's voting and elections page, the average performance of plurality (assuming honest voters) increases the more candidates there are... until you get beyond five candidates. So increasing the number of nominees to 10 might not actually help much.

But back to STV: it's not a terrible method, but it's not exactly great, particularly for this use: it's not designed to pick the best set of options out of a group, it's designed to pick a set of options that (more or less) evenly covers the voter's opinions. What I mean by that is, if there are, for instance (assuming we're picking 5 nominees) two very good movies that a very well received by a particular niche of voters, but those voters account for less than 40% of the electorate, then they can only get one of them to be nominated; STV will tend towards picking options strongly supported by other niches after one of these two is selected. STV tries to cover all the extremes, not pick 5 films that are all broadly well-liked. This is why it's used in political elections; if a region is 60% Democrat and 40% Republican, it's fairer to pick 3 Democrats and 2 Republicans than it is to pick 5 democrats (which is what would happen in 5 separate plurality elections for those same 5 seats.) This might explain why, sometimes, you get some really bizarre picks: the 20% of voters who hate everything popular randomly settle on some out-of-the-mainstream film. Addi tonally, it doesn't necessarily pick the favorite of each niche, either; like instant runoff voting (which is just STV with a single winner), if there are more than two strong competitors, you can run into problems with spoilers.

Still, STV-to-plurality is certain a better way to pick a winner than a simple plurality vote over a couple hundred candidates. But can we do better? You know what I'm going to say...

Score Voting!

That's right, the Academy should move to score voting. It's going to be a hard sell though. Hollywood (if any sci-fi film is evidence) absolutely hate math, logic, and science. So how do you go about convincing people that one method is better than another, when they have almost 80 years of examples for the old way and there's nothing to support the superiority of your new idea. Well, there's good news for us, because there is.

The Internet Movie Database allows its members to score any and every movie in their system, on 1 to 10 point scale and (barring some algorithmic scrubbing that we'll discuss in a minute), they sort them by average score. That's score voting. And IMDB has at least 430,000 "voters", who have voted on who knows how many hundreds (thousands?) of films. This is all the data we could ever ask for!

Now, about the scrubbing: IMDB doesn't used a simple arithmetic average. First, they work very hard to try to eliminate ballot stuffing (one person, one vote!), but that's probably impossible to do completely. Furthermore, Academy voting all takes place within a single year, whereas IMDB lets members vote years and years after the fact, and that may screw results as tastes change over the years. We'll just have to deal with all that in this analysis. Secondly, they realize that, since most people won't vote on most films, some films will have very few votes, and it'd be foolish to, for example, let a film with only 1 vote of 10 stars be listed as the greatest film of all time. So they do some math to correct for that, not unlike the quorum rule that scorevoting.net suggests. Their quorum is around one-third of one percent of the number of votes received by the most-voted on film; not a large number, but enough to quell the "lunatic with a fanatic following that no one else has heard of wins" fears.

So, how does score voting stack up against STV+Plurality? Compare and contrast for 2000-2008.

Year	Academy	IMDB
2008	Slumdog Millionaire	The Dark Knight1
2007	No Country for Old Men	No Country for Old Men
2006	The Departed	The Lives of Others1,3
2005	Crash2	Sin City1
2004	Million Dollar Baby	Eternal Sunshine of the Spotless Mind1
2003	The Lord of the Rings: The Return of the King	The Lord of the Rings: The Return of the King
2002	Chicago2	City of God1,3
2001	A Beautiful Mind2	The Lord of the Rings: The Fellowship of the Ring
2000	Gladiator	Memento1

1. Not nominated by Academy
2. Not contained in IMDB's "Best/Worst '2000s' Titles"
3. Listed in IMDB under original foreign title

Okay, so what does this tell us? I'll tell you my thoughts in the next post, but for now, I'd like to know your thoughts? Which method do you agree with more often? What films are you favorites that don't make either list? What do you think this years best picture is and–different question!–what film do you think the Academy will pick?