What is pairwise ranking? Why choosing between two beats star ratings

What is pairwise ranking?

Pairwise ranking, sometimes called pairwise comparison, puts a set of things in order by looking at only two of them at a time. You are shown a pair, you choose the one you prefer, and you move on to the next pair. No single decision tells you where anything belongs overall. The order emerges from the pattern of many small decisions.

The idea is older than any app. Psychologists have used paired comparisons to study preferences for about a hundred years, sports leagues effectively rank teams by results between pairs of them, and chess players have been rated by the outcomes of individual games since the 1960s. What these share is a simple observation: people are usually better at saying “this one, not that one” than at assigning an absolute number.

For a personal collection of films, books or trips, that means you never have to answer “Is this a seven or an eight?” You only answer “Of these two, which meant more to me?”

Why choosing between two is easier than five stars

Rating scales ask you to hold an imaginary standard in your head and measure each item against it. That standard drifts, and a few predictable problems follow.

• Rating inflation. People tend to use the top of a scale far more than the bottom, partly because they choose things they expect to enjoy. Over time, a three-star rating starts to feel like a criticism, and most entries cluster at four or five.
• The five-star pile-up. Once twenty favourites share the highest rating, the scale has stopped helping. It cannot tell you which of them you would recommend first or rewatch tonight.
• Anchoring. The rating you give one item is pulled towards whatever you rated just before it, or towards a public score you have already seen. A film watched after a disappointing one can look better than it would on another evening.
• Inconsistent scales. A four-star book from five years ago and a four-star book from last week may not mean the same thing, because your standard has moved in between.

A comparison avoids most of this. Both items are in front of you at once, so there is no imaginary standard to drift. A tie at the top of a scale becomes a question you can actually answer. The trade-off is that one comparison tells you less than one rating does, so you need more of them.

How Elo-style scores work, in plain words

The most familiar way to turn pairwise results into an order is the Elo system, devised by the physicist Arpad Elo for chess. Each item carries a score. When two items meet, the system first estimates how likely each one is to win, based on the gap between their scores. After your choice, the winner takes some points from the loser.

The key detail is how many points change hands. If the favourite wins, as expected, the scores move only a little. If the underdog wins, that is surprising, so the scores move a lot. Over many comparisons, items that keep winning drift upwards, items that keep losing drift downwards, and the scores settle into an order.

Notice that the scores themselves are not a verdict on quality. They only describe how a film has fared in your comparisons so far. A score of 1600 means “has tended to win against the films it met”, not “objectively a 16 out of 20”.

Draws and skips: two different kinds of “I can’t decide”

Sometimes a pair is genuinely too close to call. A good pairwise system gives you two separate ways out, and the difference between them matters.

A draw is an answer. You are saying the two are about equal for you, and the system should treat them that way. In an Elo-style system a draw nudges the two scores towards each other, as in the example above.

A skip is not an answer. You might not remember one of the books well enough, or the question simply does not make sense for that pair. Skipping should leave the scores alone and show you a different pair. Using a draw when you really mean “I don’t know” quietly pulls two unrelated items together, so it is worth keeping the two apart.

How many comparisons do you need?

Comparing every possible pair is thorough but quickly becomes impractical. Ten items have 45 possible pairs; twenty have 190; a hundred have 4,950. Nobody needs to sit through all of those.

You also do not need to. If you were sorting with perfect consistency, ten items could in principle be ordered in a little over twenty well-chosen comparisons. Personal taste is not that tidy, and an Elo-style system does not choose pairs with perfect efficiency, so expect to need more than that minimum. As a loose rule of thumb, a rough order tends to appear once each item has met several others, and the top of the list usually needs the most attention because that is where the differences are finest.

The useful question is not “Is it finished?” but “Do the neighbours look right?” If the films at positions four and five feel the wrong way round, compare them directly. If the order already matches your instincts, more comparisons will mostly confirm it.

Where pairwise ranking falls short

Pairwise ranking is a tool for describing your taste, not a method for discovering a correct answer. It is worth knowing its limits.

• It is only as clear as your question. “Which did I enjoy more?” and “Which is better made?” can produce different orders. Keep one question in mind for each list.
• Mood leaks in. On a tired evening, a comfort film may beat a demanding masterpiece. Over many comparisons this tends to average out, but a single session can tilt the order.
• Loops happen. You might prefer A to B, B to C, and C to A. That is not an error; different pairs bring out different qualities. A scoring system will place those items close together rather than resolve the contradiction.
• Early results move a lot. With only a few comparisons, one surprising choice can lift an item far up the list. Treat a new ranking as provisional.
• It does not say how much you liked anything. An order tells you that one film sits above another, not whether you enjoyed either. Pairing an order with broad tiers, or with a short note, keeps that information.

For a step-by-step approach to a personal top ten, see how to rank your favourite movies. If you would rather sort into broad bands than a strict order, how to make a movie tier list covers that.

Pairwise ranking in Crestora

Crestora’s iPhone app uses pairwise comparisons to rank the films, books and trips you collect. It shows you two at a time under the heading “Your call.” and you tap the one you prefer. Each choice updates Elo-style scores, and your collection is ordered by those scores.

Both ways out described above are there. Too tough records the two as equal in your rankings, and Skip shows another pair without changing your rankings. As scores settle, each item lands in a tier, such as A or B, based on its score band, so you can see broad groups as well as an exact order.

As with any pairwise ranking, the result reflects the comparisons you have made so far. Revisit close pairs, add new favourites as you meet them, and let the order change with you. Crestora is free on iPhone.