Skip to content
TOP NOW Did a Russian Lab Worker's Death Spark a New Plague Pandemic? / Instagram's New AI 'Muse': Creativity or Algorithmic Exploitation? / UK “rape capital” claim — what police data actually allows you to say
Rent Free News home rentfreenews. Claims vs data
Subscribe

How to Read Rankings Without Getting Fooled

Five questions to ask before you let a rank near a headline: who was asked, what was the exact ask, when, which kind of data, and who gains from your share.

Y
By Yash • Contributing Writer
Published • Updated
Share

Primary source: World Happiness Report methodology and UNODC statistical standards for crime data →

How to Read Rankings Without Getting Fooled
Illustration: Rent Free desk — By the Numbers
Why it matters

Rankings are the most shareable and the least checked format in public data. Five questions separate a usable number from a poster.

Short answer: A ranking is an instrument, not a fact. Every one is built for a specific question, from a specific sample, on a specific date, using a specific definition of the thing being measured. Five questions tell you whether the number answers the question you are about to use it for: who was asked, what was the exact ask, when was it measured, is this an administrative count or a survey, and who gains when you share it. If the piece does not answer the first four, it is not a ranking you can use, however famous the publisher.

1. Who was asked, and how many?

Start with the sample, because it decides what the number can describe.

Gallup’s Law and Order Index is built on nationally representative, probability-based samples of adults aged 15 and older in 144 countries and territories, with approximately 1,000 or more respondents per country, and a published margin of error of plus or minus 3.3 to plus or minus 5.4 percentage points. That is a national sample, and it can support a statement about a country.

A crowd survey on a comparison website cannot. Its sample is whoever was on that site, and the site’s users skew toward the people most likely to visit comparison websites. The gap between “1,000 adults chosen at random” and “some number of site visitors” is the gap between a country and a sample of one country’s internet users.

If a ranking does not publish its sample size and its method, stop there.

2. What was the exact ask?

The wording of the question changes the answer, and this is not a technicality.

The World Happiness Report builds its country ranking from six inputs, which include GDP per capita, social support, healthy life expectancy at birth, freedom to make life choices, generosity and perceptions of corruption. Gallup’s Law and Order Index uses four questions, and they are specific: whether you feel safe walking alone at night, whether you have confidence in the local police, whether money or property was stolen in the last 12 months, and whether you were assaulted or mugged in the past 12 months.

A different phrasing of “how satisfied are you with your life” produces a different ranking with the same name attached to it. Writers who merge two wordings into one chart mislead, usually by accident. Check the exact question before you compare two editions of the same index.

3. When was it measured?

A rank carries a vintage, and the vintage is usually the first thing to go.

A pandemic year, a price shock year, an election year and a harvest year are all different countries in the same table. When you see a ranking move, ask whether the inputs moved or the year changed. A twenty-place swing between editions usually marks an event rather than a trend, and a single-year comparison is a footnote, not a finding.

Our rule at Rent Free News is to plot three years rather than one. A flat line is a result. A spike is a footnote.

4. Administrative counts or survey reports?

This is the distinction that most often decides whether a crime story is true.

Police and administrative logs record what was reported and classified, so they move when reporting rules change, when filing practices change, and when a new category is created. Survey-based crime statistics ask people what happened to them, so they move with experience rather than with paperwork.

UNODC’s own statistical work sets out the comparability problems that follow from combining national administrative returns, which is why the office publishes method notes rather than a single league table. Never rank police logs across borders, and never treat a change in a log count as a change in offending without checking whether the definition changed first.

The same applies to any field where the definition is contested. Our Sweden crime piece shows the two sources disagreeing in a way that is not a contradiction but two different instruments.

5. Who gains from your share?

Outrage maps farm saves, and footnotes do not. Ask who posted the graphic first and what they sell. A visa agent gains from fear. A course seller gains from “worst” tags. A membership site gains from the traffic.

Motive does not make a fact false. It does set the bar for what you should have checked before you passed it on, and it is a reason to find the raw table yourself rather than quoting the graphic.

6. The house rule

Show the claim as it was shot. Link the raw table. List what we still lack. If the source set cannot be found, say that at the top of the piece in plain words. Readers forgive a gap that is labelled. They do not forgive a hidden one.

Three worked examples on this site: our safety maps brief on three different crime measures that answer three different questions, our Bangladesh piece on what a ranking cell does and does not contain, and our oil market piece on two forecasters who looked at the same month and returned answers 1.9 million barrels a day apart.

Sources to check yourself

Frequently asked questions

Q: What is the fastest way to spot a bad ranking? A: Check whether it publishes its sample size and its exact question wording. A ranking that hides either is a poster, not a measurement.

Q: Why do two crime rankings for the same country disagree? A: One is probably administrative and the other survey-based. They measure different things, and the gap is not an error.

Q: Does a ranking need three years to be usable? A: Not to be used, but to be trusted for a trend. A single edition is a snapshot with a vintage attached.

Q: Is it wrong to share a ranking I did not build? A: No. Share it with the caveats attached, and link the raw table so the reader can check what you checked.

How we checked: archived claim → primary dataset / VOD timestamp → quoted human with permission → verdict + corrections link. Report an error.

Reader discussion

Conversation for this story opens with memberships. Until then: found an error? Report it — we correct visibly.

The Sunday footnote — free

5 viral claims, what was missing.

Join the 2-minute Sunday read. One creator test inside. No spam.

AKM Loved by curious readers + quoted creators
Contextual Coverage

Related News — More in By the Numbers

All By the Numbers →

Keep reading

View all →