Framing & context

Simpson’s paradox

Every group goes one way. The total goes the other.

a.k.a. the reversal paradox · the Yule–Simpson effect · aggregation bias · confounded totals

Break a comparison into groups and each group can point one way while the pooled total points the other, with no arithmetic error anywhere. It happens when the two sides being compared are not made of the same mix of cases: whichever side was handed the easy ones wins the total on case mix alone. The pooled number then mostly records who got which cases, wearing the clothes of a result.

How to spot it

One headline rate per side, where each side chose its own members.
Equal sample sizes offered as proof that nothing needs adjusting — equal n says nothing about equal difficulty.
Ask what varies inside each side that also drives the outcome, then ask whether it is split evenly between them.
A gap that shrinks, vanishes, or reverses depending on which breakdown happens to be on the slide.

The fix

Break the comparison out by the thing that drives the outcome and show every group, not just the total. If a single number is needed, standardise it — score both sides against one shared mix of cases, so they are being asked the same question. And when the mix was chosen rather than assigned, say what the number can and cannot settle: undoing the confounder you happened to record is not the same as undoing the ones you didn’t.

In the gallery

Dark navy deck slide, masthead “Stone Programme Review · 700 consecutive cases · one measure”, headlined “The newer procedure clears more stones”. A subhead reads “Every renal-calculus case in the two surgical arms, scored the same way: the stone gone, or reduced below 2 mm, three months after treatment. Equal arms, one definition of success, nobody dropped — and the newer, less invasive procedure comes out in front.” Two long gold bars run against a zero-based axis labelled 0, 20, 40, 60, 80, 100%, captioned “Success at three months, % of cases treated”: Percutaneous nephrolithotomy, described as the newer, less invasive route, at 82.6% with “289 of 350 cases” beneath it, and Open surgery, the established operation, at 78.0% with “273 of 350 cases”. A short teal bracket spans the small gap between the two bar ends and is labelled “+4.6 points for percutaneous”. Below a rule, a second panel headed “Cases in each arm” draws two identical slate bars, both labelled 350, under the note “Identical arms, so the two success rates rest on the same number of patients. Nothing to weight, nothing to adjust.” Three teal-ticked boxes run along the bottom: “Zero-based axis — bar length is the number, start to finish”, “Equal arms — 350 patients on each side of the comparison”, and “Every case counted — no exclusions, one definition of success”. Down the right, a card reads “+4.6 pts, percutaneous over open, measured across every case in the series”, another reads “700 consecutive patients, one hospital series, one definition of success. None excluded.”, and a gold note headed “Nothing up our sleeve” says the axis starts at zero, the count sits on every bar, both arms are exactly the same size and every patient is in one of them — run the usual checks on this slide and all of them pass. A footnote credits Charig, Webb, Payne and Wickham in the British Medical Journal, 1986, and calls the slide a Stone Programme demonstration.

The 83% procedure that trails in both stone sizes

Simpson’s paradox