The 83% procedure that trails in both stone sizes

Dark navy deck slide, masthead “Stone Programme Review · 700 consecutive cases · one measure”, headlined “The newer procedure clears more stones”. A subhead reads “Every renal-calculus case in the two surgical arms, scored the same way: the stone gone, or reduced below 2 mm, three months after treatment. Equal arms, one definition of success, nobody dropped — and the newer, less invasive procedure comes out in front.” Two long gold bars run against a zero-based axis labelled 0, 20, 40, 60, 80, 100%, captioned “Success at three months, % of cases treated”: Percutaneous nephrolithotomy, described as the newer, less invasive route, at 82.6% with “289 of 350 cases” beneath it, and Open surgery, the established operation, at 78.0% with “273 of 350 cases”. A short teal bracket spans the small gap between the two bar ends and is labelled “+4.6 points for percutaneous”. Below a rule, a second panel headed “Cases in each arm” draws two identical slate bars, both labelled 350, under the note “Identical arms, so the two success rates rest on the same number of patients. Nothing to weight, nothing to adjust.” Three teal-ticked boxes run along the bottom: “Zero-based axis — bar length is the number, start to finish”, “Equal arms — 350 patients on each side of the comparison”, and “Every case counted — no exclusions, one definition of success”. Down the right, a card reads “+4.6 pts, percutaneous over open, measured across every case in the series”, another reads “700 consecutive patients, one hospital series, one definition of success. None excluded.”, and a gold note headed “Nothing up our sleeve” says the axis starts at zero, the count sits on every bar, both arms are exactly the same size and every patient is in one of them — run the usual checks on this slide and all of them pass. A footnote credits Charig, Webb, Payne and Wickham in the British Medical Journal, 1986, and calls the slide a Stone Programme demonstration. — Showing the misleading chart

The same 700 cases on a cream background, kicked “The same 700 cases · split by the size of the stone” and headlined “Open surgery wins both sizes and loses the total”. A legend marks open surgery in blue and percutaneous nephrolithotomy in gold. Two side-by-side panels each carry two bars on a zero-based 0–100% axis. “Stones under 2 cm · 357 cases”: open surgery 93.1% (81/87), percutaneous 86.7% (234/270), noted “Open surgery ahead by 6.4 points.” “Stones 2 cm and over · 343 cases”: open surgery 73.0% (192/263), percutaneous 68.8% (55/80), noted “Open surgery ahead by 4.3 points.” Below, a panel headed “Equal arms, unequal cases” draws each arm’s 350 patients as one bar split light for stones under 2 cm and dark for 2 cm and over: open surgery 24.9% light and 75.1% dark — 263 stones of 2 cm or more — and percutaneous 77.1% light — 270 stones under 2 cm — and 22.9% dark. Beside it a small table lists success rates for open and percutaneous with the side that is ahead: under 2 cm, 93.1% against 86.7%, open; 2 cm and over, 73.0% against 68.8%, open; both pooled, 78.0% against 82.6%, percutaneous — the pooled row highlighted, under the note “Three rows, one dataset, no arithmetic errors. The bottom row is the only one anybody put on a slide.” Three cards run down the right. “Equal in size, not in difficulty” explains that three-quarters of the open-surgery arm was large stones and three-quarters of the percutaneous arm small ones, that surgeons picked the procedure with the stone in front of them, and that pooling both procedures a stone under 2 cm cleared 88.2% of the time (315/357) against 72.0% for a stone of 2 cm or more (247/343) — a 16.2-point difficulty gap, more than three times the 4.6 points the totals argued about. “Give both arms the same case mix” notes that the series held 357 small stones and 343 large ones, 51% and 49%, and shows the arithmetic: open surgery 0.51 × 93.1 + 0.49 × 73.0 = 83.3%, percutaneous 0.51 × 86.7 + 0.49 × 68.8 = 77.9% — a 4.6-point lead becoming a 5.4-point deficit without a single patient’s outcome changing. “What the split does not settle” warns that this is a retrospective hospital series rather than a randomised trial, so the honest chart does not crown open surgery either; it shows the pooled total was never evidence in the first place. A footnote credits the 1986 British Medical Journal paper for the outcomes and Julious and Mullee’s 1994 BMJ note for the stone-size breakdown, and says the percentages, the standardisation and both drawings are ours. — Showing the misleading chart

Be the first to star this exhibit

Two ways of removing a kidney stone, 350 patients each, one definition of success: percutaneous nephrolithotomy clears 82.6% and open surgery 78.0%. The same 700 cases split by the size of the stone give open surgery 93.1% against 86.7% on the small ones and 73.0% against 68.8% on the large ones. It wins both halves and loses the total, because three-quarters of its caseload was the hard half.

Charig, Webb, Payne & Wickham, “Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy”, British Medical Journal 292 (1986): 879–882 — stone-size breakdown as tabulated by Julious & Mullee, “Confounding and Simpson’s paradox”, BMJ 309 (1994): 1480–1481, 1986Added August 13, 2026

01The claim

The newer procedure clears more stones. Every renal-calculus case in the two surgical arms is here, scored the same way — the stone gone, or reduced below 2 mm, three months after treatment. Percutaneous nephrolithotomy succeeded in 289 of its 350 cases, 82.6%; open surgery in 273 of its 350, 78.0%. That is 4.6 points to the newer, less invasive route, and the slide invites you to check its work: the axis starts at zero, the count sits on every bar, nobody was dropped, and the two arms are exactly the same size, so there is nothing to weight and nothing to adjust.

02The trick

Everything the slide says about itself is true, which is what makes it worth hanging. The axis really does start at zero, both arms really do hold 350 patients, and 289/350 and 273/350 are the figures the surgeons published. The equal arms are the sleight of hand: they say the two rates rest on the same number of patients, and the slide quietly lets that stand in for the same kind of patients. The series recorded stone size as well as outcome, and the two arms were nothing alike on it. Open surgery took 263 stones of 2 cm or more — 75.1% of its caseload — while percutaneous took 270 under 2 cm, 77.1% of its. Size drives the outcome: pooling both procedures, a stone under 2 cm cleared 88.2% of the time (315 of 357) and a stone of 2 cm or more 72.0% (247 of 343). That is a 16.2-point difficulty gap between the cases, more than three times the 4.6-point gap the totals were arguing about — so the pooled figure is mostly a record of which stones each procedure was handed. Put the size column back and both halves reverse: on stones under 2 cm open surgery took 93.1% (81/87) against 86.7% (234/270), and on stones of 2 cm or more 73.0% (192/263) against 68.8% (55/80). It is ahead in every group and behind in the total. (This exhibit is our own demonstration, drawn in the house style of a hospital programme review from the published table — no such slide was published.)

03The fix

Show the comparison broken out by the thing that drives the outcome, and every group with it. Here that is one line: open surgery leads by 6.4 points on the small stones and by 4.3 on the large ones, and the only row where it trails is the one that pools them. If a single number is genuinely needed, standardise it — score both procedures against one shared mix of cases so they are answering the same question. The whole series ran 357 stones under 2 cm to 343 of 2 cm or more, 51% to 49%; at that mix open surgery comes out at 83.3% (0.51 × 93.1 + 0.49 × 73.0) and percutaneous at 77.9% (0.51 × 86.7 + 0.49 × 68.8). A 4.6-point lead becomes a 5.4-point deficit and not one patient’s outcome has changed — only the question. The last part of the fix is knowing what the split does not buy you. This was a retrospective hospital series, not a randomised trial; surgeons chose the procedure with the stone in front of them, so stratifying by size undoes the one thing about these patients that was written down, not the things that weren’t. The honest chart does not crown open surgery. It shows that the pooled total was never evidence either way — and the tell that would have caught it is a question you can ask of any headline rate: what varies inside these two groups that also predicts the outcome, and was it split evenly between them?