The canary that deployed nothing and still won on five paths out of 24

Dark navy release-review slide, kicker “Edge platform · Canary analysis · Network stack 4.18 · Measurement fleet”, headlined “The new stack is faster to the root servers — 5 paths better, every one significant”. A subhead says stack 4.18 went to a canary half of the fleet, 7,075 probes, while the other 7,074 kept the old stack as baseline; that each probe pinged the root name servers three times in one four-minute round on 29 September at 12:00 UTC; and that the chart shows median round-trip time per root for each half, from real measurements, on a zero-based axis, with a two-sided Mann–Whitney test. A badge reads “Demonstration · drawn by us from public data”. The main panel, titled “Median round-trip time, milliseconds · IPv4 · significant paths (p < 0.05)”, is a paired horizontal bar chart on an axis from 0 to 35 ms, blue for baseline and gold for canary: m-root 30.61 against 28.94, −5.4%, p = 0.0018; l-root 26.38 against 25.08, −4.9%, p = 0.0448; j-root 19.37 against 18.67, −3.6%, p = 0.0206; i-root 17.91 against 17.41, −2.8%, p = 0.0379; h-root 23.29 against 22.72, −2.5%, p = 0.0092. To the right, a big gold figure “−1.67 ms” is captioned as m-root, 5.4% faster at p = 0.0018, with the canary leading on every path that moved; a read-out says all five significant differences point the same way, nothing significant went against the canary, and the new congestion and queueing defaults appear to be doing what the design doc promised; a recommendation says to promote the stack to the whole fleet this week and keep m-root as the headline latency check; and a table repeats the five paths’ medians and p-values. The footer says 24 paths were compared, 12 root servers over IPv4 and IPv6, and only the five with p < 0.05 are shown; that each probe’s value is the average of three ICMP pings, with medians taken across probes; and that the slide is a demonstration in which no stack was deployed, the halves were drawn by a random number generator, and every round-trip time is real. — Showing the misleading chart

Cream-background grid headlined “Split a fleet by coin, test 24 paths, and something ‘wins’”. A subhead says the slide’s halves were drawn at random, so the stack is a label on a coin toss; that nineteen more coin-toss splits of the same 14,149 probes, run through the same 24 tests, give 11 of 20 splits with at least one significant path; and that across 1,000 such splits the average is 1.19 hits and 55.8% find at least one. The grid has twenty rows, one per split, and twenty-four columns, the root servers a to m without g (whose measurements return no data from the Atlas API), first over IPv4 and then over IPv6. Each cell is a small grey dot, or a large dot where the test came out at p < 0.05: ochre where the “canary” half was faster, blue where it was slower. In split 1 the small dots are tinted too, pale ochre or pale blue by which half was faster: all twelve IPv4 columns lean the canary’s way, while IPv6 is split six and six. A count of hits runs down the right. Split 1, highlighted and labelled “the slide”, has five ochre dots in a row on the IPv4 side, at h, i, j, l and m. The other rows are mostly empty: splits 2, 4, 5, 6, 8, 12, 16, 19 and 20 have none, splits 3, 9, 10 and 13 have one each, splits 14, 15, 17 and 18 two each, and splits 7 and 11 three each, scattered across both address families and both colours. A total line reads “20 splits × 24 paths: 23 hits in 480 comparisons (4.8%) · chance alone expects 24 · 11 of 20 nothing-changed canaries flag at least one path”. Two side panels explain that a test at p < 0.05 calls a difference that isn’t there one time in twenty, so a dashboard of 24 is 24 chances; that split 1’s five hits share the same probes, its canary half having drawn slightly lower round-trip times to all twelve IPv4 roots (a-root by just 0.01 ms), so one coincidence is counted five times; and that a primary metric named in advance, a count of every comparison, an A/A run for calibration and a fresh split on new hosts, where a real effect tends to turn up again, would have tested the stack — noting that a correction caps false alarms rather than ending them, since m-root’s p = 0.0018 even clears a Bonferroni bar of 0.05/24 = 0.0021, as 4.0% of 1,000 nothing-changed splits also did. The footer adds that split 1 was the first and only split drawn before the tests were run. — Showing the misleading chart

Be the first to star this exhibit

A canary-analysis slide we drew from one round of real RIPE Atlas pings: a “new network stack” on half of 14,149 probes is faster to five root servers, every gap significant, m-root by 1.67 ms at p = 0.0018. The halves were drawn by a random number generator and nothing was deployed. The slide shows the five paths that cleared p < 0.05 out of 24 it tested, and 11 of 20 coin-toss splits of the same fleet find a winner of their own.

RIPE NCC, RIPE Atlas — built-in ping measurements to the DNS root servers (twelve roots: IDs 1001, 1004–1006, 1008–1013, 1015 and 1016 over IPv4 and their 20xx twins over IPv6; g-root’s 1014 and 2014 return no data), one round, 29 September 2026, 12:00–12:04 UTC, 2026Added October 5, 2026

01The claim

The new network stack makes the fleet faster. A canary-analysis slide of the kind a platform team might put in a release review, which we drew from real measurements: stack 4.18 goes to a canary half of a 14,149-probe measurement fleet, the other half keeps the old stack, and in the same four-minute round each probe pings the DNS root servers three times, over IPv4 and, for the half of the fleet that has it, over IPv6. The canary half reaches five root servers faster than the baseline, every gap significant on a two-sided test. m-root, the biggest, is 1.67 ms faster, 28.94 against 30.61 ms (5.4%), at p = 0.0018. All five point the same way and none goes against the canary, on a zero-based axis with every median printed. Recommendation: promote it to the whole fleet.

02The trick

Nothing was deployed. We split the probes into two halves with a random number generator, drawn before any test was run, so “stack 4.18” is a label on a coin toss. Every round-trip time is real, and every p-value is correct for its own comparison. The mistake is in which comparisons reached the slide. The review tested 24 paths, twelve root servers over IPv4 and over IPv6, and drew the five that cleared p < 0.05. A test at that threshold calls a difference that isn’t there one time in twenty, so 24 tests on two identical halves expect about 1.2 hits, and better-than-even odds of at least one. The small print says “24 paths compared”, but the bars show only the winners. The five agreeing with each other is not extra evidence either. They are the same probes measured five times: by luck of the draw the canary half came out with slightly lower round-trip times to all twelve IPv4 roots (a-root by just 0.01 ms), so one coincidence produced five hits. That is what correlated metrics do: they don’t raise false alarms more often, they bunch them, so a split either finds nothing or finds several at once. Five or more hits turned up in 4.5% of 1,000 coin-toss splits of the same fleet, about seven times what 24 independent tests would give. On IPv6 the same split is a coin toss, canary faster to six roots and slower to six, none significant. This split, the first and only one we drew, happened to be a loud one.

03The fix

Run the same analysis where nothing changed. Draw nineteen more coin-toss splits of the same 14,149 probes, run the same 24 tests on each, and lay all 480 results out in one grid, the misses next to the hits. That gives 23 hits (4.8%, against the 24 chance expects), and 11 of the 20 splits flag at least one path, scattered across roots, address families and both directions. Across 1,000 splits the average is 1.19 hits, 55.8% find at least one, and each single test fires 4.9% of the time, exactly the rate it promises. A correction helps but doesn’t settle it. m-root’s 0.0018 even clears a Bonferroni bar of 0.05/24 = 0.0021, which 4.0% of the nothing-changed splits also cleared, because a correction caps how often a family of tests raises a false alarm and doesn’t stop it happening. What tests a stack is naming one primary metric before the rollout and reporting it first, counting every comparison shown, calibrating against an A/A run on the same fleet, and checking a surprising win on a fresh split of new hosts, where a real effect tends to turn up again, usually smaller, and a lucky draw tends not to.