Framing & context

The rate at the wrong grain

Ninety-nine per cent per word is eighty-six per cent per sentence.

a.k.a. per-word accuracy · word error rate · the fine-grained denominator · per-item versus per-episode · uptime nines · five nines · parts per million · the compounding rate · accuracy per unit · the small denominator

A rate is a fraction, and the thing in its denominator is a choice. Pick a small enough unit — a word, a character, a packet, a minute, a part — and any error rate becomes a number just under 100%, because the smaller the unit, the less of anything can go wrong inside one of them. But nobody consumes the small unit. A reader reads sentences, a viewer watches programmes, a customer places an order, a patient has a stay, and each of those is made of many small units in a row, every one of which has to come out right. So the rate the reader cares about is the per-word rate compounded across the length of the thing, and that number can be worlds away from the headline while both are computed from the same measurements with no error anywhere. It is not a distortion of the data and it survives every audit of the drawing: the axis can start at zero and run to a hundred, every case can be present, nothing smoothed or dropped, and the picture is still answering a question nobody asked. Two features make it hard to catch. A rate near 100% has almost no room left to move, so real differences arrive as hundredths of a point and read as noise, while the same differences in the complement — the error rate — are whole multiples of each other. And the compounding is exponential, so the gap between the quoted rate and the experienced one widens with the length of the thing, fastest exactly where the stakes are highest.

How to spot it

Read the denominator out loud. “Per word”, “per character”, “per packet”, “per part”, “per minute”, “per component” — then ask what unit the person affected actually receives, and how many of the small ones are inside it.
Do the compounding in your head. A rate p per unit across n units leaves (1 − p)ⁿ intact, which is roughly 1 − np while np is small: 1% per word over a 15-word sentence is about 14% of sentences touched, and over a 100-step process it is 63%.
Any rate quoted with a 9 in front of it. 99%, 99.9%, “four nines”, “six sigma”, “parts per million” — the nines are a sign that the unit was chosen small enough to produce them, and the useful question is always how many of that unit make up one of the things that has to work.
Plot the complement instead. A series moving from 99.2% to 99.6% is a flat line at the top of any zero-based axis; the error rate behind it halved, which is the largest change the measure can record.
Watch for a threshold set in the fine unit and a consequence felt in the coarse one. “98% accurate” as a pass mark says nothing about how often a viewer, a reader or an operator meets the failure, and the two can be moving in opposite directions.
Check whether the errors are weighted. A score that discounts small errors and counts serious ones in full is not counting errors at all; its complement is a weighted quantity, and the number of actual mistakes is larger than it by however much the discount was.
Ask what fell out of the denominator on the way in, because the two failures ride together: a per-word accuracy computed over the words that were delivered has no opinion about the words that were dropped. That second half belongs to the late denominator, and a rate can be at the wrong grain and start counting late at the same time.
A familiar grain is the fastest test there is: convert the same percentage into time. 99% of a year is 3 days 15 hours of outage, 98% is 7 days 7 hours, and a figure that sounds like an A in one unit sounds like a crisis in another without a single number changing.

The fix

Quote the rate at the grain of the thing that has to come out right, and name the grain the way you print a unit. Where the natural unit is small because that is how the measurement is made — and often it is, since a word error rate is exactly the right instrument for comparing two recognisers — keep it, and publish the compounded figure beside it: per word and per sentence, per step and per run, per component and per assembly. The second number is one line of arithmetic and it is the one the reader was trying to work out. Plot the complement rather than the rate, on a zero-based axis, because errors per hundred words has room to halve where 99.2% has nowhere to go; the two are the same measurement and only one of them can show a change. Put a count next to the ratio, since a count is already at a grain a person can hold — fifty errors in a ten-minute clip settles what 98.55% leaves open. Be explicit about what the compounding assumes, because independence is rarely true: real errors cluster, which packs more of them into fewer items and makes the naive calculation an upper bound on how many items are touched, so give the bound and say which way it leans rather than pretending to a precision the assumption cannot carry. And when a threshold is being set, set it in the unit the consequence lands in. A pass mark written in the fine grain is a pass mark nobody has costed, and the whole trick lives in the gap between the unit that was measured and the unit that was lived in.

In the gallery

Dark navy scorecard slide. The masthead reads “Access services quality board · quarterly scorecard” on the left and “84 programmes · five broadcasters · Ofcom fourth sampling exercise · April–May 2015” on the right. The headline is “Live subtitling passed, and it is getting better.” A bordered gold chip beneath it reads “Demonstration · drawn by us from a published regulator dataset”. The standfirst says every programme in the regulator’s final sampling exercise is on the slide — 84 ten-minute clips, five broadcasters, three genres — each scored on the same published accuracy model and re-scored independently by a university review team whose figures differed from the broadcasters’ own by 0.07 points on average; that the accuracy axis is linear, runs from zero to 100%, and nothing is truncated, broken or clipped; and that four rounds rose every time, 98.28, then 98.34, then 98.37, and now 98.55% accurate, with 80% of programmes clearing the 98% acceptability threshold, the highest the project has recorded. The main panel, “Accuracy across four rounds of measurement · all programmes and news bulletins”, subtitled “Average NER accuracy rate, five broadcasters · percentage axis linear and starting at zero · every programme measured is in the average, none dropped, nothing smoothed or rescaled”, is a grouped bar chart on a percentage axis running 0 to 100 with gridlines every 20. Four pairs of bars stand at 1st round Oct–Nov 2013, 2nd round Apr–May 2014, 3rd round Oct–Nov 2014 and 4th round Apr–May 2015: a blue bar for all programmes reading 98.28, 98.34, 98.37, 98.55 and a gold bar for news bulletins reading 98.49, 98.62, 98.86, 99.02, each value printed inside the top of its own bar. Every bar reaches almost exactly the same height, a hair under the top of the frame, and a green rule labelled “98% — acceptable quality threshold” runs across just below them. A line under the plot reads “Every bar is drawn against the same zero-based percentage axis”. Three panels sit below. “Programmes clearing the threshold”, subtitled “Share of programmes scoring 98% or better”, is four bars on a 0–100% axis reading 76%, 74%, 77% and 80%, the fourth picked out in gold. “Average accuracy by genre”, subtitled “Fourth round, all broadcasters”, is three bars on a 0–100% axis reading news 99.02, entertainment 98.31 and chat shows 98.26, with the green 98% rule across the top. “The scorecard, tabulated”, subtitled “Grades read off the reviewers’ published bands”, is a monospace list — news bulletins 99.02% very good, entertainment 98.31% acceptable, chat shows 98.26% acceptable, then a rule and all programmes 98.55%, threshold 98.00%, margin +0.55 pts, status PASS, exceptions raised none, actions arising none — under a note observing that every genre clears the threshold, that the headline figure has risen at every round, that the thinnest margin on the card is more than three times the average 0.07-point gap between the broadcasters’ own scoring and the reviewers’, and that the card is carried forward to next quarter unchanged. Six cards run down the right. “The finding, in one figure” prints a large gold 98.55% beside “average accuracy across the 84 programmes — the highest of the four rounds”. “Read-out for the board” says quality is good and improving on every measure the model reports and recommends keeping the present measurement régime and reporting the headline figure each quarter. “What one point is” says the unit is one word of subtitle and gives the score as (N − E − R) / N, where N is the number of words in the subtitles and E and R are editing and recognition errors, weighted 0.25 if minor, 0.5 if standard and 1 if serious. “The set as read” tabulates programmes 84, broadcasters 5, genres 3, material 14 h, subtitles about 23,000, words spoken about 145,000, words subtitled (N) about 112,000, mean reviewer gap 0.07 pts, programmes dropped 0. “Why nothing here is bent” notes one model, one unit, one population, a linear percentage axis from zero, the threshold drawn where the reviewers set it and every programme measured in the average, ending “Whatever is wrong with this slide, it is not the drawing.” “Where the threshold came from” records that the reviewers identified 98% as the rate above which live subtitling can be considered acceptable, with 98.5–99% good to very good and 99–99.5% very good to excellent. Footer chips read “Every programme measured”, “One model, one unit”, “Linear axis from zero”, “Threshold as published” and “Nothing dropped”, above a footnote naming the source and stating that the board, the grades, the status line and the read-out are invented while every figure behind them is the regulator’s own.

The 98.55% pass that is an error every twelve seconds

The rate at the wrong grain The late denominator