The ±1-point poll that missed by 61

Dark navy briefing slide, masthead “The Mailbag Desk · National Parenthood Survey · 1976”, headlined “Seven in ten parents would not do it again”. A subhead reads: More than 10,000 readers answered a single question — “If you had it to do over again, would you have children?” Every reply was counted. Nothing was weighted, adjusted or estimated. Under the label “Share of responding parents · by answer given”, two columns stand on a zero-based axis marked 0%, 25%, 50%, 75% and 100%: a tall gold column at 70% labelled NO, “would not have children if they had it to do over”, and a shorter blue column at 30% labelled YES, “would have children again”. A short white whisker sits on top of each column, annotated “± 0.9 pts”. A strip beneath the axis reads “Replies counted — more than 10,000 — seven times the sample of a national opinion poll”. Down the right, a card carries 70% in large white type over the words “of parents say they would not have children if they had it to do over again”. A second card, “The sample, in numbers”, lists replies received 10,000+, margin of error ± 0.9 pts, confidence 19 times in 20, replies discarded none. A gold-bordered note reads “The arithmetic leaves almost no room for doubt. At ten thousand responses the interval is under a single point,” with a small grey disclosure beneath it: “Respondents were readers who chose to write in.” Three ticked boxes run along the bottom: “Full axis — 0% to 100%, no truncation, no log scale, no second axis”, “Every reply counted — no weighting, no adjustment, no estimation, nothing set aside”, and “Uncertainty shown — the 95% interval is drawn on both bars; at this sample size it is thinner than the line”. A footnote credits Ann Landers’ syndicated column of 23 January 1976 and her account of the survey for Good Housekeeping, June 1976, notes that the margin of error is the textbook 95% interval for a simple random sample of n = 10,000 at p = 0.30 computed by us, and calls the slide a Mailbag Desk demonstration drawn by us. — Showing the misleading chart

The same question on a cream background, kicked “The same question · four samples · 1975–76” and headlined “The ±1-point poll missed by 61 points.” A subhead explains that four surveys asked American parents the same question within a year of each other: the two whose respondents chose themselves land 65 points apart, the two that drew their respondents at random land 3 points apart, and the smaller of those, 409 people, is nearer the national figure than 10,000 write-ins. The main panel, headed “‘If you had it to do over again, would you have children?’ · share answering yes · one zero-based axis, 0% to 100%”, is a dot plot in two blocks on a shared 0–100% axis. Under the heading “Respondents chose themselves — readers who wrote in”, a gold dot for Ann Landers, syndicated column, 10,000+ replies, sits at 30% yes, and a gold dot for Good Housekeeping, magazine sidebar, count not reported, sits at 95% yes. Under the heading “Respondents were drawn, then asked — random samples”, a blue dot for the Kansas City Star, Kansas City area, n = 409, sits at 94% with an error bar of ± 2.3 points, and a blue dot for Newsday, national, n = 1,373, sits at 91% with an error bar of ± 1.5 points. Below, a panel headed “The interval and the error, drawn to the same scale” lays a gold bar spanning 30% to 91% along the same axis, labelled “61 points — how far the write-in poll sat from the national random sample”, with a barely visible blue sliver at its left end labelled “± 0.9 — the margin of error it published, drawn to the same scale”. Its caption reads: the formula was applied correctly; it rests on one assumption, that the sample was drawn at random, and that is the assumption that failed — an interval prices sampling noise, it never prices its own assumptions. A gold-ruled strip beneath reads “409 drawn at random beat 10,000 who volunteered — by 58 points,” with the line “Precision and accuracy are different quantities, and only one of them improves when you collect more.” Five cards run down the right. “The columnist diagnosed it herself” quotes Ann Landers’ follow-up column of June 1976: “I believe the logical explanation for this phenomenon is (a) the hurt, angry and disenchanted tend to write more readily than the contented, and (b) people tell me things they wouldn’t dare tell anyone else.” “Sample size is not the variable” notes that 10,000+ write-ins landed 61 points off the national figure while 409 people drawn at random landed 3 points off it, and that more draws from the wrong population buy a sharper picture of the wrong population. “Two things moved, not one” records that the write-in pair also differed in wording and audience — the letter Landers printed recounted friends who resented their children, the magazine’s sidebar opened by quoting her “horror”, and it addressed mothers — so the 65-point gap is a warning rather than a measurement, and the random pair, asked plainly, is the anchor. “The same mailbag, now” lists the in-app rating prompt, the NPS pop-up, the post-incident survey, the speed test, the exit interview and the star rating, every one of them answered by whoever felt like answering. “The check that catches it” says to ask what a person had to do to appear in the dataset, whether doing it correlates with the answer, and then to look for the response rate rather than the sample size — and that if it is not published, that is the finding. A footnote gives the dates, sample sizes and figures for all four surveys, credits the compilation of the column and magazine citations to a University of Maryland, Baltimore County course reading, explains that the margins of error are the 95% intervals computed by us at each n and that none is published for the two write-in surveys because neither was a random sample, and says both drawings are our own. — Showing the misleading chart

Be the first to star this exhibit

In January 1976 Ann Landers reported that 70% of parents would not have children if they had it to do over again — more than 10,000 replies, a sample so large that the textbook margin of error comes to ±0.9 points. Five months later a national random sample of 1,373 parents answered the same question and 91% said yes. The interval was pricing sampling noise, and there was hardly any. The error was in who chose to write in.

Ann Landers, syndicated column, 3 November 1975 and 23 January 1976, and her report on the survey for Good Housekeeping, June 1976; Good Housekeeping reader sidebar, October 1976; Kansas City Star and Newsday random samples, 1976 — dates, sample sizes and figures compiled in “Ann Landers Survey on Parenthood”, course reading for POLI 300, University of Maryland, Baltimore County, 1976Added August 19, 2026

01The claim

Seven in ten American parents would not have children if they had it to do over again. The question went out in an advice column read in more than a thousand newspapers, more than 10,000 people answered it, and every reply was counted — no weighting, no adjustment, no estimation, nothing set aside. Draw that on a bar chart and the drawing is beyond reproach: a zero-based axis running the full 0% to 100%, no log scale, no second axis, and the 95% confidence interval printed on both bars, because at a sample of ten thousand the interval comes to ±0.9 points and the arithmetic leaves almost no room for doubt. Ten thousand replies is seven times the sample a national opinion poll would use. Whatever else this survey is, it is not small.

02The trick

Everything on that slide is true and the chart is drawn honestly. The problem arrives before the drawing, in how the sample was assembled: nobody chose these ten thousand people — they chose themselves. A write-in poll samples the population of readers moved to write in, and what moves a person to write in about parenthood is, fairly reliably, how parenthood went. Landers worked it out herself in a follow-up column: “the hurt, angry and disenchanted tend to write more readily than the contented.” That makes the bias directional, and it makes it immune to volume. Ten thousand replies is ten thousand draws from the wrong population, so the estimate gets sharper and stays wrong. Which is exactly what the ±0.9 fails to warn you about. The margin-of-error formula has one input, the sample size, and one assumption, that the sample was drawn at random. It prices sampling noise — how much a random draw of this size would wobble — and there is almost none at n = 10,000, so it reports a very small number and looks like a guarantee. It is not measuring the thing that went wrong, and no interval ever measures its own assumptions. The scale of the miss is the whole exhibit: Newsday commissioned a national random sample that same June and 91% of its 1,373 respondents said yes, against the column’s 30%. Sixty-one points, from a poll that published an interval of nine-tenths of one. And the demonstration runs the other way too — the Kansas City Star drew 409 people at random and got 94%, three points from the national figure on a sample one-twenty-fourth the size of the mailbag. (This exhibit is our own demonstration, drawn in the style of a 1976 newsroom graphic. The figures are the ones the four surveys reported.)

03The fix

Put all four samples on one zero-based axis and sort them by how the respondents got there, and the picture explains itself. The two write-in surveys — Landers at 30% yes, and Good Housekeeping, which reprinted the question for its own readers and reported 95% — sit 65 points apart. The two random samples sit 3 points apart, at 91% and 94%. Selection, not sample size, is what separates the pairs. Then draw the interval and the error on the same scale, because seeing a 0.9-point sliver against a 61-point span is the fastest way to learn that precision and accuracy are different quantities and only one of them improves when you collect more. Two honest caveats belong on the chart and are on ours. The write-in pair differed in wording and audience as well as in who replied — the letter Landers printed recounted friends who resented their children, the magazine’s sidebar opened by quoting her “horror”, and it addressed mothers — so their 65-point gap is a warning about opt-in data rather than a clean measurement of self-selection alone; the random pair, asked plainly, is the anchor. And Good Housekeeping never published how many people responded, which is worth noticing in its own right. The habit to keep is a question you can ask of any dataset in ten seconds: what did a person have to do to end up in here, and does doing it correlate with the answer? Then look for the response rate rather than the sample size, because the response rate is the number that bounds the damage, and if nobody published it, that is the finding. The mailbag never went away — it just moved into the rating prompt, the NPS pop-up, the post-incident survey and the exit interview, all of them answered by whoever felt like answering. Where you cannot draw a sample, label the chart with the population you actually have, and run a few hundred people chosen at random alongside it; the gap between the two will be the most useful number of your year.