Cherry-picking

The filter that invents a trade-off

Two things that go together, sliced until they look like a choice.

a.k.a. Berkson’s paradox · collider bias · conditioning on a collider · selection-induced correlation · explaining away · collider stratification bias · the admitted-sample trade-off · the budget trade-off

Some things are worth holding constant and some things are made of the very quantities you are comparing. Price is made of size and quality; getting hired is made of experience and interviewing well; reaching significance is made of sample size and effect size. Slice a dataset by one of those and you have not controlled for anything — you have required the parts to compensate for each other, because the only way into the slice with less of one is more of the other. The two then appear to trade off inside the slice however unrelated, or however positively related, they are outside it. Nothing is exaggerated, nothing is omitted from the chart, and the correlation is computed correctly. The finding is simply about the entry rule rather than about the world.

How to spot it

Ask what a row had to do to be in this table, then ask whether the two columns you are comparing helped it do that. If they did, the relationship between them is partly the entry rule.
A trade-off between two things that are both plainly good, inside a group that was selected — the funded start-ups, the admitted students, the hired candidates, the published studies, the homes you can afford.
The word “among”. Among Olympians, among hospital patients, among best-sellers: the qualifier names the filter, and the filter is doing the arguing.
Draw the two-by-two and find the empty corner. A genuine trade-off thins out both extremes; a filter deletes exactly one corner — the one that could not clear the bar — and sometimes the opposite one too.
Tighten the slice and watch. If narrowing the band deepens the negative relationship and widening it brings the positive one back, the relationship is a dial and the filter is the hand on it.
Check the direction of the arrows. If the thing you sliced on is caused by your two variables rather than causing them, you have conditioned on a collider and manufactured the result.
A “surprising” negative correlation in an elite or pre-qualified population, reported without the unfiltered population beside it.

The fix

Take the filter off and look, because the unfiltered scatter with the filter drawn on it is the whole explanation in one picture: the slice is a diagonal, and the corners it deleted are visible sitting outside it. Where the filter cannot be lifted — the rows that failed to convert were never recorded, the rejected applicants were never measured — the direction is still knowable in advance, which is the most useful thing about this one: selecting on a common effect pushes the association between its causes negative, so you can say which way a filtered number leans before you have seen it — and why a tighter slice makes a surer-looking trade-off. Say what a row had to do to be in the table, in a line on the chart, the way you would name a unit. And keep the distinction from Simpson’s paradox sharp, because the two look identical on the page and want opposite treatment: Simpson’s paradox as you usually meet it comes from a confounder, a cause the two variables share, and you fix it by breaking the comparison out by it; a collider is an effect they share, and breaking the comparison out by that is what caused the trouble. The test is a question about direction rather than about statistics — does this thing produce my two variables, or is it produced by them? Finally, when the slice genuinely is the question, as it is for anyone with a fixed budget or a fixed shortlist, keep the finding attached to the slice: it is a fact about the bracket, not about the population, and it will not survive the bracket moving. Be warned that the usual reassurances will not catch it — split the filtered data by year, drop its outliers, rerun it on a cleaner subset, and every check comes back agreeing, because the filter is still there in every one of them.

In the gallery

Dark navy estate-agent slide, masthead “Prairie & Vine Realty · Buyer briefing · Ames, Iowa · 2,930 recorded sales, 2006–2010”, headlined “The $200,000 law: every 1,000 square feet costs you twenty years”. A subhead says this is every residential sale in the assessor’s record that closed between $180,000 and $220,000 — 485 homes, all of them, nothing sampled and nothing smoothed; each dot is one house that actually changed hands, floor area across and year of construction up; both scales are linear and unbroken, neither truncated nor logged, no second axis, and every one of the 485 sales is on the panel and in the tables; the correlation is −0.34 and the least-squares line falls 20.1 years for every 1,000 square feet added, so in this bracket space and newness are not two things you shop for but one thing you divide. Down the left, under the label “Homes sold $180,000–$220,000 · floor area vs. year built · n = 485”, a scatter plot runs from 900 to 5,200 square feet across and 1870 to 2015 up, gridlines every 1,000 square feet and every 20 years, both labelled. Four hundred and eighty-five gold dots cluster densely between about 1,000 and 2,600 square feet, thickest near the top around the year 2000, with a long tail of older houses reaching down past 1900 and a handful of very large homes out to 5,095 square feet. A thick white least-squares line runs from the upper left to the lower right, marked at two points: “1,200 sq ft → built 1997” and “2,400 sq ft → built 1973”. In the lower right of the panel, in gold, “r = −0.34”, above the lines “485 sales · slope −20.1 yr per 1,000 sq ft” and “every sale in the bracket, 2006–2010”. A note beneath records the axis ranges and that the fit is ordinary least squares on all 485 points. The middle column holds three tables. “Your budget, cut five ways by size · 97 homes in each fifth · 485 divides exactly” gives floor-area range, homes, median year built and median price: 988–1,364 sq ft, 97, 2004, $193,879; 1,364–1,516, 97, 2001, $192,000; 1,522–1,674, 97, 1998, $195,800; 1,675–1,851, 97, 1998, $196,000; and 1,855–5,095, 97, 1978, $202,500 — captioned as twenty-six years between the smallest fifth and the largest for a median price difference of $8,621. “The same 485 homes, cut the other way · by decade built” gives before 1950, 35 homes, median 1,824 sq ft, $200,000; 1950s–60s, 48, 1,689, $191,000; 1970s–80s, 83, 1,621, $192,000; 1990s, 110, 1,658, $194,500; 2000s, 209, 1,504 sq ft, $199,000, captioned that cut by age the same law appears from the other side, the oldest homes running 320 sq ft larger than the newest. “The law, year by year · the bracket, split by year of sale” gives homes, correlation and slope for each year: 2006, 102, −0.27, −21.8; 2007, 123, −0.24, −11.3; 2008, 99, −0.46, −40.4; 2009, 101, −0.52, −30.1; and 2010 to July, 60, −0.45, −26.3 — five separate years, five negative numbers, no exceptions. Down the right-hand side, a gold-bordered card carries “−20.1 years” over “per 1,000 sq ft added · least squares, 485 sales · r = −0.34 · every sale in the bracket included”. A card headed “What the line prices” reads the fit at 1,200 sq ft (1997), 1,800 (1985) and 2,400 (1973), concluding that doubling your floor area buys a house a quarter of a century older. A card headed “We tried to break it” reports that dropping the five largest houses in the file, which the data’s own author suggests doing, gives −0.44, and keeping only ordinary arm’s-length sales gives −0.38, so the law survived everything they could think of. A card headed “Advice to buyers” tells readers to decide which kind of household they are before viewing, because the market has already decided they cannot be both. A card headed “Why we trust this number” stresses that these are recorded transactions from the city assessor, every residential sale in the file that landed in the bracket, nothing sampled, smoothed or trimmed. A final card, “Basis of measurement”, names the Ames City Assessor’s Office, the three fields used, and the 485 sales out of 2,930, or 16.6%, that closed in the bracket. Three ticked boxes run along the foot: “Nothing dropped”, “No axis tricks” and “One source, one definition — one assessor’s office, one continuous window from January 2006 to July 2010”. A footnote credits the Ames Housing data set of Dean De Cock, Journal of Statistics Education 19(3), 2011, gives the file and retrieval date, the bracket definition, r = −0.3354 and the OLS slope, records that the 2,930 are the residential sales De Cock retained from an original 3,970 with repeat sales de-duplicated, notes that sales run January 2006 to July 2010 and how medians of even-sized groups are rounded, and states that the slide is our own demonstration drawn in the manner of an estate agent’s buyer briefing, that the firm is invented, and that every figure on it is real.

The space-or-newness trade-off that the budget invented

The filter that invents a trade-off