The hay pilot that picked the counties already due a better year

Dark navy review slide, kicker “Range outreach desk · Forage recovery pilot · Year-one review · Montana”, headlined “The pilot counties grew more hay — in a year the rest of the state lost 17%”. A subhead says the nine lowest-yielding of the 42 counties USDA estimated in both 2016 and 2017, every one at or below 1.3 tons an acre of grass hay, joined a pilot of soil testing, rest-rotation planning and a reseeding cost-share, and that the chart shows USDA county survey figures for the 2016 and 2017 harvests, unsmoothed, on a zero-based axis. A badge reads “Demonstration · drawn by us from public data”. The main panel is a two-point slope chart of average county yield in tons per acre on an axis from 0 to 2.0: a blue line for the other 33 counties falls from 1.88 to 1.56, labelled −16.8%; a gold line for the nine pilot counties rises from 1.14 to 1.24, labelled +8.7%. To the right, a big gold figure “+25.5 pts” is captioned as the gap the program opened in its first season; a read-out says the pilot counties, which started furthest behind, were the ones that moved up; a recommendation says to enroll the nine lowest-yielding counties of 2017 next; and a table lists the nine counties’ 2016 and 2017 yields, from Carter at 0.95 and 1.05 to Prairie at 1.30 and 1.45, including Powder River falling from 1.00 to 0.70 and Golden Valley rising from 1.15 to 2.10. The footer notes that Wibaux, the lowest county of 2016 at 0.90, has no 2017 estimate and is left out, and says the desk, pilot and recommendation are invented and every yield is real. — Showing the misleading chart

Cream-background dot plot headlined “Pick the worst fifth of any harvest, and next year it ‘recovers’”. Fifty-four thin rows, one for each pair of consecutive Montana harvests from 1964→65 to 2017→18, labelled every five years. On each row a gold dot marks the next-year change in average county grass-hay yield for the lowest-yielding fifth of counties and a blue dot marks the change for every other county, joined by a thin grey line, on a horizontal axis from −60% to +160% with a darker line at zero. In almost every row the gold dot sits to the right of the blue, often far to the right — at about +114% after 1985 and +160% after 1988. Four rows, 1965→66, 1987→88, 1989→90 and 1999→2000, are annotated as years the bottom group trailed. The 2016→17 row is highlighted and labelled “the ‘pilot’ year: +8.7% against −16.8%”. The subhead says that in 50 of 54 years the picked counties beat the rest by a median of 18.2 points. Side panels explain that one season’s yield mixes what a county can reliably grow with that summer’s weather and survey noise, so the bottom of a one-season ranking is crowded with counties that had a bad summer; that 20 of the other 53 years opened a wider gap than 2016’s 25.5 points; and that a fair test would enroll a random half of the low-yield counties and look for a gap that holds up across seasons and beats chance. — Showing the misleading chart

Be the first to star this exhibit

A year-one review slide we drew from USDA county survey data: the nine Montana counties with the lowest grass-hay yields of 2016 “join a pilot” and gain 8.7% the next season while the other 33 lose 16.8%. Apply the same enrollment rule to every pair of harvests in the county series, 1964 to 2018, when no program picked anyone, and the bottom fifth beats the rest in 50 of 54 years.

USDA National Agricultural Statistics Service, Quick Stats — county survey estimates of hay (excluding alfalfa) yield, Montana, 1990–2018 harvestsAdded October 4, 2026

01The claim

The pilot works. A year-one review slide of the kind an extension office might circulate, which we drew from public survey numbers: after the 2016 harvest, the nine lowest-yielding of the 42 Montana counties USDA estimated in both 2016 and 2017 — every one at or below 1.3 tons an acre of grass hay — are enrolled in a forage recovery pilot. In 2017, a season that cost the other 33 counties 16.8% of their average yield, the pilot counties went the other way, from 1.14 to 1.24 tons an acre, up 8.7%. A 25.5-point gap in one season, on a zero-based axis, with every county’s numbers printed beside it.

02The trick

Every number on the slide is right, and the axis is honest; the mistake is in who got enrolled. One season’s yield mixes what a county can reliably grow with what that summer’s rain, heat and hail did to it, plus some survey noise. Rank counties on 2016 alone and the bottom nine are, disproportionately, counties that had a bad 2016. Their soils and meadows carry into 2017; their bad summer doesn’t, so the group drifts back toward its own normal whether or not anyone visits. The table on the slide shows how thin the evidence is: only five of the nine rose, Fallon stood still, Powder River, Mineral and Toole fell, and the group average is carried by Golden Valley jumping from 1.15 to 2.10 — without it the pilot counties are down 0.5%. Comparing against “everyone else” makes the gap look bigger than the bounce: most of the other counties’ −16.8% is just a bad year for the whole state (all 42 fell 13.2%), and the slide credits the difference to the pilot.

03The fix

Run the enrollment rule where there was no pilot. Take the lowest-yielding fifth of Montana counties (ties at the cutoff counted in) for every pair of consecutive harvests in the county series, 1964–65 through 2017–18, and compare their next-season change with every other county’s. The picked counties beat the rest in 50 of 54 years, by a median of 18.2 points; 2016’s 25.5 is unremarkable, and 20 of the other 53 years opened a wider gap on their own. A real test needs a comparison group chosen the same way: enroll a random half of the low-yield counties, and both halves are expected to bounce back alike, so a gap that holds up across seasons and is bigger than chance would give with a handful of counties a side is the program. At the very least, judge the pilot against the usual bounce, not against the rest of the state.