What I got wrong

This page is not an apology. It is the most useful thing on the site.

Every error below was found after the wrong number had been written down as though it were true, and two of them had already reached a draft that was close to being published. They are listed with the number that was wrong, the number that is right, and the specific mistake that produced the gap.

If you find a further one, that is the point of publishing the queries.

The substring bug

The query matched topics with LIKE '%ai%' — the substring "ai", anywhere in the string. It matched:

The bug did not produce an obviously broken chart. It produced a plausible, boring one: AI drifting from 60% to 95%, up and to the right, exactly what everyone expects. That is what made it survive review. The corrected curve is both more interesting and harder to argue with — AI declines for four years to a floor of % in 2022, then jumps × the year ChatGPT arrives.

The rule that came out of it: never match a controlled vocabulary with a substring. Exact string equality or a word-boundary regex, always.

The same class of error came back twice more, which is why the rule is stated so flatly. A transcript probe using the bare string "the spec" matched "the specific" and "the spectrum" at nearly a 100% false-positive rate. A probe on "hand off" as two words matched "chop your hand off."

The extractor artifact

This one is worse than the substring bug, because the query was correct. The database really does contain only 26 claims tagged survey programming / survey hosting platforms, flat across eight years.

But that number measures where the extraction model's controlled vocabulary put things, not what was said. The vocabulary had no good slot for a discussion about scripting a questionnaire, so those discussions were filed under automation / research operations and ai / artificial intelligence instead. The count of 26 was a fact about the schema.

Grepping the raw transcripts directly told a completely different story. That is why this site has two independent instruments and why the regex probes exist at all.

The rule that came out of it: any finding of the form "X is never discussed" must be checked against the raw text before it is believed. A model-based extraction can only tell you about the categories you gave it.

The claim that was too strong, twice

The corrected finding above was itself first written in a version that was too strong: that the spec-to-code handoff is discussed zero times. The original probe used four narrow phrases and returned a clean 0% everywhere, which felt like a result.

Widening it to how people actually talk — "the questionnaire doc", "Word doc", "handoff" — showed the topic is discussed. Rarely, at 0–2% of episodes, and flat, but not never. "Rare and flat" is a weaker sentence than "never" and it is the one the data supports.

A third version of the same mistake was caught by sampling matches by hand rather than trusting the count. The probe for the survey programmer originally included the bare word programmer, which matched software engineers:

"imagine how many programmers you can hire" "a set of programmers about to start coding"

Roughly half the matches were false, on the single probe the whole finding rested on. It is now split in two: a narrow pattern for the survey-programming role, which is what any published figure uses, and a broad one kept only as a baseline. The broad one is not shown on this site.

The rule that came out of it: read a sample of what a probe actually matched before quoting its total. A clean-looking number is not a validated one.

The duplicate-episode bug that would have halved a show

Not a published error — this one was caught during a routine refresh — but it is the closest call in the project.

One podcast's host migrated its audio to a new URL scheme. Because the episode audio URL was the deduplication key, all 176 of that show's existing episodes looked brand new, and 143 duplicate rows were inserted, orphaned from their transcripts.

Nothing errored. Nothing looked wrong. It would have silently inflated the episode counts for that show and roughly halved every share-of-episodes figure for it from 2021 onward — including the AI trend that this site's second beat rests on. The deduplication key now falls back to podcast, title, and publish date, and a duplicate check runs after every feed refresh.

The rule that came out of it: the dangerous data bugs are the ones that do not raise an error.

An entire analysis that was thrown out

Conference agenda data was scraped for 17 industry events — 1,346 session rows, 884 distinct sessions. The plan was to compare when a topic first appeared on a podcast with when it first appeared on a conference agenda, and see which one led.

It produced a beautiful result: podcasts lead conferences on nearly every topic, often by years.

It is meaningless. The conference data starts in 2021 and the podcast data starts in 2018, so any topic already being discussed in 2018 is guaranteed to "lead" by three years, whether or not that is true. The finding measures where the two datasets happen to begin. It is not fixable without pre-2021 conference agendas, which are not available.

Five thousand rows of analysis are sitting in the database, unused, and are deliberately not exported to this site so that no chart here can accidentally reach them.

The rule that came out of it: when two series start at different times, the earlier one wins every "first" comparison by construction.

Smaller things

• The AI figure moved once more, for a defensible reason, and it is still a change. Counting only episodes whose extraction tagged them with AI gives 67% for 2025. Counting an episode when the extraction tagged it or attributed a claim in it to AI gives 71%. The second is the definition this site uses, because tag-only matching misses exactly the episodes the extractor artifact above was about — but a four-point swing produced by a definitional choice belongs on this page, not buried in a footnote. The rule is stated in full on the methodology page and shipped inside the data as a match_rule field.
• Sponsor reads were counted as industry discourse. The most-mentioned company in the corpus was a show's own sponsor, read aloud in 183 of its episodes. Sponsors are now excluded per show.
• Case variants split real topics. The extraction emitted both qual / qualitative methods and Qual / qualitative methods as separate strings, along with truncations like privacy / data regulation for privacy / GDPR / data regulation — 108 episodes' worth, silently dropped until an audit caught it. Variants are now folded through an explicit, hand-checked alias map rather than a fuzzy rule.
• A pooled trend was published before it was checked per show. The AI trend turned out to replicate independently inside all four shows, so it survived — but that check was run after the chart was drawn, not before. It is now shown on the front page as part of the finding rather than kept as a private reassurance.
• Transcription silently lost three episodes. A subprocess wrapper decoded audio-tool output as strict UTF-8; some MP3 ID3 tags are Latin-1, so a decode error was raised after a successful transcription and the good result was discarded. Three episodes were missing from the original corpus for months before anyone asked why the total was 1,031 and not 1,034.