Humans Hallucinate More Than LLMs

Earlier this year I was in a meeting with our media buying agency, and we were all reciting the same stat about our audience. Everyone on our side said it the way you say your own phone number.

I said it too. I have put it in decks. I have said it to people who then said it to other people.

Nobody knew where it came from. It predated everyone in the room. You heard it your first week and were repeating it by your second. It sounded true, but it was also the type of metric that was tough to verify.

Then someone from the agency stopped us. "Are you sure?" And then: "I think you need to separate myth from fact."

Agencies are generally paid to nod, so this landed.

It was the first time it occurred to me that our favorite fact might be a ghost story, or at least a hallucination. Something told around the campfire by people who heard it from a guy who swears he saw it.

So we went digging. Sure enough, it was absolutely untrue.

Every company is a little haunted

A ghost story survives because it has no source. No source means nothing to audit, nobody to blame, no export to pull. It just floats through onboarding decks, rattling chains.

"Our customers don't buy on mobile." "We tried that in 2021, it didn't work." etc. etc.

I used to think these were harmless. Then I started handing Claude actual source data all day (exports, APIs, MCP connections) and watched what happens when a real number walks into the haunted house.

Good numbers get interrogated

When the result is good, people get suspicious. Are you sure? That sounds too good to be true.

And if they find out AI touched it, the suspicion doubles. I have watched a number go from exciting to contaminated in one sentence, and the sentence was "I had Claude pull it."

Bad numbers get the source blamed

When the result is bad, nobody argues with the math. They go straight for the source. The tracking is broken. The export is missing something. The source is wrong, and the source's source is probably wrong too. Or the AI doesn't understand a nuance that is buried away in someone's head. In other words, another human hallucination.

I wrote about this crowd in August. They're still out there. Doing great.

So good numbers are too good, and bad numbers come from a bad source. Now notice which number never got any of this treatment.

The ghost story.

The stat we all recited for years faced zero cross-examination. No one asked for the export. It had no source to blame and no AI to be suspicious of. It got through on vibes and seniority.

That's the evidentiary standard in most companies. Fresh data from the actual system is guilty until proven innocent. A thing said in 2019 by somebody who no longer works at the company is innocent forever.

Claude wasn't at the campfire

Here's what changes. The model never got onboarded. Nobody told it the story. You hand it the order export, ask who actually buys this stuff, and it tells you what's in the export. It has no idea it just contradicted slide 4 of every deck the company has made.

Every company that plugs a model into its real data is about to find out which of its facts are load-bearing and which are ghosts. It's going to be an uncomfortable few quarters.

The industry defines hallucination as confidently stating something with no basis in the source. By that definition, I hallucinated in that agency meeting. Fluently. Citing colleagues who were also hallucinating.

I have still never seen a model invent a number when handed real source data. My count of humans doing it in meetings, me included, is considerably higher than zero.

Models use math, not myth, and they really want to deal in fact, not fiction. Give a model good data and it doesn't hallucinate.

People do. We just call it institutional knowledge.