The biggest fool: six confident AI errors in one afternoon

Nije najveća budala onaj koji ne zna čitati, nego onaj koji misli da je sve ono što pročita istina.

It's a Bosnian saying, and for most of its life it was about newspapers, books and the people who quote them. Its warning was about volume: reading a lot doesn't make you right, and a person with a hundred books can be more confidently wrong than a person with none.

It reads differently now, because the thing that produces text has changed.

A book was written slowly, by someone, for a reason. It could be wrong, but it was wrong in a way that had an author and a date. The tools I use every day produce text instantly and in unlimited quantity, in a tone of total confidence, with no author and no position to defend. The old warning assumed you would have to go looking for things to believe. Now they arrive faster than you can check them.

"Don't trust AI" is advice most people agree with and then ignore, so here is a specific case: the research behind my post on serverless WebSocket costs, and the six things in it that were confidently wrong.

I was modeling what a WebSocket service costs on AWS, using real prices and the real architecture of a system I had built and run. I used an AI to do the arithmetic and structure the comparison.

The first error was the crossover point. The AI told me a plain server becomes cheaper at around 1,135 always-on connections. The number was precise and stated with confidence, and it came from comparing connection charges against a bare instance. That comparison ignored that both architectures need a connection store, and it left out a required networking component that costs more than the connections do at small scale. The figure was wrong by roughly half.

The second was its conclusion that messages were never more than 7% of the bill. That held for the scenarios it had chosen itself, which assumed absurdly low message volumes. With a realistic assumption, messages become the largest line on the invoice. The conclusion came from the setup, and it was presented as a finding about the world.

The third and fourth were the same mistake. Both times it found a gap in what I had told it, filled the gap with a plausible inference, and wrote that inference into a draft as my own statement. In one case the draft had me not knowing something I had known before I ever built the service. Both times the text read perfectly, and both times I would have published a claim about my own work that wasn't true.

The fifth was a comparison of serverless against a burstable instance, a cheaper machine with a fraction of the sustained CPU. The workload does a socket write per recipient and a TLS handshake per reconnection, so the comparison wasn't fair, and it made the server look better than it is. Nothing flagged it, and the table looked tidy.

The sixth was a recommendation to remove a cache expiry entirely. The AI hadn't noticed that the same expiry also released a record lock. Had I taken the advice, an admin whose browser crashed would have held a record locked for two hours.

Four of those six were on their way into a published post.

A random error is manageable and a patterned one is not. These six had a lot in common.

Each of them made the story cleaner: the tidy crossover number, the striking finding that messages don't matter, the comparison that produced a neat table, the simplification that removed a mechanism. None of them made the answer more complicated. All of them made it more presentable.

Each of them also sat in a gap I had left. I hadn't stated my assumption about session length, explained what the expiry was for, or spelled out part of my own history. Wherever I had said nothing, something plausible filled the gap, and a plausible answer looks the same as a correct one until you check.

None of them was hedged. There was no "this assumes", no "I'm not sure", no "you should verify". The wrong numbers were delivered in the same tone as the right ones, and there were many right ones, which is what makes the wrong ones dangerous. People throw away a tool that is wrong half the time. A tool that is right most of the time, in the same voice, teaches you to stop looking.

Asking again didn't help either. When I pushed back on the crossover figure, I got a revised number, delivered just as confidently. It was also wrong, in the same direction, for a related reason. Asking a model again only draws a second answer from the same distribution, so it can't verify the first one.

One afternoon is an anecdote. The same pattern shows up wherever someone has bothered to count.

Court filings: from one case to nearly sixteen hundred in three years

In June 2023 a New York lawyer stood in front of a federal judge and tried to explain six court decisions that did not exist. He had asked a chatbot for authority supporting his client's claim, and it produced confident citations with fabricated quotes and invented case numbers. The court fined him, his co-counsel and their firm $5,000.

At the time it read as a freak accident.

A public database maintained by the researcher Damien Charlotin now tracks court decisions where a party relied on AI-fabricated material. It held roughly 200 cases in mid-2025, 719 by January 2026, 1,227 by April and 1,598 by June 2026, with about six new cases a day. The largest single penalty is around $110,000. By mid-2026 a federal judge had canceled a trial and suspended lawyers on both sides of it.

The individual cases say more than the totals. The errors don't come only from consumer chatbots. Three attorneys at one large firm filed motions citing nine cases, and eight of those cases didn't exist. The citations came from the firm's own in-house AI platform. In another matter, counsel used two purpose-built legal research tools and still filed sixteen fabricated quotes.

Grounding the model in real documents doesn't fix it either. Stanford researchers found that retrieval-based legal tools, which search actual case law before they answer, still hallucinate 17% to 33% of the time.

The courts have converged on one sentence. Two separate appellate panels used nearly identical language: no filing should contain a citation the responsible attorney has not personally read and verified. In other words, the attorney has to read the case itself; reviewing it or checking it with a tool is not enough.

Package registries: invented names anyone can register

The finding that changed how I think about this comes from software.

Researchers from three American universities generated 576,000 code samples across sixteen language models, then checked every package the models recommended. Of the recommended packages, 19.7% did not exist: 205,474 distinct invented names. Commercial models averaged 5.2%, open-source ones 21.7%, and some configurations exceeded 33%.

That is one package in five, which on its own would be a nuisance.

The researchers then re-ran prompts that had produced a fake name, ten times each. Of the hallucinated names, 43% came back on every run and 58% appeared more than once. Only 39% never reappeared.

So hallucinations are reproducible. They come from the way a model answers a particular question, and the same question tends to bring back the same invention.

That makes them attackable. You don't have to guess what a developer might mistype. You run the model, watch what it invents, and register that name on the package registry before anyone else does. The attack has a name, slopsquatting, coined by the Python Software Foundation's developer-in-residence.

The names aren't obvious fakes either. Only 13% were typos of real packages. Roughly half bore no resemblance to anything real: they were plausible, well formed, on-brand for the ecosystem, and invented. A registry's similarity filter has nothing to catch.

Why repeatable errors are harder to catch

Reproducibility is what connects these numbers to my six errors.

If a model's mistakes were random, they would be survivable. You'd get a wrong number now and then, it would look out of place, and you'd catch it the way you catch a typo.

They aren't random. All of mine bent in the same direction, toward a tidier story. The package names come back identically across ten runs. The fake citations follow the shape of real ones closely enough to pass a lawyer's eye.

Systematic error is much harder to see than random error, because it is consistent with itself. Ask again and you get the same answer, which feels like confirmation and confirms nothing.

I caught all six because it was my system. I knew what the expiry was for. I knew what I'd known and when. I knew a burstable instance can't sustain that load. Every catch came from knowledge I already had.

So the errors I would have missed are in the areas where I know least, and those are the areas where I'm most likely to be asking in the first place. A more skeptical attitude doesn't change that, which is why I don't think "be skeptical" is useful advice.

It also means the usual advice, to use AI for things outside your expertise, is close to backwards. AI earns its keep inside your competence, where it drafts faster than you can write and you see the errors immediately. Outside your competence it is a confident stranger, and you have no way to tell a good one from a bad one.

I'd be making the same mistake in reverse if I left this as "the machine is unreliable, I am not."

The same session showed that I had derived the expiry interval from something unrelated, a platform limit that had nothing to do with what I'd applied it to. I would have defended that number if asked. It was wrong, and the system worked anyway, for a reason I hadn't planned.

I also had one of the platform's own timeouts wrong by five minutes, a number I'd been carrying around for years.

Verification is what worked, in whichever direction it ran. My six catches came from domain knowledge, and the AI's two came from reading the documentation. Neither of us was reliable. The result held up because the checking went both ways.

Instead of general skepticism, I rely on five specific habits, and all of them are cheap.

1. I ask for the assumptions before the answer. The 7% error lived entirely in assumptions that were never stated. If I had asked what the model assumed about session length, I would have caught it in one line instead of three drafts.
2. I treat any clean number as suspect. Real systems produce awkward answers. "Roughly 1,135" felt authoritative, and it was the output of a comparison no one had validated. Round, quotable, elegant findings are where I look first now.
3. I don't let it narrate me. If a draft says what I knew, thought or decided, and I didn't say it in those words, it comes out. It may not be wrong, but I would be publishing a claim about myself that I can't source.
4. I make it argue against the result. The most useful question I ask is what would have to be true for this to be wrong? It answers that well, better than it handles a request to reconsider, because it is a different question.
5. I never install a package on a recommendation alone. One in five recommended packages doesn't exist, and someone may have registered the name. I check the registry, the download history, and whether the repository's history goes back further than last week.

The old version warned against believing what you read. Its reason was scarcity: text was hard to produce, so anything written down carried weight it hadn't earned.

Now text is free and unlimited, and it arrives in the voice of someone who knows. What's scarce is the willingness to check a sentence that sounds right.

I use these tools for hours a day, and I'd give up a lot before I gave them up. They are the most useful thing to arrive in my work in ten years. And in one afternoon, on a topic I know well, they were confidently wrong six times.

Both are true, and the second doesn't argue against the first. It is the price, and the price is paid in attention.

Nearly sixteen hundred court cases now record someone paying it in public, several of them involving lawyers who used tools built specifically for their profession. A fifth of the packages AI recommends don't exist, and attackers have learned which of those names are worth registering. None of the people caught out were careless in the ordinary sense. They read something that sounded right and didn't check it, which every one of us does a hundred times a day and gets away with.

The biggest fool was never the one who couldn't read. It is the one who stopped checking because reading got easy.