The model isn’t lying to you. It’s guessing with confidence. This piece explains why that happens and gives you the few-minute catch routine I run before anything written with AI goes out.
You ask an AI for a source, a quote or a niche fact, and it gives you something that looks perfect. The formatting is clean, the author sounds real and there’s a link. Then you click the link and it goes nowhere. The quote was never said. The feature it described was never built.
It took me a while to stop being annoyed and see what was actually happening:
AI isn’t lying. It’s guessing with confidence. Your job is to build catches, not to trust the vibe.
Seen that way, a hallucination is no longer a mystery. It’s a known failure, like a typo on an invoice. You don’t panic about it. You put a check in the process.
What counts as a hallucination
For work, I use a plain definition: the model states something specific as fact, and that claim is false, can’t be verified, or isn’t supported by what you gave it.
The usual suspects:
A book quote nobody wrote
A real court case with the wrong year
A software function that never shipped
“According to your handbook...” when the handbook says the opposite
A link that returns a 404
The dangerous part is that none of these look strange. They read just as smoothly as the true sentences around them, and that’s how they get past you.
Made-up material in creative writing doesn’t count, because there you asked for it. Made-up footnotes in a due-diligence memo do.
Why models make things up
A language model doesn’t look facts up. It predicts which words are most likely to come next. That works very well for patterns like grammar, tone and the usual shape of an email. It works badly for rare one-off facts, such as a specific date or the exact wording of a clause, because patterns can’t predict those.
There’s also an incentive problem. OpenAI’s own research on hallucinations argues that standard training and testing reward a guess over an honest “I don’t know”. Think of a multiple-choice exam where a blank always scores zero and a guess sometimes scores a point. Over thousands of questions, the student who always guesses looks better on the scoreboard. Models are scored much the same way, so they learn to guess.
The causes you’ll actually run into:
Missing knowledge. The fact isn’t in the training data, or it isn’t in the material you provided.
Pressure to answer. Your instructions, or just your tone, reward a full answer over an admission of a gap.
Blended memories. Similar names, papers or product features get merged into one confident fake.
Lost in the paste. The right fact was in the middle of a huge document, and the model skipped over it.
Retrieval failure. The tool was meant to fetch a source, missed it, and the model filled the gap without saying so.
Newer and “reasoning” models get some of this wrong less often. They don’t get it to zero. Anyone promising zero hallucinations is selling a feeling.
Where it hurts most
I’m a systems engineer, not a lawyer or a doctor. Wherever an error can hurt someone, AI is a drafting assistant working behind a qualified person. I apply the same rule in my own work: anything that spends money or goes out in my name waits for a human yes.
A catch routine you can run in minutes
Use this whenever the answer matters:
Spot the specifics. Names, numbers, dates, quotes, links, legal or medical claims.
Ask for evidence. “Quote the exact sentence from the source.” “Link the primary page.” “Show me the line in the file I uploaded.”
Open the source yourself. If you can’t open it, you don’t have a citation.
Read one level deeper. Does the source really say that, or did the model stretch it?
Cross-check when the stakes are high. Try a second model or a normal search.
Ground the next run. Put the source into the chat and ask for answers from that material only.
Here’s what it looks like in practice. Say you ask an assistant to summarise a supplier contract, and it tells you the payment terms are 60 days. That’s a number, so it’s a specific. You ask it to quote the clause. You open the contract and find the clause. The contract might say 60 days from delivery acceptance, not from invoice date, and that changes your cash flow. It took five minutes, and what you caught wasn’t even an invention. It was a stretch, which is the more common kind of error.
The same goes for other content. Code: run it. Maths: recalculate it. Policy: open the policy. “It sounded sure” is never an exception.
Grounding beats arguing with the model
When a model invents something, arguing with it in the same chat rarely helps. Asking “are you sure?” sometimes gets an apology and sometimes gets the error doubled down. Give it evidence instead:
Paste in the relevant section of the official docs, or connect it to documents you trust.
Ask: “Answer only from the text above. If it isn’t there, say it isn’t there.”
If it still invents, your instruction is being drowned out. Shorten the context, make the rule stricter, or start a fresh chat.
Evidence in the chat works better than general scepticism.
Disagreement between models is a signal
If two strong models give confidently different answers to a factual question, you don’t have an answer yet. You have a research task. Go find the primary source. The disagreement is useful. Averaging two guesses gives you nothing.
Prompt patterns that reduce (not erase) risk
“If you’re not sure, say so. Don’t invent citations.”
“Answer only from the documents provided. If something is missing, tell me what.”
“List your assumptions separately from the facts.”
“Three bullets max, and each one needs a source I can open.”
For agents that browse: “Before you state a number from the web, fetch the page and quote it.”
Just as important, remove the pressure to bluff. “I need this now, fill in whatever you don’t know” is how you teach a session to lie politely.
Red flags in the prose itself
Train your eye to catch these:
When two of these show up in one draft, stop and verify before anything ships. Five minutes of checking beats five days of cleanup.
Build a 10-question trap set
This is the most useful hour you’ll spend on AI this month:
Write down 10 questions from your real work where a wrong answer would hurt.
Make sure 2 of them have answers that are not in your documents. A good assistant should say it doesn’t know.
Run them through the assistant you normally use.
Score each answer: correct, wrong, or honestly uncertain.
Add your source documents or tighten the instructions, then run them again.
Keep the set, and re-run it whenever you change model or prompt.
That small set is an evaluation. It tells you more than “this one feels better”.
After you catch one
Don’t just fix the paragraph. Add the miss to your trap set, update the process, and ask whether grounding in sources should have been mandatory for that task. Every hallucination you catch should improve the system, not only the document.
Make it a team habit
Technical checks fail when the team is rewarded for speed over accuracy. Set a few norms:
“Unknown” is an acceptable answer, from the model and from people.
Shipping a fake citation is worse than shipping late.
Every AI draft that faces a customer gets a human check on the specifics.
When a hallucination gets through, fix the process, not just the person.
I treat a hallucinated specific like a production bug. It gets a ticket, and it gets a new question in the trap set.
The checklist before you hit publish
Keep this next to your screen if you ship AI drafts often:
Every number traced to a source I opened
Every quote traced to a source I opened
Every link opens the right page
Names and spellings match the primary source
Every “always” and “never” is either backed up or softened
Questions outside the scope were declined or escalated
Policy wording compared line by line with the original
I use the same list whether the draft came from a junior colleague or a model. They fail in similar ways.
FAQ
Why does it invent sources? Text shaped like a source is everywhere in training data, and people reward answers that look complete. Ask for links you can open, then open them.
Is a “creativity” or temperature setting the cause? It can play a part, but it’s not the main lever. For business questions, grounding and incentives matter much more.
Who’s responsible when it’s wrong? Whoever ships the output. Your process owns what goes out, so build the checks around that.
The takeaway
A hallucination is a confident guess. Catch it by hunting for specifics, demanding sources you can actually open, grounding the model in your own documents, and keeping a human on anything you can’t undo. Trust is earned claim by claim, not by which AI brand wrote the chat.
About Me
I’m Laurent Terrijn, an engineer and business builder. I build the systems businesses run on. On the side, out of curiosity, I explain AI in plain English for busy business people. I’m a nerd who speaks ten languages and loves learning. If this helped, follow along for the next one. More at laurentterrijn.com.
