AI Hallucinations & False Confidence
The AI did not hesitate. It did not flag uncertainty. It gave you a complete, authoritative answer - and some of it was invented.
The Dosage That Did Not Exist
Marcus had been prescribed a new medication. He wanted to understand whether it interacted with a supplement he had been taking for years.
He asked an AI chatbot. The response was clear, detailed, and specific: the combination was safe at standard doses, interactions were unlikely, and a named study from a named university confirmed this. He felt reassured. He continued both.
Two weeks later, his pharmacist noticed the combination on his records and asked who had cleared it. When Marcus described the AI's answer, the pharmacist looked up the named study. It did not exist. The university's research database had no record of it. The interaction was not well-studied - which meant the AI had not said "I don't know." It had generated a plausible, specific, confident answer about something it could not know.
The answer looked exactly like accurate medical information. It was not.
What Is Actually Happening
AI language models generate the next most probable word, given what came before. They do not access a database of verified facts. They produce text that sounds like what accurate text sounds like - whether or not the content is accurate.
27%
of AI responses contain incorrect information on tasks requiring factual precision.
The confidence of the answer is not correlated with its accuracy.
Source: Stanford HAI AI Index Report, 2024Wrong 73% of medical queries
A study testing AI chatbots on medical questions found incorrect or incomplete information in 73% of responses - including specific drug interactions and dosage guidance. The answers were confidently stated in all cases.
Lawyers sanctioned for fake cases
Multiple lawyers in the US were sanctioned by courts in 2023-2024 after submitting AI-generated legal briefs that cited cases which did not exist. The citations were formatted correctly and appeared real. The cases were entirely fabricated.
AI agrees when challenged
AI systems are trained to produce agreeable responses. When users challenge a correct AI answer with a wrong assertion, AI models frequently revise their answer to agree with the user - even when the original answer was accurate.
Real journals, invented studies
AI models frequently generate citations using real journal names, real author name formats, and plausible titles - for studies that do not exist. The format is accurate. The content is fabricated. The citation passes casual inspection.
Using AI for Research: The Specific Risks
AI is genuinely useful for summarising, explaining, and brainstorming. It is dangerous when used as a primary source for facts you will act on.
For medical information: AI does not know your medical history, your contraindications, or your other medications. It produces general-population guidance for your specific situation. These are not the same thing.
For legal information: Law varies by jurisdiction, by year, by specific circumstances. AI often produces a blend of rules from multiple jurisdictions presented as though they apply to your case.
For financial information: AI can explain concepts accurately and give projections that are entirely wrong for your individual situation, tax position, and risk profile.
For historical and scientific facts: AI can confidently state a date, a statistic, or a study finding that was never recorded anywhere in that form.
In each case, the answer is delivered with the same confident tone regardless of whether it is accurate.
When AI Confidently Lies
The word "hallucination" makes AI errors sound like a malfunction. They are not. They are the expected output of a system designed to generate plausible text - and plausible text sometimes contains false claims.
The model does not experience uncertainty the way a human does. It does not hesitate, hedge, or decline when it does not know. It generates the most probable continuation of your question - and the most probable continuation of a medical question is a specific, authoritative medical answer.
Confidence is a property of the output format, not the underlying information.
Try It: Ask the Confident Liar
Five AI-generated answers to real questions. Mark the ones you would trust and act on. Then see which were accurate, partially wrong, or entirely fabricated.
What That Just Showed You
1. You cannot tell the difference from tone. The accurate answers and the fabricated answers were written in the same style, with the same confidence, at the same length. Tone is not a signal of accuracy.
2. Citations make fabricated answers more convincing, not more reliable. An AI-generated citation formatted like a real academic reference is harder to dismiss than a claim with no citation. Checking whether the cited work actually exists is now a necessary step.
3. Sycophancy compounds the problem. If you push back on an AI answer with a wrong correction, many AI systems will revise their answer toward your wrong version. The model's agreeableness is a feature that makes it less reliable as a research tool.
4. The two questions matter for every AI answer you act on. Can you verify this independently from a primary source? And what happens to you if this answer is wrong?
Three Things Worth Doing
1. For medical, legal, and financial questions, verify with a licensed professional. AI can help you understand a question better, but the answer you act on should come from a primary source - a doctor, a solicitor, a certified financial planner.
2. Check every AI-provided citation before using it. Search for the named study or case in the journal's own database. This takes 90 seconds and catches fabricated citations before they cause a problem.
3. Ask the AI where it is uncertain. "What are the limits of your confidence in this answer?" and "What would make this answer wrong?" are questions that surface uncertainty the model would not otherwise volunteer. They do not guarantee accuracy - but they often reveal what you need to verify.
One Question Before You Continue
Marcus received confident, specific medical information from an AI that included a citation to a study - and the study did not exist. What does this reveal about how AI language models work?