Why AI Hallucinated Citations Keep Showing Up in Court

AI hallucinated citations are references to court cases, laws or articles that a chatbot invents and presents as real. They happen because a language model predicts likely-sounding text rather than looking facts up. A made-up case name with a plausible court and page number looks exactly like a real one, so it slips into filings.

This is no longer a rare embarrassment. A public tracker of court decisions now lists more than 2,200 cases where a party relied on hallucinated material, and the count keeps climbing. Most come from people representing themselves, but lawyers, experts and even a few judges appear in the list.

Oddworldz covers the strange things AI does and why it does them. This piece is for anyone who uses a chatbot for research, legal or otherwise, and wants to know why it invents sources so confidently. It covers what the research says about the cause, what the numbers show and how to catch a fake before it costs you.

What Is an AI Hallucinated Citation?

A hallucinated citation is a reference that looks real but points to nothing. It might be an invented case, a real case with the wrong holding, or a quote that never appears in the source. The format is usually perfect. That is the trap.

The National Institute of Standards and Technology (NIST) has a more careful name for it. Its Generative AI Profile, published July 26, 2024, lists "confabulation" among the risks of generative AI. NIST defines it as confidently stated but false content that may mislead users.

The NIST profile also singles out citations. It notes that generated answers may include confabulated logic or citations that seem to justify the answer, which can lead people to trust it more than they should. A fake source does double damage. It is wrong, and it makes the wrong answer look checked.

We prefer NIST's word to "lying." Lying needs intent. A language model has no picture of the truth to hide. It is doing the one thing it was built to do, which is produce text that fits.

Why Does a Chatbot Make Up a Case That Doesn't Exist?

A chatbot makes up a case because it generates the most likely next words, not the most accurate ones. Legal citations follow a rigid pattern of names, volume numbers, court abbreviations and years. A model that has seen thousands of them can produce a new one that fits the pattern perfectly, with no real case behind it.

NIST describes this as a natural result of how generative models are designed. They approximate the statistical patterns in their training data. That can produce accurate answers. It can also produce text that is fluent, specific and false, especially for open-ended questions in specialist fields.

Law is close to a worst case. The questions are specific, the format is easy to imitate and the cost of an error is high. A request like "find me three cases supporting this argument" invites the model to fill the gap with something shaped like an answer.

A related quirk makes it worse. Push back on a correct answer and many models cave, as we covered in our piece on chatbot sycophancy. Ask a model to confirm its own fake case and it may simply agree that the case exists.

The Numbers Behind Hallucinated Citations in Court

The best public count comes from researcher Damien Charlotin. His AI Hallucination Cases database listed 2,202 cases as of October 19, 2026. It counts court and tribunal decisions where a judge found, or clearly implied, that a party relied on hallucinated material. Mere allegations generally do not qualify.

The breakdown is revealing. Here is how the database's filters split the cases:

Two cautions apply. These filter counts overlap, so they do not add up to the total. And the database only sees decisions that mention AI. Fake citations caught quietly by a clerk, or never caught at all, are not in it.

Even so, the shape is clear. People without lawyers lean on chatbots most, and they have the least ability to spot a fake. That is a fairness problem, not just a tech problem.

Lawyers face sharper consequences when it happens. Courts have fined attorneys, referred them to bar regulators and ordered them to tell their clients. A lawyer signs every filing and vouches for every case in it. Blaming the chatbot has not worked as a defense.

What Are Courts Doing About Hallucinated Citations?

Courts are mostly responding with verification rules rather than bans. Some judges have issued standing orders that require filers to disclose AI use or confirm that a human checked every citation. The federal judiciary has also issued guidance for its own staff, after AI errors turned up in court orders, not just in briefs.

The Administrative Office of the U.S. Courts sent interim AI guidance to federal courts on July 31, 2025. According to FedScoop's report on the guidance, it allows experimentation but warns against handing decision-making to AI. It urges extreme caution on novel legal questions and says users stay accountable for AI-assisted work.

The trigger was awkward. Senator Chuck Grassley had asked about error-filled orders from two federal judges. The judges said clerks had used generative AI tools and that new checks were in place. If a judge's chambers can be caught out, a tired self-represented litigant has little chance without a habit of checking.

Individual courts still set many of their own rules. That means the same AI-drafted brief could be fine in one courtroom and need a signed disclosure in another. Anyone filing should read the local rules first.

Why Training Rewards a Confident Guess

A 2025 research paper offers a blunt explanation for why the habit persists. In Why Language Models Hallucinate, researchers Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang argue that training and testing reward guessing over admitting uncertainty.

Think of a multiple-choice exam with no penalty for wrong answers. A student who guesses on every question scores higher than one who leaves blanks. Most AI benchmarks grade answers as simply right or wrong. So a model that says "I don't know" loses points, and a model that guesses confidently wins.

The authors' fix is not a new hallucination test. They suggest changing how the main leaderboards score answers, so that honest uncertainty stops being punished. That is a policy fix as much as a technical one, and it is not settled yet.

The incentive also shows up in marketing. In February 2025 the Federal Trade Commission (FTC) finalized an order against DoNotPay, which had billed itself as "the world's first robot lawyer." The FTC said the company never tested whether its AI matched a human lawyer's work. The order required $193,000 in monetary relief.

How to Catch a Fake Citation Before It Bites

The fix is boring and reliable: check every citation in a real source. Here is the routine we would use for any AI-assisted research:

Here is a small experiment worth trying. Ask a chatbot for three court cases on a narrow, obscure point, then search for each one yourself. The result varies by model and by whether it can browse, which is part of the lesson. Tools that search the web and show sources tend to do better, though how answer engines pick sources has quirks of its own.

One counterpoint is fair. AI tools can be genuinely useful for drafting and summarizing, and some legal research tools are built on verified case databases. The risk is highest with general chatbots used as if they were search engines.

The Bottom Line on AI Hallucinated Citations

A chatbot invents citations because it is a pattern machine, and legal citations are an easy pattern to copy. NIST calls it confabulation, researchers trace it to training that rewards guessing, and the court record now runs past 2,200 decisions.

None of that means AI is useless for research. It means the output is a draft, not a source. At Oddworldz, our rule is simple: if a citation came from a chatbot, it has not been checked yet. The same caution applies in reverse, since AI detectors also flag human writing by mistake. Check the source, every time.