You reduce AI hallucinations with prompts by doing four things: giving the model the source material instead of relying on its memory, explicitly allowing it to answer “I don’t know”, forcing it to quote the text behind each claim, and narrowing the question so there is less room to guess. These habits lower the rate of confident wrong answers, but no prompt removes hallucinations entirely, so anything high-stakes still needs a check against a primary source.
This guide explains why models make things up, then walks through a single realistic task, rewriting the prompt step by step and showing what each change is meant to prevent. Copy-ready templates are at the end.
What a hallucination is, and why prompts can only partly fix it
A hallucination is output that is fluent and confident but not supported by facts or by the material the model was given: an invented citation, a wrong date, a function that does not exist in the library, a policy clause that is not in the document. The OWASP Top 10 for LLM Applications lists this under LLM09: Misinformation and names hallucination as its main cause, with overreliance by users making the damage worse.
The root cause sits in training, not in your prompt. OpenAI’s 2025 explanation, Why language models hallucinate, makes two points that matter for prompt writers:
- Rare facts cannot be learned from patterns. Pretraining teaches the model to predict the next word. Spelling and grammar follow consistent patterns; a specific person’s birthday or an obscure statistic does not, so the model fills the gap with something plausible.
- Scoring rewards guessing. Benchmarks that grade only accuracy give credit for a lucky guess and zero for “I don’t know”, so models are pushed toward answering anyway. OpenAI’s example: guessing a birthday is right about 1 time in 365, which still beats abstaining on an accuracy-only scoreboard.
A prompt cannot retrain the model. What it can do is change the situation the model is in: supply the facts so it does not need to recall them, make abstaining an acceptable answer, and make unsupported claims easy to spot. That is the whole strategy in one sentence.
Where hallucinations show up most
Some question types are far riskier than others, and knowing which ones lets you spend effort where it matters. Watch for these:
- Specific numbers: prices, limits, version numbers, dates, percentages, quotas.
- References: paper titles, case law, URLs, book page numbers, author names.
- Niche or recent facts: anything after the model’s training cutoff, or anything rarely written about.
- APIs and libraries: parameter names, CLI flags and package names that sound right but do not exist. OWASP calls out insecure or non-existent library suggestions as a specific risk.
- Long documents: summaries that blend what the document says with what documents like it usually say.
Questions about general concepts (“what is a load balancer”) are lower risk. Questions that need one exact, rarely-repeated fact are high risk, and that is where the techniques below earn their cost.
Worked example: summarizing a vendor contract
The following scenario is illustrative. Imagine you are an infrastructure lead who needs to know the uptime commitment, the service credits and the termination notice period in a 40-page SaaS agreement before a renewal call. Each version of the prompt below fixes one failure mode of the previous one.
Version 1: the prompt most people write
What are the SLA terms in the Acme Cloud contract? Uptime, credits, and how much notice we need to give to cancel.The problem: no document is attached. The model will answer from what SaaS contracts typically say (“99.9% uptime, 10% service credit, 30 days’ notice”) and present it as if it were your contract. Nothing in the answer tells you it was invented.
Version 2: provide the source and restrict the model to it
<contract>
[paste the full contract text here]
</contract>
Using only the contract above, not your general knowledge of SaaS agreements,
answer these three questions:
1. What monthly uptime percentage does the vendor commit to?
2. What service credits apply if that commitment is missed?
3. How many days' notice must the customer give to terminate or not renew?Supplying the document is the single biggest improvement, because the model no longer needs to recall anything. The “only the contract above” line is what Anthropic’s documentation calls external knowledge restriction. The XML-style tags separate the document from the instructions, so the model does not confuse contract text with your request.
Remaining failure mode: if the contract does not mention termination notice at all, the model may still produce a number because you asked for one.
Version 3: make “not found” a valid answer
If the contract does not state an answer to a question, write
"Not stated in the contract" for that question. Do not infer a value
from industry norms. A "Not stated" answer is more useful to me than
a plausible guess.This addresses the guessing incentive directly. Anthropic lists “allow Claude to say I don’t know” as its first basic technique, and the last sentence in the block tells the model why abstaining is preferred, which tends to work better than a bare rule. If you have seen negative instructions get ignored, the reasoning in why vague negative instructions get ignored applies here too: say what to do instead, not only what to avoid.
Version 4: quotes first, answer second
Step 1. Inside <quotes> tags, copy word for word every sentence from the
contract that is relevant to uptime, service credits, termination or renewal.
Include the section number for each quote. If none are relevant, write
"No relevant quotes found."
Step 2. Inside <answers> tags, answer the three questions using only the
quotes from Step 1. After each answer, put the section number it relies on
in square brackets.Extracting quotes before answering anchors the response in the actual text. Anthropic recommends this specifically for long documents (it mentions inputs over 20k tokens). It also makes the output auditable: you can search the contract for each quote in seconds. If a quote is not in the document, you have caught a hallucination before it reached your renewal call. For a deeper version of this pattern, see making Claude cite the exact sentence behind every claim.
Version 5: a verification pass
Now review your answers. For each answer, check that the quoted text
actually supports it, including numbers, units (days vs. business days),
and conditions (e.g. "excluding scheduled maintenance"). If an answer
goes beyond what the quote says, correct it or replace it with
"Not stated in the contract". List any corrections you made.A second pass catches a subtler error: a real quote paired with a wrong conclusion, such as reading “30 business days” as “30 days” or dropping a maintenance exclusion. Anthropic describes a similar step where the model must retract any claim it cannot support with a quote. A more structured version is Chain-of-Verification, from a 2023 Meta AI paper: draft an answer, plan verification questions, answer them independently so the draft does not bias them, then write the final answer.
What each technique protects against
Each technique targets a different failure, so it helps to choose based on what is going wrong rather than stacking everything every time. The table summarizes the trade-offs.
| Technique | Failure it targets | Cost | Use when |
|---|---|---|---|
| Provide source text | Answering from memory | Longer prompt, more input tokens | Any factual question about a specific document, product or policy |
| Restrict to provided sources | Blending document with general knowledge | One sentence | Always, when you provide sources |
| Allow “I don’t know” | Guessing to fill a gap | One sentence; more “not stated” answers | Questions that might have no answer in the material |
| Quotes before answer | Plausible but unsupported claims | Longer output | Long documents, legal, compliance, financial text |
| Per-claim citations | Claims you cannot audit | Longer output, more review effort | Anything someone else will rely on |
| Verification pass | Real quote, wrong conclusion | Extra turn or call | Numbers, conditions, exceptions |
| Best-of-N comparison | Unstable, low-confidence facts | N times the cost | Automated pipelines where you can compare runs |
| Narrow the question | Broad prompts inviting filler | More prompts to write | Research and summary tasks |
Best-of-N, also from Anthropic’s list, means running the same prompt several times and comparing results. If three runs give three different notice periods, that inconsistency is a strong signal that at least two of them are not grounded in the text.
Prompting habits that make hallucinations worse
Several common prompt patterns increase the odds of invented content, usually by pressuring the model to produce something specific. Avoid these:
- Asking for citations without giving sources or tools. “Include five academic references” with no retrieval or web search often produces references that look real and do not exist. Either supply the sources or use a product feature that actually searches and cites.
- Demanding an exact count. “List 10 studies showing X” pushes the model to reach 10 even if it only knows 3. Ask for “up to 10, only ones you are confident exist”.
- Leading questions. “Why does the contract allow termination with 15 days’ notice?” presupposes the fact. Ask “What does the contract say about termination notice?” instead.
- Stuffing the prompt with irrelevant material. Huge context with the answer buried in the middle makes it easier for the model to miss the relevant passage and fill in from general knowledge. Put the most relevant material where it is easy to find and ask for quotes.
- Treating confidence as accuracy. The model’s tone does not change when it is wrong. Fluent prose is not evidence.
Does chain-of-thought reduce hallucinations?
Asking a model to reason step by step mainly helps on multi-step problems, and it helps with hallucinations indirectly by making flawed reasoning visible. Anthropic’s guide suggests using extended thinking and reviewing the reasoning when an answer looks wrong, because the reasoning can reveal a bad assumption. It does not supply missing facts: a model that does not know a statistic will not discover it by thinking longer. Use chain-of-thought prompting for calculations, logic and multi-condition questions, and grounding for facts.
When prompting is not enough
Prompting reduces hallucinations at the level of a single conversation; production systems need controls outside the prompt as well. OWASP’s mitigations for LLM09 include retrieval-augmented generation (RAG, which fetches relevant documents from a trusted store and inserts them into the prompt), automatic validation of key outputs, human review for critical content, and interface design that tells users about reliability limits.
In practice, that means a few concrete checks. If the model outputs a package name, check it exists in the registry before installing it. If it outputs JSON, validate it against a schema. If it quotes a document, verify the quote with a string search. If it produces a citation, open the link. Platform features can help: Anthropic offers a Citations API that ties answers to passages in documents you provide, compared with prompt-only quoting in Claude’s Citations API vs. manual quote-prompting.
Copy-ready templates
Grounded question-answering over a document
<document>
{{DOCUMENT_TEXT}}
</document>
Answer the question below using only the document above.
1. First, inside <quotes></quotes>, copy word for word the sentences
from the document that are relevant to the question. If there are none,
write "No relevant quotes found."
2. Then, inside <answer></answer>, answer using only those quotes.
Mark each claim with the number of the quote that supports it, like [2].
3. If the document does not contain enough information, say
"The document does not answer this" and stop. A clear "not answered"
is more useful to me than a reasonable-sounding guess.
Question: {{QUESTION}}Why each part exists: the tags separate data from instructions; the quotes step anchors the answer; the numbered references make the answer auditable; the final rule removes the incentive to guess and explains why.
Questions from model knowledge, with confidence labels
Answer the question below from your general knowledge.
- Separate your answer into "Confident" and "Unsure" sections.
- Put any specific number, date, name, version or URL in "Unsure"
unless you are confident it is correct.
- Do not invent references. If you mention a source, describe it
in general terms rather than giving a title or link you are not sure of.
- End with a list of the claims I should verify before relying on this.
Question: {{QUESTION}}When you cannot supply sources, the next best thing is a map of where the answer is weak. Self-reported confidence is imperfect, so treat the “verify” list as a starting point, not a guarantee.
Review pass on an existing draft
<sources>
{{SOURCE_TEXT}}
</sources>
<draft>
{{DRAFT_TEXT}}
</draft>
Check every factual claim in the draft against the sources.
Return a table with columns: Claim | Supporting quote | Status.
Status is "Supported", "Partly supported" (explain what differs),
or "Not found in sources". Do not rewrite the draft.Keeping the review separate from the rewrite stops the model from quietly “fixing” claims by inventing support for them. You decide what to change.
A short checklist before you trust an answer
- Did the model have the source material, or was it answering from memory?
- Was it told that “not stated” or “I don’t know” is acceptable?
- Can every specific number, name and reference be traced to a quote or link?
- Did you open at least the links and check the quotes that the decision depends on?
- For code: do the packages, functions and flags exist in the version you use?
If any answer is “no” and the stakes are real, do the check before acting on the output.
