LLM Hallucination: Confident Answers With Nothing Behind Them

What problem does this solve?

A government entity deploys an Arabic-language assistant over its published service catalogue. A citizen asks about the documents required for a licence renewal. The assistant lists five documents fluently and correctly formatted. Four are right. The fifth was never required.

Nobody notices for weeks, because the answer looks exactly like the four correct ones. There is no hedging, no uncertainty marker, no citation. The failure is invisible precisely because the output is well-formed.

The same mechanism causes invented regulation numbers, plausible but non-existent case references, and confident answers about policies that changed last quarter. In a citizen-facing service, each of these is a governance incident rather than a quality issue.

How the solution works

Ground the answer. Retrieve passages from an authoritative corpus and instruct the model to answer only from them. This converts an open-ended generation problem into a reading-comprehension problem, which models do far better.

Enforce citation. Require every factual claim to carry a reference to a retrieved passage, and reject or flag answers that do not. An uncited claim in a grounded system is a hallucination by definition.

Permit abstention. A model that cannot say 'this is not covered in the documents I have' will invent something instead. Abstention has to be an explicitly rewarded behaviour, not an accident.

Escalate. For high-consequence categories, route to a human rather than answering. The decision about which categories those are belongs to the service owner, not the engineering team.

1. 1
   The objective rewards plausibility Next-token prediction optimises for likely continuations. A fabricated regulation number is highly likely in context; nothing in training penalises it for being false.
2. 2
   Parametric memory is lossy Facts absorbed during pretraining are stored as distributed weights, not as records. Recall degrades and blends related facts together.
3. 3
   Retrieval supplies evidence RAG places authoritative passages in context at answer time, shifting the task from recall to comprehension.
4. 4
   Retrieval can still fail If the retrieved passages are irrelevant, incomplete or contradictory, a model instructed to answer will often answer anyway.
5. 5
   Verification closes the loop A separate check — entailment scoring or citation validation — tests whether the answer is actually supported by the passages retrieved.

Reference architecture

Hallucination is reduced at four independent points. A system with only one of them will still produce confident fabrications.

Deployment options: For government and regulated data, the corpus and often the model itself must stay inside the entity's boundary, which constrains model choice to what can be self-hosted and shifts the evaluation burden onto the deploying team.

Key capabilities

Integrations

The controls span the document platform, the retrieval service and the case-management system that receives escalations.

Industry use cases

Implementation approach

1. 1
   Establish the corpus Confirm authority, currency, ownership and coverage before building anything. Most grounded-answer failures start here.
2. 2
   Measure retrieval separately Evaluate recall@k on real questions before judging the generator. A generation problem is often a retrieval problem.
3. 3
   Constrain and require citation Instruct answering from passages only, and make every factual sentence carry a reference.
4. 4
   Define abstention Agree with the service owner which categories the assistant must refuse, and what the citizen sees instead.
5. 5
   Verify before display Validate citations, and route failures to a human rather than showing an unverified answer.

Security & deployment

Retrieval must respect the asker's permissions: a grounded assistant that retrieves from documents the user is not entitled to see becomes an access-control bypass with a friendly interface. Enforce document-level authorisation at retrieval time rather than filtering afterwards. Log the passages used for each answer so a disputed response can be reconstructed. Treat prompt injection embedded in retrieved documents as a live threat, since a corpus that accepts user-submitted content can carry instructions to the model.

A worked example

The same question through three configurations, showing where the control actually bites.

1. No retrieval. The model answers from parametric memory. It produces five requirements, one invented, with no way for the reader to check any of them.
2. Retrieval, no citation requirement. Three relevant passages are retrieved. The model uses them but adds a sixth requirement from memory, blended seamlessly into the list.
3. Retrieval plus enforced citation. Every item must carry a passage reference. The invented requirement has no source to cite, so it is dropped or flagged before display.
4. Retrieval, citation and abstention. The corpus does not cover renewals for one licence category. The assistant says so and offers escalation rather than filling the gap.

Retrieval alone removed some of the problem. Citation enforcement removed the specific failure. Abstention handled the case where the honest answer was that the corpus does not cover it — which retrieval by itself will never produce.

In code

Grounding is a prompt-and-verification contract, not a single setting. This shows the structure: retrieve, constrain, then verify that every claim is supported.

import re

SYSTEM = (
    "Answer ONLY from the numbered passages provided. "
    "Every factual sentence must end with a citation like [2]. "
    "If the passages do not contain the answer, reply exactly: "
    "INSUFFICIENT_CONTEXT. Do not use outside knowledge."
)

def build_prompt(question, passages):
    body = "\n".join(f"[{i}] {p}" for i, p in enumerate(passages, start=1))
    return f"{SYSTEM}\n\nPASSAGES:\n{body}\n\nQUESTION: {question}"

def uncited_sentences(answer, n_passages):
    """Return factual sentences carrying no valid citation."""
    problems = []
    for sentence in re.split(r"(?<=[.!?\u061F])\s+", answer.strip()):
        if not sentence:
            continue
        cites = [int(c) for c in re.findall(r"\[(\d+)\]", sentence)]
        if not cites:
            problems.append(("uncited", sentence))
        elif any(c < 1 or c > n_passages for c in cites):
            problems.append(("bad_citation", sentence))
    return problems

# Gate the response before it reaches the user.
answer = call_model(build_prompt(question, passages))
if answer.strip() == "INSUFFICIENT_CONTEXT":
    route_to_human(question)
else:
    issues = uncited_sentences(answer, len(passages))
    if issues:
        route_to_human(question, reason=issues)  # never display unverified claims
    else:
        display(answer)

Either a fully cited answer, or a routing event. The sentence splitter includes the Arabic question mark (U+061F) so Arabic responses segment correctly — a Latin-only splitter silently treats an entire Arabic answer as one sentence and the check becomes meaningless.

Diagnostic checks

• Ask questions the corpus provably does not cover. A system without abstention will invent an answer, and you will see it immediately.
• Check every citation resolves to a passage that actually contains the claim, not merely a related one.
• Measure retrieval recall@k independently of answer quality to separate the two failure modes.
• Run the same question in Arabic and English against a bilingual corpus and compare — a large gap indicates a retrieval problem, not a generation one.
• Ask about a policy that changed recently. A stale corpus produces a confidently wrong grounded answer, which is harder to spot than an ungrounded one.
• Sample production answers weekly and score support against cited passages; faithfulness degrades quietly as the corpus drifts.

Limitations & prerequisites

• Grounding reduces hallucination substantially but does not eliminate it; models still over-extend and misread supplied passages.
• Citation enforcement verifies that a source was cited, not that the source supports the claim, unless entailment checking is added.
• Abstention trades coverage for reliability, and an assistant that abstains too often gets abandoned by users.
• A current, well-owned corpus is a prerequisite that most organisations underestimate and that no model choice compensates for.

What each control actually removes

These are cumulative, not alternatives. Each closes a failure the previous one leaves open.

Deploying only the first row and calling the result grounded is the most common way these systems reach production unsafely.

Key takeaways

• Hallucination follows from the training objective; it is not a bug to be prompted away.
• Retrieval converts recall into comprehension and removes a large share of the problem.
• Citation enforcement is what makes a grounded claim checkable rather than merely sourced-looking.
• Abstention must be an explicitly permitted answer or the model will fill the gap.
• In Arabic deployments the hard part is retrieval quality, and it must be measured on real Arabic queries.

FAQ

Related topics & solutions

• AI Products Portfolio