Abstaining when evidence is missing: fluent is not correct

Why a fluent but wrong answer is worse than not knowing

A wrong answer that reads well does more damage than a plain admission that the system does not know. When the system reports missing information, you still know to check somewhere else. When it writes a confident, grammatical paragraph with citations, you tend to believe it and stop — and the error goes straight into a decision.

This is not about writing style. Abstention is behaviour you have to design, not a sentence you add to a prompt. The rest of this post walks through the public evidence for why telling a model to abstain is not enough, and what a RAG system has to build instead.

Missing evidence and wrong evidence are two different problems

These two situations often get merged into one, but they differ in kind. Missing evidence means the retriever brought back no relevant passage. Wrong evidence means there is a relevant passage, but it supports an incorrect answer.

Different in kind means different in handling. A mechanism that only ever answers the question 'is the context sufficient?' will never catch the case where the context is complete but wrong.

• Missing: you need a sufficiency check that runs before generation.
• Wrong: you need to check whether the context contradicts trusted knowledge — much harder, because a doctored passage looks fully valid.
• Both: the context is thin and contains a bad passage; the system must separate the two signals before it decides to stay silent or answer.

The numbers: small models still answer under misleading context

The GRAB-RAG paper checks the same question set under four context conditions: supportive (supports the correct answer), degraded (missing information), missing (no relevant passage), and misleading (falsified).

Under the misleading condition, one gold passage is edited to support a wrong answer and then mixed in with the retrieved passages. On the surface, it looks no different from the others.

• 5 different abstention policies are tried on the same set.
• 3 frozen small models, ranging from 3.8B to 8B.
• 2 question-answering sets: Natural Questions and HotpotQA.

The result is not where you would guess. The models abstain fairly reliably when evidence is missing. But when evidence is wrong, even after being told to abstain, they still answer 41.6% of the questions. Of those, 63% repeat verbatim the false entity that was injected into the context.

Chain-of-thought adds almost nothing. A conflict checker placed on the generator side pulls the answer rate under misleading context down to 13.3%, but in exchange it throws away many correct answers. An NLI verifier recovers coverage, but fails when the model's parametric memory agrees with the wrong passage.

Why telling the model to stay quiet is not enough

The failure is built into the structure of the problem. Prompt-based abstention asks one question: is this context sufficient to answer? It does not ask the second question: is this context correct?

A falsified passage is still 'sufficient'. It is relevant to the question, it contains the needed entity and relation, it answers directly. A sufficiency classifier has no signal with which to refuse it — so it lets it through, and the model produces a wrong answer in a confident voice.

The paper's conclusion is terse: prompt-based abstention only asks whether the context is sufficient, not whether it is correct, and no checker in the paper closes that gap without trading off coverage against safety.

Citations can be fabricated too

Citations are usually read as a signal of reliability: a source means verification. But a citation is just one more claim the model generates, and it can be fabricated as well.

Stanford's study of legal research tools shows this is not an academic concern. Some vendors advertise that RAG removes or avoids hallucination, and even guarantee that legal citations are not fabricated. The study tested those claims and found the tools still produce fabricated content.

The lesson is not 'don't use RAG'. The lesson is that a citation is not automatically evidence. A link only has value when it points at the right source, and that source actually supports the sentence it sits next to.

Groundedness is not truth

A single faithfulness score easily becomes a target to optimise — and just as easily hides failures. An answer can hug the retrieved text and still be wrong, simply because that text is outdated, incomplete, contradictory, or not authoritative.

The paper Groundedness Is Not Truth (ACL 2026) puts the problem exactly there: a single faithfulness number collapses several failure modes onto one axis, and each failure mode needs a different fix.

Groundedness is not truth. A single faithfulness score can hide several failure modes that need different fixes — an answer may rest on text that is outdated, incomplete, contradictory, or not authoritative.

The direct consequence: if you only report one groundedness number, you know there is an error but not whether it lives in retrieval or in generation. And if you don't know where the error lives, you can't fix it.

Measuring citation quality: three axes and one uncomfortable number

ALCE is the first benchmark for automatically evaluating LLM citations. It measures three axes: fluency, correctness, and citation quality. These metrics correlate strongly with human judgement, which means you can track them without hand-grading every answer.

• Fluency: the answer reads naturally and does not fall apart.
• Correctness: whether the content is right.
• Citation quality: whether each citation actually supports the exact span it sits beside.

The uncomfortable number: on the ELI5 set, even the best models still lack full citation support in 50% of cases. That is, half of the answers have at least one part that the sources do not fully back.

The paper names three directions for improvement, and all three are things to build rather than things to instruct: a better retriever, a longer-context LLM, and the ability to synthesise across multiple sources.

Longer context does not fix this by itself (Lost in the Middle)

If citations are weak because the context is thin, the first reflex is to stuff in more context. Lost in the Middle shows that reflex is not enough.

Performance is usually highest when the relevant information sits at the start or the end of the input context, and drops clearly when the information has to be pulled from the middle of a long context. This holds even for models advertised as long-context, across two tasks: multi-document QA and key-value retrieval.

The RAG design consequence: if the right evidence is buried in the middle of a long context, the model may skip it — even when it is present. Adding context is not the same as adding evidence, and ranking order becomes part of accuracy.

Four things to build instead of one instruction

Taken together, the work is not to write a longer instruction. It is four things that belong to architecture, not to a prompt.

1. 1Retrieval must bring back evidence that can be challenged, not just evidence that resembles the question. A single similarity signal cannot tell a correct passage from one written to look correct.
2. 2The provenance and authority of evidence must be checked separately, apart from the similarity score. Old, incomplete, or non-authoritative text can still be 'grounded'.
3. 3Citation constraints must live in the structure, not only in the model. A required marker is something the pipeline can check; a promise in a prompt is not.
4. 4Measurement must split missing from wrong into two axes, and must be replayable on the same set to compare policies.

Designing the measurement: separate missing from wrong

If you only measure one accuracy rate, abstention looks like regression: answering less means scoring lower. To see the value of refusing, the measurement has to separate the two situations.

{  "trace_id": "...",  "route": "normal | cache | sensitive",  "context_condition": "supportive | degraded | missing | misleading",  "abstained": false,  "cited_sources": ["S1", "S3"]}

1. 1Run the same question set under all four context conditions: supportive, degraded, missing, misleading.
2. 2Compute the abstention rate when evidence is missing and the abstention rate when evidence is wrong as separate numbers — they mean different things.
3. 3Measure coverage and safety as two sides of one trade-off: the stricter the checker, the more correct answers it throws away.
4. 4Hold everything else fixed — same model, same set — so exactly one variable changes.

This gives you an honest picture: you can see which mechanism pays in coverage and how much safety it buys back. There is no free lunch here, and a good measurement states the price.

Where OpenVie implements these things

OpenVie does not claim that one instruction fixes everything. Instead, these behaviours sit at specific points in the system:

• M3 — Abstention: when there is no relevant evidence, the system reports missing information instead of guessing.
• M2 — Citations [S1]: sources are shown only after Spring checks access permission and stores the final response; drafts carry no links or source cards.
• M12 — The Vietnamese/English sensitive-keyword route skips the semantic answer cache and requires a source marker. This is a structural check, not a trained legal model.
• M25 — Offline evaluation replays anonymised, labelled JSONL traces to grade decisions, routing, and citations, without calling a model.
• M4 — Hybrid retrieval dense + sparse (BM25) + graph, with follow-up question rewriting and RRF rank fusion, supplies the evidence feeding the step above.

What these have in common is that they live in the structure and the measurement layer, not in the wording of a prompt. A required source marker is something the pipeline checks; an instruction to 'cite your sources' is not.

The boundary: what the system does not promise

It is worth stating what is not there, so a structural mechanism does not get turned into an overblown promise.

• There is no promise of 'never hallucinates' or 'perfect accuracy'. Abstention removes one class of error, not all of them.
• M12 is a structural marker check, not a trained legal model.
• There is no Vietnamese law corpus, no trained verification model, and no VLQA/SFT/QLoRA/DPO training.
• The code in the repo is not evidence of a production service; optional provider integrations and deployment hardening have to be verified in your own environment.

In other words: the mechanisms here make errors rarer and easier to detect, but they do not turn a hard problem into a solved one.

In short

Abstention is a behaviour of the system, not a sentence. The public evidence shows that an instruction handles missing context but collapses under wrong context — where 41.6% of questions still get answered, and 63% of those repeat the injected false entity.

1. 1Separate 'missing' from 'wrong' in both the mechanism and the measurement.
2. 2Check the provenance and authority of evidence, not just similarity.
3. 3Constrain citations structurally and re-evaluate offline.
4. 4Acknowledge the coverage-versus-safety trade-off instead of hiding it.

A fluent answer is not a correct answer. A good system is one that knows when to stay silent — and knows that silence is only half of the story.