What should an enterprise RAG system do when the knowledge base does not actually contain the answer? The short answer: do not rely on one similarity threshold. Put four gates in front of every answer: search scope, relevance filtering, evidence sufficiency, and post-generation grounding. If any gate fails, the assistant should limit the answer, ask a clarifying question, or refuse instead of turning loosely related text into a confident policy.
This is one of the most common production failures in enterprise knowledge base projects. Vector search always returns a Top-K list. Large language models are very good at stitching nearby fragments into fluent prose. That combination can make a wrong answer look operationally credible, especially in HR, finance, procurement, legal, and IT support workflows.
Why “Retrieved” Does Not Mean “Answered”
Vector retrieval answers a ranking question: which chunks are closest to the query? It does not answer the governance question: do these chunks prove the conclusion that the assistant is about to give?
In production, you need to separate three cases. First, the evidence is complete: the entity, condition, time period, location, version, and conclusion are all present. Second, the evidence is partial: the topic is related, but one or more required conditions are missing. Third, the evidence is merely similar and cannot support the answer. The second case is the dangerous one, because the model may fill the gap with plausible business logic.
Gate 1: Lock the Search Scope Before Retrieval
No-answer control starts before the model sees any context. The system must know who the user is, which knowledge bases they can access, which document versions are valid, and which sources are excluded for compliance reasons.
If an employee only has access to public IT policy documents, the retrieval layer must not bring back internal draft procedures. Even if a draft contains the right answer, the assistant cannot use it. Permission filtering belongs before retrieval, not inside a polite instruction that asks the model to behave.
Gate 2: Use Relevance as a Filter, Not a Final Judge
A similarity threshold is still useful, but it is only a coarse filter. Embedding models, distance functions, chunk size, document length, and query rewriting can all move the score. Copying a threshold like 0.78 or 0.82 from another project is not production governance.
Build two evaluation sets instead: questions the knowledge base can answer and questions it cannot answer. Then tune the threshold against both false refusals and false approvals. In many B2B scenarios, a false approval is much more expensive than a false refusal, but the right tradeoff depends on the workflow risk.
Gate 3: Test Evidence Sufficiency
Evidence sufficiency asks a stricter question: do the retrieved passages cover every condition needed to answer? A process question needs complete steps. A comparison needs evidence for both sides. A regional policy question needs the right location. A version-sensitive question needs the current document. A multi-hop question may require several passages to support one conclusion.
A practical pattern is to parse the user query into required facts, then bind each fact to cited evidence. The model can help produce the checklist, but it should not fill missing conditions from general knowledge.
{
"decision": "answer | limited_answer | clarify | refuse",
"evidence": ["policy_v3#chunk_18", "travel_expense_faq#chunk_04"],
"missing_conditions": ["overseas receipt replacement rule", "current effective version"],
"reason": "retrieved passages are related, but do not cover the overseas scenario",
"next_action": "ask_clarifying_question"
}Gate 4: Verify the Generated Answer
Even after retrieval and sufficiency checks pass, generation can still introduce unsupported numbers, dates, exceptions, owners, or deadlines. Enterprise RAG should not only append citations. It should verify whether each factual claim can be traced back to the evidence selected for this turn.
Unsupported statements should be removed, narrowed, or converted into a clarification or refusal. A real file name is not enough. The answer needs a traceable evidence chain.
How Should the System Degrade?
- Controlled retry: rewrite the query, expand candidates, or add one keyword search pass, but keep a strict attempt limit.
- Clarification: ask for the missing region, product, version, department, or date instead of guessing.
- Limited answer: answer only the part supported by the documents and name the missing conditions.
- Refusal: refuse when the evidence is missing, unauthorized, outdated, or conflicting.
A Minimal Engineering Pattern
The most reliable implementation is to place the answerability decision in the RAG pipeline, not only in the prompt. The prompt is one control, but production systems need observable gates, logs, and reviewable decisions.
def should_answer(query, evidence, policy):
scope_ok = check_scope(query.user, evidence.documents)
relevance_ok = max(e.score for e in evidence) >= policy.min_relevance
sufficiency = judge_required_facts(query, evidence)
grounded = verify_generated_facts(evidence)
if not scope_ok:
return "refuse", "outside_allowed_scope"
if not relevance_ok:
return "refuse", "low_relevance"
if not sufficiency.ok:
return "limited_answer", sufficiency.missing_conditions
if not grounded.ok:
return "revise_or_refuse", grounded.unsupported_claims
return "answer", "grounded"
Pre-Launch Checklist
- Are user permissions, source scope, and document version filtered before retrieval?
- Do you test both answerable and unanswerable questions when tuning thresholds?
- Can the system distinguish low relevance, partial evidence, conflicting evidence, and outdated evidence?
- Does every answer record the evidence used, missing conditions, and release or refusal reason?
- Does post-generation verification remove unsupported numbers, deadlines, exceptions, and strong claims?
- Do you have separate actions for retry, clarification, limited answer, and refusal?
FAQ
Can the highest similarity score decide whether there is no answer?
No. The highest score only means one chunk is closer than the other retrieved chunks. It does not prove that the evidence covers the required entity, scope, time period, version, and conclusion.
Is a prompt instruction enough?
No. A prompt such as “refuse when context is insufficient” helps, but it cannot replace access filtering, sufficiency scoring, and generated-claim verification. Prompts are part of the control plane, not the whole control plane.
What if the assistant refuses too often?
Diagnose the reason before lowering the global threshold. The real cause may be broken query rewriting, missing source permissions, poor chunking, stale documents, or domain terms that the embedding model does not capture well. Lowering the threshold can quietly approve a new class of unsafe answers.
How DELine Helps Enterprise Teams Implement This
DELine helps organizations design, evaluate, and deploy enterprise RAG and AI Agent workflows with practical controls: source governance, access filtering, retrieval evaluation, evidence sufficiency rules, post-generation verification, audit logs, and rollout testing. The goal is not to make the system better at improvising. The goal is to make it reliable enough to stop when the evidence is not there.
If your team is preparing an internal knowledge base assistant or AI Agent workflow, start with a no-answer control review. Testing 20-50 real high-risk questions usually reveals whether the system is answering from evidence or filling gaps with fluent guesses.
