幻觉检测是识别大模型输出中与事实不符、无来源支撑或逻辑自相矛盾内容的任务,核心能力是在模型自身的陈述与外部证据之间做交叉验证,给出可信度评估而非仅仅判断真假。它解决的是模型会流畅地编造事实这一根本特性带来的风险,也解决了专业场景无法直接采信模型输出的问题。
模型说错话时语气和说对时完全一样,因此不能靠「读起来通顺」来判断可信度。
事实一致性校验:将生成内容拆成可核验的断言,逐条比对权威知识库。
来源可追溯:要求每个关键陈述都能指向具体出处,缺失即标记为存疑。
内在一致性:检查同一输出内部是否存在前后矛盾或推理断裂。
置信度评估:结合检索证据强度与模型自身的不确定性给出分级判断。需要警惕的是检测器自身也会出错:过于严格的判定会大量误报,把大量正确内容标为存疑,最终导致整套机制被使用者忽略。评测这一能力本身也需要基准:常用做法是构造带已知错误的陈述集,测量检测器在真实错误与合理表述之间的区分能力。检测结论也应随模型版本更新,否则依据会随模型迭代而失效。
幻觉检测让大模型从直接采信变成有条件采信,是它进入法律、医疗、金融等专业场景的必要前提,因为这些场景的错误代价无法由模型的流畅表达来承担。
翻译
Hallucination detection is the task of identifying content in a model’s output that is factually wrong, unsupported by any source, or internally contradictory, and its core capability is cross-checking the model’s own claims against external evidence so that confidence can be graded rather than a binary true-or-false verdict applied. It addresses the underlying property that a model states falsehoods with exactly the same fluency as truths.
When a model is wrong it sounds precisely as confident as when it is right, so plausibility cannot be used as a reliability signal.
Fact-consistency checking: the output is decomposed into verifiable assertions and each is compared against an authoritative knowledge base.
Source traceability: every significant claim must point to a concrete origin, and the absence of one marks the claim as suspect.
Internal consistency: contradictions or broken reasoning within a single output are detected.
Confidence grading: the strength of retrieved evidence is combined with the model’s own uncertainty to produce a tiered judgement.One should be wary that the detector errs too: an over-strict judgement produces large numbers of false positives, marking much correct content as suspect, and a mechanism whose output is mostly noise eventually gets ignored by the people relying on it.The capability itself needs a benchmark: the usual approach is to construct a set of statements with known errors and measure how well the detector separates genuine mistakes from reasonable statements.Detection conclusions must also be updated with each model version, otherwise the basis lapses as the model evolves.
Detection changes large models from something to be taken at face value into something to be accepted conditionally, which is the precondition for use in professional settings.