How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions

MATS 学员:

Lorenzo Pacchiardi, Alex Chan, Ilan Moscovitz

作者:

Lorenzo Pacchiardi, Alex J. Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y. Pan, Yarin Gal, Owain Evans, Jan Brauner

引用次数

87 Citations

摘要:

Large language models (LLMs) can"lie", which we define as outputting false statements despite"knowing"the truth in a demonstrable sense. LLMs might"lie", for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a predefined set of unrelated follow-up questions after a suspected lie, and feeding the LLM's yes/no answers into a logistic regression classifier. Despite its simplicity, this lie detector is highly accurate and surprisingly general. When trained on examples from a single setting -- prompting GPT-3.5 to lie about factual questions -- the detector generalises out-of-distribution to (1) other LLM architectures, (2) LLMs fine-tuned to lie, (3) sycophantic lies, and (4) lies emerging in real-life scenarios such as sales. These results indicate that LLMs have distinctive lie-related behavioural patterns, consistent across architectures and contexts, which could enable general-purpose lie detection.

近期研究

非大国冲突与 AI 风险

作者:

Kristina Kempkey

日期:

August 26, 2026

引用次数:

合成角色预训练:从第一个 token 开始对齐

作者:

Julian Minder

日期:

August 13, 2026

引用次数:

常见问题

什么是 MATS 项目?
项目持续多久?