Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.

Prompt engineering and finetuning aim to maximize language model performance on a given metric (like toxicity reduction). However, these methods do not fully elicit a model's capabilities. To reduce this gap, we introduce activation engineering: the inference-time modification of activations in order to control (or steer) model outputs. Specifically, we introduce the Activation Addition (ActAdd) technique, which contrasts the intermediate activations on prompt pairs (such as"Love"versus"Hate") to compute a steering vector (Subramani et al. 2022). By tactically adding in e.g. the"Love"-"Hate"steering vector during the forward pass, we achieve SOTA on negative-to-positive sentiment shift and detoxification using models including LLaMA-3 and OPT. ActAdd yields inference-time control over high-level output properties (like topic and sentiment) while preserving performance on off-target tasks. ActAdd is lightweight: it does not require any machine optimization and works with a single pair of data points, which enables rapid iteration over steering. ActAdd demonstrates the power of activation engineering.

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment. In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

大语言模型（LLM）经常生成看似合理的思维链（CoT），但其中可能掩盖了模型内部的偏见。我们将这些称为“未言明的偏见”。因此，依据模型明确说出的推理进行监控并不可靠；现有偏见评估通常还需要预先定义类别，并手工制作数据集。本文提出一个全自动黑盒流程，用于检测特定任务中未言明的偏见。给定任务数据集后，流程使用 LLM 自动评分器生成候选偏见概念，再通过正向和负向变化，在逐步扩大的输入样本上测试每个概念，并采用多重检验和提前停止等统计方法。如果某个概念显著影响模型表现，却没有出现在模型 CoT 的理由中，就会被标记为未言明的偏见。我们在六个 LLM、三类决策任务（招聘、贷款审批和大学录取）上评估了这一流程。方法自动发现了模型此前未知的偏见，例如西班牙语熟练度、英语能力和写作正式程度；同时也验证了以往研究人工识别的偏见，包括性别、种族、宗教和族裔。总体而言，该方法为自动发现特定任务中的偏见提供了一条实用且可扩展的路径。

领先的大语言模型服务商如今会隐藏模型逐步推理的过程，也就是思维链，以保护知识产权并减少信息泄露。服务商不会把这些轨迹保存在服务器端，而是以加密文本块的形式返回给客户端；客户端在之后的每次请求中再将这些内容传回。基于以往研究，我们发现一个架构层面的漏洞：同一服务商生态内，不同会话、用户和模型之间的加密文本块完全兼容、可以互换。我们利用这一兼容性，设计出可大规模实施的解密越狱攻击。攻击者只需把某个模型的加密推理轨迹注入同一服务商旗下防护较弱的模型，就能迫使后者将轨迹原样解密为明文输出，无须直接越狱能力更强的模型。该漏洞带来四类攻击。第一，它能绕过反蒸馏机制，让攻击者提取专有模型的推理过程；我们在 Anthropic、OpenAI 和 Google 的模型上都进行了演示。第二，它能用于大规模提取私人数据。开发者经常公开分享会话日志，却不知道日志中的加密文本块包含什么。我们从公开代码库抓取的 315,320 个推理块中解码出 367 条个人身份信息（PII）和 182 个凭据。第三，即使模型最终可见的回复安全地拒绝了恶意请求，解密内容仍可能暴露推理过程中隐藏的危险信息。第四，攻击者可以把恶意载荷完全藏在加密文本块中，通过有毒数据污染公开的智能体运行记录，从而实施隐形提示注入。我们在负责任地披露问题后，提出了具体的密码学和系统层缓解方案，以保护客户端推理过程。