1. **Mapping threat models for AI agent insider threats in military and intelligence contexts.** I'd like to lay out the pathways to harm that could arise from misaligned or compromised AI agents operating in high-stakes national security deployments. Some example use cases include cyber offense and defense, decision support, AI-enabled military R&D, and autonomous weapons systems.
2. **Making control sketches to mitigate AI-insider threats, focusing on AI-based monitoring and response.** I want to describe the general control architecture/approach for each use-case (e.g. where monitors sit in the workflow, what response measures are apt, what infrastructure components are needed), and expand how protocols and parameters might vary across sub-cases (e.g., network defense vs. incident response within cyberdefense; analytic triage vs. finished products within intelligence analysis).
3. **Your own project.** We're also open to mentoring fellows who want to pursue their own project in an adjacent area, assuming there's a good fit.
Standard (1-2 hours of weekly 1:1s)
Washington, D.C.
Strong preference
Required
Sambhav 是 IAPS 前沿安全团队的研究助理,研究重点包括国防和国家安全领域的 AI 部署、内部部署模型的标准,以及威胁建模。
加入 IAPS 之前,Sambhav 曾任 Cambridge AI Safety Hub 联合主任,并负责运营 MARS(面向对齐研究学生的导师计划)。
Jan 是 Institute for AI Policy and Strategy(IAPS)的研究员,致力于通过技术治理降低 AI 带来的灾难性风险。他目前在分析国家安全领域使用前沿 AI 可能带来的严重风险,并设计蜜罐机制来保护内部 AI 智能体。
加入 IAPS 之前,他曾参加 GovAI 冬季项目和 Pivotal 项目,并在 CISPA Helmholtz Center for Information Security 攻读博士。他此前的研究涉及可解释性、机器学习安全和 AI 对齐。他拥有信息系统学士学位和计算机科学硕士学位。
1. **Essential**:
**Strong writing:** Can write clearly and precisely for both technical and policy audiences (or has shown clear potential to).
**Transparent reasoning:** Excels at conceptual reasoning, lays out arguments and assumptions explicitly, and brings a skeptical mindset.
**Research Independence:** Has previously taken ownership of a research project and can make progress without frequent direction on what to do next.
**GCR Familiarity:** Is broadly familiar with the core GCR/TAIS literature, and if not, has demonstrated ability to upskill quickly on relevant topics.
**Any of the following would strengthen the application:**
1. **Military/intelligence background:** Has been in, or spent substantial time working on topics relating to, the military or IC.
2. **Technical AI safety background:** Has conducted research in technical AI safety, and in particular, AI control.
3. **LLM power user:** Is skilled at getting useful conceptual work out of LLMs.
4. **Policy background:** Has written memos or policy reports for a non-technical audience, and/or thought extensively about AI governance.