OpenAI Safety Team
Projects on this stream cluster into a few broad areas from the empirical track: scalable oversight, AI control, monitorability and interpretability, adversarial robustness, and security.
Most fellows will work closely with one or two mentors on something that fits into the mentors' ongoing research. The above list of mentors above is tentative.
Stream overview
Projects on this stream cluster into a few broad areas from the empirical track: scalable oversight, AI control, monitorability and interpretability, adversarial robustness, and security. Narrower threads include personas and character training, reward hacking, model spec, automated AI research, and safety eval infrastructure. The list below pulls from what individual mentors on this stream are actively working on, and you can see more of our team's published work at alignment.openai.com. Most fellows will work closely with one or two mentors on something that fits into that mentor's ongoing research line.
Some example projects:
- Oversight for autonomous coding agents, e.g. an evaluator that approves or denies a sandboxed agent's permission escalation requests (auto-review), agent and overseer dynamics such as collusion or post-denial behavior, and comparisons of automated and human oversight.
- CoT and activation monitorability, representation-based interpretability, and privacy-preserving embeddings or monitorability for long-context models.
- Control and anti-scheming work, including control red-teaming and CoT monitoring studies that use production data to build evals.
- Adversarial robustness and automated red-teaming, including universal jailbreaks against held-out defenders, prompt injection, training via games, and stress-testing abuse defenses like burner-account and KYC checks.
- Data poisoning and synthetic data work tied to pretraining and alignment.
- Personality and character training and evaluation.
- Safety training infrastructure, grader and eval tooling, and Codex or agent workflows that productionize safety research.
- More conceptual work on the model spec, automated AI research, and concentration of power.
Mentors
Gabriel Wu is an AI alignment researcher at OpenAI. Previously, he directed the AI Safety Student Team at Harvard, where he earned a Master's degree in Computer Science and a bachelor's degree in Mathematics.
Dylan is a safety researcher at OpenAI, where he works on curating better/safer training data and monitoring models for harmful behavior.
Before that he completed a PhD in the Machine Learning Department at CMU.
Kaiwen is a researcher at OpenAI working on AI Safety and RL. He earned my Ph.D. from Cornell Tech, where he researched and taught RL, causal inference, and LLMs.
He previously worked at Google, Microsoft, and Netflix on projects spanning core RL theory to scalable LLM algorithms. Before grad school, Kaiwen spent two years at Facebook building the RL Platform.
Tom is a research scientist at OpenAI, working on interpretability of language models, for AI safety. He was also a core developer of scikit-learn between 2015 and 2022.
Isak is a Member of Technical Staff at OpenAI. Previously a Software Engineer at Google, he worked on applications of computer vision, natural language processing, and LLMs.
Isak earned a Master of Computer Science at Carnegie Mellon University, with published work in natural language processing, style transfer, multilingual grapheme-to-phoneme modeling, and computer vision.
Xiangyu is a researcher at OpenAI, where he works to make LLMs robust. Previously, he obtained his Ph.D. from Princeton University, advised by Prof. Prateek Mittal and Prof. Peter Henderson.
Joseph works in Detections and Response at OpenAI. His public security work includes using large language models to detect malicious macOS activity.
Juan is a researcher in OpenAI’s Safety Systems team. He is broadly interested in mitigating catastrophic risks. He works on adversarial robustness training and automated red-teaming (recent work https://openai.com/index/instruction-hierarchy-challenge/).
Hani is a software engineer at OpenAI.
My focus these days is on adversarial machine learning: safety, security, and alignment of frontier models. I am particularly interested in alignment/safety RL and evaluations. In the past, I studied memorization, privacy, and security harms in language modelling, including auditing for risks and mitigating them. I've also worked on DP training algorithms, unlearning, collaborative learning approaches, and methods for ownership-verification.
James is a Researcher at OpenAI working on model personality, post-training, and personalization.
Jason is a Member of Technical Staff at OpenAI working on alignment and model behavior.
Ollie is a researcher on OpenAI’s Alignment team interested in red-teaming and control. He was previously on the Control team at UK AISI.
Micah is a researcher on OpenAI’s safety team interested in AI deception, scalable oversight, and monitorability. He is on leave from a UC Berkeley PhD focused on AI alignment with influenceable humans, AI manipulation from RL training, and recommender-system effects.
Bijan is a Technical Program Manager at OpenAI. He previously worked as a research engineer at Scale AI, where he coauthored work on LLM jailbreaking and red-teaming workflows.
Maja is a researcher at OpenAI, working on techniques for improving control and alignment as AI systems become more capable and agentic. Her team’s work combines longer-horizon research with hands-on deployment. They study long-term questions about how increasingly intelligent systems can be supervised, constrained, and corrected, while also building oversight systems that are used in practice today, both internally and externally (see recent work on code review and action monitoring for codex).
Sam is a Research Engineer on OpenAI’s Alignment team. Previously worked in NYU’s Alignment Research Group on scalable oversight and as a Software Engineer at Amazon. His research includes training language models to win debates with self-play, and recent OpenAI work on auto-review for agent actions.
I’m a Member of Technical Staff at OpenAI working on monitoring LLM agents for misalignment. Previously, I worked on AI control and safety cases at the UK AI Security Institute and on honesty post-training at Anthropic. Before that, I did a PhD at the University of Sussex with Chris Buckley and Anil Seth focusing on RL from human feedback (RLHF) and spent time as a visiting researcher at NYU working with Ethan Perez, Sam Bowman and Kyunghyun Cho.
Mentorship style
Fellows we are looking for
Essential:
- Strong research ability, technical judgment, and the capacity to execute a substantial project (paper, benchmark, dataset, or production-quality tooling) over the course of the fellowship.
- Genuine interest in AI safety and in at least one of the project areas above. Fellows are matched to mentors based on fit, so concrete overlap with one or two mentors' interests matters more than broad interest.
Preferred (at least one of):
- Empirical ML research experience with large language models, and strong Python and software engineering skills.
- Prior research or applied work in one of the project areas above.
- For more conceptual work on model spec, automated AI research, and concentration of power, strong writing and conceptual analysis skills instead of empirical ML.