Tomek Korbak
I mostly interested in AI control and scalable oversight. I'm excited to work with scholars interested in empirical projects building and evaluating control measures and oversight techniques for LLM agents, especially those based on chain of thought monitoring. I'm also interested in the science of chain of thought monitorability, misalignment and control. An ideal project ends with a paper submitted to NeurIPS/ICML/ICLR.
Stream overview
Some example projects:
- Forcing AI agents to externalize their cognition, either through information bottlenecks in multi-agent architectures or training techniques
- Training AI agents to be more honest
- Training AI agents to be have more monitorable chain of thought traces
- Agentic monitors: ones that take actions, retrieve additional context and interrogate overseen agents
Mentors
I’m a Member of Technical Staff at OpenAI working on monitoring LLM agents for misalignment. Previously, I worked on AI control and safety cases at the UK AI Security Institute and on honesty post-training at Anthropic. Before that, I did a PhD at the University of Sussex with Chris Buckley and Anil Seth focusing on RL from human feedback (RLHF) and spent time as a visiting researcher at NYU working with Ethan Perez, Sam Bowman and Kyunghyun Cho.
I'll meet with mentees once a week and will be available on Slack daily. By default, I'll try to pair mentees to work on a project together.
Mentorship style
I'll meet with mentees once a week and will be available on Slack daily.
I'll meet with mentees once a week and will be available on Slack daily. By default, I'll try to pair mentees to work on a project together.
Representative papers
- Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
- AI Control: Improving Safety Despite Intentional Subversion
- Eliciting Language Model Behaviors with Investigator Agents
- LLM Critics Help Catch LLM Bugs
- When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Ctrl-Z: Controlling AI Agents via Resampling
- Alignment faking in large language models
- Auditing language models for hidden objectives
- CoT May Be Highly Informative Despite “Unfaithfulness”
- Training LLMs for Honesty via Confessions
Scholars we are looking for
An ideal mentee has a strong AI research and/or software engineering background. A mentee can be a PhD student and they can work on a paper that will be part of their thesis.
By default, I'll try to pair mentees to work on a project together.
Project selection
I'll talk through project ideas with scholar
Community at MATS
MATS Research phase provides scholars with a community of peers.
.webp)
During the Research phase, scholars work out of a shared office, have shared housing, and are supported by a full-time Community Manager.
Working in a community of independent researchers gives scholars easy access to future collaborators, a deeper understanding of other alignment agendas, and a social network in the alignment community.
Previous MATS cohorts included regular lightning talks, scholar-led study groups on mechanistic interpretability and linear algebra, and hackathons. Other impromptu office events included group-jailbreaking Bing chat and exchanging hundreds of anonymous compliment notes. Scholars organized social activities outside of work, including road trips to Yosemite, visits to San Francisco, and joining ACX meetups.