实证研究

This stream focuses on secret loyalties, where an LLM covertly tries to advance a principal's interests. Secret loyalties have been established as a pressing threat [1], and model organisms of narrow secret loyalties have been constructed and audited [2]. This stream aims to advance the empirical foundations of our understanding of secret loyalties. The goal being that humanity is well-equipped to deal with secret loyalty installation attempts as and when catastrophic secret loyalties become possible in the future.

​

We think being well-equipped looks like having sufficient security measures in place in frontier AI companies, understanding the dynamics and behaviours of secretly loyal AI systems, having effective auditing and verification protocols for secret loyalties and attempts to install them, and these protocols actually being followed by relevant stakeholders.

This coalition of mentors make up the “Anthropic Stream”. This stream spans a range of empirical research areas in AI safety on LLMs, including AI control, scalable oversight, model organisms, model internals, model welfare, security, and more. You’ll be pitched, and have the option to pitch, a variety of safety research projects, and then be matched to projects and mentors based on your interests/preferences on research and what you’d like to get out of MATS. Fellows in this stream frequently receive funding and continued mentorship after MATS to complete their research project, usually leading to a (co-)first author paper. People in this stream often end up in long-term homes for safety research after MATS (e.g. Anthropic, Redwood Research, OpenAI).

​

Anthropic mentors share an application, tend to collaborate and co-mentor projects together, and generally share infrastructure to streamline the fellow experience. By applying to this stream, you are being considered for all of the Anthropic mentors.

We will continue working on black-box monitors for scheming in complex agentic settings, building on the success of the previous stream. Concretely, we will work on scaling our datasets and fine-tuning efforts, as described in the scalable monitoring agenda

​

Most likely the next projects will be about automated iterated red-team vs. blue-team games. We are currently training the blue team. We will then train the red-team and within this stream, we will try and close the loop to train them both synchronously.

This stream focuses on building a Science of Scheming, i.e. what are the mechanisms by which future models might become schemers, even though current models are not. We want to discover empirical Scaling Trends for Scheming. For example, does deceptive alignment become easier to discover with improved model capabilities?

Theory of change: Soon, most important work will be done by AI. AI is going to increasingly advise people and help with important things, many of which are time-sensitive and path dependent, e.g., work on alignment/safety (including various things like how LLMs should behave given that they’re very persuasive); how to think about acausal trade; how to organize society. It seems good for AI to do well at those things.

​

Of course, a lot of the relevant skills for doing well at these tasks are the same skills that cause AI risk and that AI companies work on (and are incentivized to work on) by default; like coding, some kinds of forecasting, etc.

​

We want to make models better at things that are net positive for the future, but that likely won’t benefit much from said default training (or perhaps will even be made worse by such training – e.g., via sycophancy).

​

In practice, a lot of the tasks that we’re interested in from this perspective are what we call “conceptual”: tasks that are hard to verify and don't have clear ground truth but where we nonetheless feel like we can make progress through argument and reason.

​

You can visit conceptualreasoning.ai to get a sense of our work to date.

​

We also take a keen interest in projects directly aimed at making future acausal interactions go well.

I have two broad areas.

​

Security:

I am interested in building demonstrations for hacking real-world AI deployments to show that they are not secure. The goal is to force companies to invest in alignment techniques that can solve the underlying security issues.

​

Verification:

Verification via TEEs or ZKPs

This stream will focus on model motivations and character, open-ended environments, and new forms of misalignment.

This stream will focus on monitoring, stress-testing safety methods, and evals, with a focus on risks from scheming AIs. Examples include (black-box) AI control techniques, white-box monitors (probes etc.), chain-of-thought monitoring/faithfulness, building evaluation environments, and stress-testing mitigations.