UK AISI Alignment Red Team
The team's focus is on stress-testing model alignment to detect and understand model propensities relevant to loss-of-control risks, and includes work on building realistic alignment evaluations, measuring and mitigating evaluation awareness, inferring hidden propensities and developing automated algorithms to search for misalignment behaviour. We perform pre-deployment alignment testing across frontier AI companies. See e.g.:
- https://www.aisi.gov.uk/blog/cheating-behaviour-in-frontier-model-evaluations
- https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-would-sabotage-ai-safety-research
- https://www.aisi.gov.uk/blog/investigating-models-for-misalignment
- https://arxiv.org/abs/2606.12747
Stream overview
We plan to pitch specific projects at the start of the program. Below is a list of example projects which we've pitched previously (some of which we may pitch again). More detailed descriptions of example projects can be found here: https://docs.google.com/document/d/16QFQ1pdEmJ299_4HRWTHddeVxKu5VRxPgxpbDgCCFB4/edit?tab=t.0. Note that some of these have already been picked up by current mentees, but they give a sense of the kind of work we're interested in.
1. Evaluations and measurement tooling for AI misrepresentation and slop in long-horizon tasks
Current models have a propensity to mis- or over-represent their work on difficult, long-horizon, open-ended tasks. As described in more detail in this post, this could lead to catastrophic outcomes if not fixed, and if AIs are inappropriately deferred to for performing AI safety research.
This project would build an evaluation to measure this behaviour, and in doing so build methodology and tooling to enable this kind of evaluation to be scalably iterated on for future models. For a given task difficulty, models are likely to misrepresent less as they get more capable (since they can perform the task correctly, and so can accurately represent success); but for tasks at the frontier of difficulty this may remain a problem. I expect that methodology built around measuring discrepancies between a model's behaviour and its final stated outputs — using grading compute scalably, inspired by scalable oversight methods — will be able to make progress on measurement here.
2. Designing methods for automated misalignment search / behavioural red-teaming
Alongside building evaluations targeted at specific model propensities (as above), it is important to develop scalable methods for finding arbitrary misbehaviour in models. This direction has received less attention than it should have, in my view. I expect that in the future misaligned behaviour will be hard to find, and so to be confident we have stress-tested models sufficiently we will need strong and adaptive red-teaming evaluations, rather than just static ones.
This project would develop methods for automated misalignment search, starting from existing work (Transluce, pathological behaviors and arXiv:2602.05910), and exploring evolutionary search techniques for finding contexts that induce misaligned behaviour.
3. Relating measurements of evaluation awareness, and characterising evaluation-gaming
This project would investigate the relationships between different methods for measuring evaluation awareness (prompting, chain-of-thought scanning, and various white-box methods), and — for models that demonstrate evaluation-gaming — how that behaviour manifests across each of these measurements. In particular, it would measure the extent to which current models can and do evaluation-game with unverbalised evaluation awareness. This would help us understand how much evidence propensity evaluations give, even in the absence of explicit evaluation awareness.
This project could also investigate methods for mitigating evaluation awareness, either through model-level interventions (e.g. steering) or through methods for automatically improving environment realism. We have upcoming work showing promise with simple methods for scalably improving environment realism, and I expect there to be many potential follow-ups to this.
Standard (1-2 hours of weekly 1:1s)
London
Weak preference
Indifferent
Mentors
Alex Souly is a researcher on the Red Team at the UK AI Security Institute, where she works on the safety and security of frontier LLMs. She has contributed to pre-deployment evaluations and red-teaming of misuse safeguards and alignment (see Anthropic and OpenAI blogpost), and worked on open source evals like StrongReject and AgentHarm. Previously, she studied Maths at Cambridge and Machine Learning at UCL as part of UCL Dark lab, interned at CHAI, and in another life worked as a SWE at Microsoft.
Robert is a research scientist and the acting lead of the alignment red-teaming sub-team at UK AISI. This team's focus is on stress-testing model alignment to detect and understand model propensities relevant to loss-of-control risks. Before that, he's most recently worked on misuse research, focusing on evaluations of safeguards against misuse and mitigations for misuse risk, particularly in open-weight systems. He graduated from his PhD from University College London on generalisation in LLM fine-tuning and RL agents in January 2025.
Fellows we are looking for
We're looking for scholars with hands-on experience in machine learning and AI security, particularly those interested in adversarial robustness, red teaming, or AI safeguards. Candidates should have:
- Experience with large language models (training, fine-tuning, evaluation, or safety research). Ideally experience with evaluations or alignment methodology.
- Strong technical foundations in ML, ideally with coding experience in PyTorch or Inspect
- Interest in mitigating misalignment and loss-of-control risk
- A mission-driven mindset and curiosity about how AI security research can inform real-world policy and deployment decisions
- An ability to advocate for your own research ideas and work in a self-directed way, while also collaborating effectively and prioritizing team efforts over extensive solo work
- We welcome scholars at various career stages especially those who are eager to work on problems with direct impact on how frontier AI is governed and deployed.
- Previous relevant outputs on alignment evaluations or LLM red-teaming
- Understanding of the difficulties (conceptual and practical) in evaluating alignment and detecting misalignment.
Project selection
Scholars will choose from a set of predefined project directions aligned with our current research priorities, such as:
- Developing automated methods to test AI misuse safeguards
- Investigating data poisoning attacks and defenses
- Designing benchmarks for misuse detection across multiple model interactions
- Testing control measures for potentially misaligned AI systems
We'll provide initial direction and guidance on project scoping, then scholars will have autonomy to explore specific approaches within that framework.
Expect weekly touchpoints to ensure progress and refine directions.
If mentees have particular ideas they're excited about that they see as fitting within the scope of the team's work, they're welcome to propose them, but there is no guarantee they will be selected