实证研究

At Fourth Eon Biosecurity we're building adaptive, AI-native safeguards across the bioengineering stack, with a focus on function-based DNA synthesis screening. Fellows in this stream will work on technical research projects at the intersection of AI safety and biosecurity, aimed at reinforcing screening and generalizing detection beyond known threat signatures. Projects span mechanistic interpretability of bio foundation models, model evaluations for biosecurity-relevant capabilities, and agentic sequence analysis workflows.

I'm interested in better understanding and controlling how post-training causes alignment-relevant behavior. This is a pretty broad area, and I’m open to many approaches to these problems! Potential areas of study / methods of attack might include model organisms, training run science/ablations, root causing strange behaviors, or studying how best to robustly induce behaviors or values or beliefs into models.

The stream focuses on evaluating and/or mitigating catastrophic risk emerging from dangerous scientific capabilities in frontier AI systems, with an emphasis on the challenges that emerge from lab integrations and novel science. Potential research directions include evaluation design, risk mitigations and evaluation science.

This stream focuses on critical challenges in AI safety and alignment, including risks from automating AI research, bottlenecks to recursive self-improvement, and the automation of safety and alignment research. Priority topics also include AGI privacy, measuring long-horizon agentic capabilities, developing new alignment methods, and advancing the science of post-training.

Research papers (technical governance or ML) related to evaluating and mitigating dangerous AI capabilities, with a focus on what's actionable and relevant for AGI companies

Neel takes a pragmatic approach to interpretability: identify what stands between where we are now and where we want to be by AGI, and then focus on the subset of resulting research problems that can be tractably studied on today's models. This can look like diving deep into the internals of the model, or simpler black box methods like reading and carefully intervening on the chain of thought - whatever is the right tool for the job. This could look like studying how to detect deception, understanding why a model took a seemingly concerning action, or fixing weak points in other areas of safety, e.g. using interpretability to stop models realising they are being tested. You can learn more about Neel's approach in this podcast.

He has spent far too much time having MATS scholars, and has worked with ~60 so far - he’s excited to take on even more!

Computational/modelling problems in biosecurity.

Projects in this stream will be on AI welfare and moral status; more specifically, on what it takes to be a moral patient and how we can determine whether AI systems meet the conditions. I'm looking for applicants who have ideas about these topics and are motivated to explore them in more detail.