Lee Sharkey, Lucius Bushnaq
Lee's stream will focus primarily on improving mechanistic interpretability methods for reverse-engineering neural networks.
Stream overview
Understanding what AI systems are thinking seems important for ensuring their safety, especially as they become more capable. For some dangerous capabilities like deception, it’s likely one of the few ways we can get safety assurances in high stakes settings.
The ability to reverse engineer what neural networks have learned promises one of the few ways that we might get assurances for safe generalization behaviour, especially as AI systems become vastly more capable than humans.
Safety motivations aside, capable AI systems are extremely interesting objects of study, and doing digital neuroscience on them is comparatively much easier than studying biological neural systems.
A majority of reverse-engineering-focused interpretability work has involved sparse dictionary learning, which has various issues Sharkey et al., 2025. In response to these issues, my team and I at Goodfire (and previously Apollo Research) developed a new approach to decomposing neural networks, called Parameter Decomposition. We have used parameter decomposition to resolve feature splitting, identify attention-head distributed computations, identify circuits, and more.
We believe there are gaps in our method that remain to be solved. In our work (whether SAEs or parameter decomposition), we minimize 'description length'. But we are not yet confident that we have the right 'type of description'. We think understanding computational manifolds (which are projections of activation manifolds) is likely part of the answer here, since they may offer an even more concise description of neural computation than SDL latents or VPD parameter subcomponents.
MATS projects in my stream should primarily be aimed at improving methods for reverse engineering neural networks and at least be conceptually informed by parameter decomposition, manifolds, and minimum description length framings of interpretability, if not build on them directly.
Standard (1-2 hours of weekly 1:1s)
London
Weak preference
Indifferent
Mentors
Lee Sharkey is a Principal Investigator at Goodfire.
His team has focused on improved interpretability methods, including parameter decomposition methods such as Attribution-based Parameter Decomposition and Stochastic Parameter Decomposition and adVersarial Parameter Decomposition.
Previously, Lee was Chief Strategy Officer and cofounder of Apollo Research, and a Research Engineer at Conjecture, where he worked on sparse autoencoders as a solution to representational superposition.
Lucius Bushnaq is a Research Scientist at Goodfire.
He works on parameter decomposition methods for interpretability, such as Attribution-based Parameter Decomposition, Stochastic Parameter Decomposition, and adVersarial Parameter Decomposition. Alongside this, he works on learning theory and the theory behind interpretability, for example theoretical frameworks for computation in superposition and connections between singular learning theory and algorithmic information theory.
Previously, Lucius was a member of the interpretability team at Apollo Research, where he worked on the Local Interaction Basis and degeneracy in the loss landscape. He holds a PhD in physics.
Fellows we are looking for
- Previously demonstrated interpretability interest: At least some nontrivial demonstration of prior experience with mechanistic interpretability. This could include a research paper or a small research project with public results. Work from closely related fields (such as computational neuroscience) would also pass, as long as it is very similar to mechanistic interpretability/reverse engineering neural networks. This requirement may be waived for otherwise exceptional candidates.
- Scientific taste. Evidence of having generated and tested their own hypotheses, designed experiments that could actually falsify them.
- Mathematical depth beyond the basics. Fluency in areas relevant to interpretability, including ones not yet widely used in the field but with clear potential: optimisation, information theory, statistical learning theory, sparse coding and compressed sensing, tensor decompositions, group theory or representation theory, dynamical systems, singular learning theory.
- Strong engineering. Experience with larger codebases, distributed or multi-GPU training, profiling and performance work, and good practices around testing and reproducibility. Comfort with the wider scientific Python stack, and, increasingly important, experience with agentic coding workflows.
- Experience with specific interpretability approaches. Hands-on familiarity with e.g. sparse autoencoders, transcoders, circuit analysis, DAS, causal abstraction, or other frameworks, and awareness of their failure modes. Prior interest in decomposing parameters rather than only activations is a plus.
- A developed theory of impact. Engaged deeply with the AI safety literature and arrived at research directions that follow coherently from a view about what actually reduces risk. Able to argue for and against their own agenda.
- Conscientiousness. A track record of finishing things: shipped code, written-up results, completed theses, published posts or papers. In interpretability, as in art, projects are never finished, only abandoned, so the ability to bring work to a completed-enough state matters a great deal.
- Clear writing and exposition. Blog posts, talks, or papers that explain technical ideas well to a non-expert audience.
- Independence with good judgement about when to ask. Can make progress without supervision for days at a time, but escalates blockers early rather than sinking time into them.
Project selection
In general I'd like projects in my stream to at least be conceptually informed by parameter decomposition, manifolds, and minimum description length framings of interpretability, if not build on them directly.
Scholars and I will discuss projects and come to a consensus on what feels like a good direction. I will not tell scholars to work on a particular direction, since, in my experience, intrinsic motivation to work on a particular direction is important for producing good research.