Dan Murfet, Jesse Hoogland

We expect to support research projects applying singular learning theory (SLT) to AI safety, especially interpretability and alignment. 

​

We support both empirical projects (e.g., applying Bayesian Influence Functions to study or improve unlearning) and theoretical projects (e.g., proving convergence properties of SGMCMC-based sampling methods). Most projects will involve both components: your specific focus will depend on your background and interests.

​

On the empirical side, a typical project will involve the following SLT-based tools:

• Local Learning Coefficients (LLCs) and refined LLCs, which capture model complexity and generalization in singular settings.
• Susceptibilities, which measure how model components respond to perturbations/distribution shifts.
• Bayesian Influence Functions (BIFs), which measure how model predictions respond to perturbations/distribution shifts.

​

With these tools, we investigate diverse areas, such as the following:

• Developmental interpretability: How do models acquire structure over training?
• Spectroscopy ("Circuit discovery"): How can we locate which structures are responsible for model generalization? 
• Unlearning: Can we erase structure/knowledge in a targeted way?
• Emergent misalignment: Can we predict at what fraction of harmful code data we end up with emergent misalignemnt? 
• Elicitation: Can we design data perturbations that elicit harmful outputs?

The team will meet weekly together with both mentors. Separately, you will meet 1-on-1 with at least one of the mentors every other week. We conduct our asynchronous communications through an internal Discord server. We expect you to schedule additional pair-programming/debugging calls with other people on the team as needed.

​

We'll help with research obstacles, including outside of meetings.

An overview of our research agenda: 

• You Are What You Eat -- AI Alignment Requires Understanding How Data Shapes Structure and Generalisation

​

SLT for Data Attribution:

• Bayesian Influence Functions for Hessian-Free Data Attribution
• The Loss Kernel: A Geometric Probe for Deep Learning Interpretability
• Influence Dynamics and Stagewise Data Attribution

​

SLT for Interpretability:

• Loss Landscape Degeneracy and Stagewise Development in Transformers
• Structural Inference: Interpreting Small Language Models with Susceptibilities
• Embryology of a Language Model

​

Foundations of SLT:

• Programs as Singularities
• Modes of Sequence Models and Learning Coefficients
• Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

If you're interested in working on more of the empirical side, you should have prior experience with ML engineering (at least at the level of a program like ARENA) and prior research experience (potentially in a field outside of ML). A bonus would be prior familiarity with designing and running ML experiments or research specifically in AI safety. 

​

If you're interested in working on more of the theoretical side, you should have prior research experience in a relevant field like mathematics, theoretical physics, or theoretical computer science. 

​

Please make sure that your background and interests are clearly described in your application. By default, we'll be looking for evidence of research ability in the form of publications. 

​

We do not expect you to already be aware of SLT, but if you pass the first round, please prepare by conducting some background reading (see: timaeus.co/learn). 

You will most likely be working in a team with 2-4 other people, led by one of this stream's mentors. In some cases, scholars also work on their own or with their own collaborators.