Targeted Recovery of Weight-Space Mechanisms From Neural Networks

MATS Fellow:

Antoine Vigouroux

Authors:

Antoine Vigouroux, Lee Sharkey

Citations

Citations

Abstract:

Parameter decomposition (PD) decomposes neural networks into interpretable computational components that faithfully reflect the original network's operations. However, scaling PD to large models requires vast compute, making it a costly and risky endeavor. Here we propose targeted PD (tPD), which identifies only the components that process specific inputs of interest -- from isolated prompts to large subtasks -- by introducing a high-rank catch-all component that handles all non-target data. We validate tPD on toy models and on transformer language models trained on The Pile, where it recovers reproducible, mechanistically faithful circuits. We extract a CSS-only submodel of a 4-block transformer using 7% of the FLOPs of its published decomposition, and in a 12-block transformer we surgically ablate and rewire memorized sequences, with negligible side effects on other inputs.

Recent research

Uncovering Hidden Propensities in Language Models via Limited-Parameter Finetuning

Authors:

Boyd Kane, Jo Jiao, Elizabeth Donoway, Alex Cloud

Date:

December 7, 2026

Citations:

Frequently asked questions

What is the MATS Program?
How long does the program last?