Toolkit / Mechanistic interpretability
Inference-time decomposition of activations
Scalable, cross-model alternative to SAEs for mechanistic interpretability
| Dataset | Model | Application |
With Inference-time decomposition of activations (ITDA) Patrick Leask, Neel Nanda, and Noura Al Moubayed contribute a new technique to the field of mechanistic interpretability. ITDA is similar in concept to Sparse Autoencoders (SAEs), yet offers three key advantages:
- Efficiency: with hundred-fold reductions in training time and tokens over SAEs, ITDAs expand the range of viable open source mechanistic interpretability studies to models with hundreds of billions of parameters.
- Transferability: unlike SAEs, the dictionaries ITDAs learn transfer across models, e.g. from one LLM to another.
- Interpretability: ITDA labels each learned atom with both the prompt and token that occassioned it; thus, unlike SAE latents, ITDA atoms feature an inherent interpretability grounded in the context of their generation.
Compared to SAEs in terms of reconstruction performance, ITDAs achieve similar results. For some models, they are slightly better—for others, slightly worse.
Stylized system-level diagram of inference-time decomposition of activations. To decompose a target token in a prompt (1), we compute its activation at a certain point in the LLM (2), for example after layer 8 in GPT-2. We use matching pursuit to decompose the activation into a sparse code of other activations collected from the same point in the LLM (3). Our dictionary atoms are labeled with the prompt and token for which the activation was collected, which allows direct interpretation of the sparse code (4). (Leask, Nanda & Al Moubayed 2025, p.2)
