Although invented in the 1990s, SAEs came to prominence as a promising mechanistic interpretability technique im the 2020s through research carried out, in particular, Anthropic (see e.g. 2022, 2023, 2024).

These studies suggest that the apparently uninterpretable, polysemantic features learned by neural networks’ individual neurons are actually superpositions of interpretable, monosemantic, features; in this way, a given model can learn more features by ‘cramming’ them into orthogonal subspaces. SAEs allow researchers to recover a list of interpretable, monosemantic features from the neural network. The number of features that a SAE learns is determined by its dictionary size hyperparameter.

Especially in light of the fact that SAEs are computationally demanding to train, the import of this hyperparameter choice calls for a systematic exploration of its effects. Leask et al. developed the SAE stitching technique in order to investigate this particular probblem.

Stitching

In general, at any given size, SAEs feature an inherent tradeoff between sparsity (L0) and reconstruction error (MSE). Yet, as one dials-up the dictionary size hyperparameter, the resulting SAE learns ever more features. How useful is this?

Using the cosine similarity the features of a larger and smaller SAE, Leask et al. show that, of the ‘additional’ features picked up in this way, about 35% can be characterised as “novel” whereas the remaining 65% merely “reconstruct” the features already present in the smaller SAE:

Effect on MSE from large vs. small features by cos similarity — Change in MSE when adding each feature from GPT2-1536 to GPT2-768, plotted against the maximum cosine similarity of that feature to any feature in GPT2-768. Features with cosine similarity less than 0.7 tend to improve MSE, while more redundant features hurt performance. A few extreme outliers with very high cosine similarity and effect on MSE are not visible in this plot. (Leask et al. 2024, p. 2)

Thus, Leask et al. develop SAE stitching in order to get only the novel features’ beneficial effect on reconstruction performance:

A chart showing interpolation between differently-sized sparse autoencoders — It is possible to smoothly interpolate between sparse autoencoders of different sizes by inserting or switching latents, where every insertion or switch results in a strict improvement in reconstruction (MSE). First, novel latents are added result in an increase in the L0; then the remaining latents are replaced with their similar latents in the larger SAE, leading to a decrease in the L0. (Leask et al. 2024, p. 4)

This technique plays a key role in the study “Sparse Autoencoders Do Not Find Canonical Units of Analysis” (Leask et al. 2025).