Can researchers genuinely reverse-engineer internal representations and algorithms rather than produce suggestive but unreliable explanations?
Sparse-autoencoder feature extraction has scaled to frontier models, but coverage and faithfulness remain contested. Amodei frames interpretability as being in a race against capability — one it is currently losing.
View on the map → · Open in Browse →
The field split into a public strategy debate — DeepMind's 'pragmatic' pivot away from full reverse-engineering versus Leo Gao's 'ambitious' defense of it — while shipping a possible successor to sparse autoencoders: natural-language autoencoders that verbalize activations without labeled data, already used to surface hidden evaluation awareness. Running underneath both is a faithfulness worry, with multiple results showing the sparsity objectives behind current circuit tools reward interpretable-looking explanations that misdescribe the actual computation.
Drop full reverse-engineering; target safety-relevant understanding grounded in proxy tasks.
The Google DeepMind mechanistic interpretability team has made a strategic pivot over the past year, from ambitious reverse-engineering to a focus on pragmatic interpretability
The GDM interp team's manifesto abandoning full reverse-engineering in favor of proxy-grounded, safety-targeted interpretability — one pole of the field's central strategy debate.
Fully understanding networks is feasible and uniquely catches what behavioural tests miss.
The goal of ambitious mechanistic interpretability (AMI) is to fully understand how neural networks work.
The direct rebuttal to the pragmatic pivot: defends fully reverse-engineering networks as both feasible and uniquely valuable for catching scheming invisible to behavioral tests — the other pole of the live strategy debate.
Sparsity objectives can reward interpretable-looking explanations that misdescribe the computation.
pruned circuits can achieve low cross entropy loss on a nonsensical task; solve tasks using uniform attention patterns even when the original model's attention pattern was importantly non-uniform
Replicates the weight-sparse circuits result but presents four lines of evidence that the extracted circuits misdescribe the model's actual computation — arguing the loss-vs-circuit-size metric rewards interpretable-looking but unfaithful explanations.
An NLA consists of two LLM modules: an activation verbalizer (AV) that maps an activation to a text description and an activation reconstructor (AR) that maps the description back to an activation.
A candidate successor to sparse autoencoders: a natural-language bottleneck that verbalizes activations without labeled data, already used to surface unverbalized evaluation awareness during audits.
we train a single LoRA adapter, the IA, that causes all of these fine-tuned models to state what they learned.
Trains a single adapter that makes fine-tuned models verbally report behaviors they acquired during training, turning introspection into an auditing tool for discovering hidden learned behaviors.