AI circuitry may be easier to read than researchers thought
- Nick Cammarata argues that many neural-network detectors have clear, readable functions, contrary to years of assumptions.
- Researchers see potential for better safety checks, but readable parts don’t prove a whole model is safe.
Nick Cammarata’s research challenges the idea that neural networks are inherently too tangled to interpret. He argues that researchers mistook blurry inspection tools and outdated model designs for evidence that AI circuitry was unreadable.
A famous example was a detector thought to respond to both cat faces and car fronts, a sign, researchers believed, that one neuron could hold unrelated concepts. Cammarata says it was actually responding to pointed, V-shaped features. He also claims that roughly nine out of ten detectors in modern OpenAI models have one primary function.
Cammarata reports that larger models can be easier to interpret, with broad detectors becoming more specific as models scale. OpenAI researcher Rohun called the work a historic scientific breakthrough and argued that inspecting a model’s internal plans could help reveal deception before it appears in the model’s answers.
That possibility is not a safety guarantee. Google DeepMind interpretability lead Neel Nanda noted that language models store more facts than they have neurons, while researcher Liv Boote argued that branching concepts do not rule out capacity-driven cramming. Even if individual detectors are readable, researchers still have to show that they can understand how those parts work together.
Source Intelligence
- Deep dive into what was said in the episodes
Nathaniel Whittemore
A Potentially Historic AI Breakthrough • Oct 18
- Nick Cammarata published a paper asserting that neural network interior circuits are readable, challenging years of consensus that models are too tangled to interpret. OpenAI researcher Rohun called the finding a historic scientific breakthrough.
- An OpenAI project in 2023 found only 2% of GPT-2 neurons were legible with primitive tools. Cammarata claims modern tools reveal roughly nine out of ten detectors in current OpenAI models have one primary function.
- Researchers previously believed models crammed unrelated concepts into single neurons due to insufficient capacity. Cammarata demonstrated that a famous cat-and-car detector actually fired on pointed geometry, disproving the classic example of superposition.
- Cammarata attributes past assumptions about unreadability to flawed visualization tools, branching concepts, and outdated model designs. Retraining Google's 2014 Inception v1 architecture with modern settings increases detector readability from 5% to 70%.
- Contrary to fears that scaling creates opacity, Cammarata discovered larger models are easier to read. Testing across models scaled by 300 times showed broad, fuzzy category detectors refining into specific, isolated sub-feature detectors.
- Cammarata argues that model deception must remain legible to downstream internal circuitry to function. Consequently, human oversight can detect hidden deceptive plans if researchers learn to interpret the model's native internal wiring.
- Industry reactions to Cammarata's claims split sharply between optimism and skepticism. OpenAI researchers compared the paper to Attention Is All You Need, while AI safety researcher Steven Casper dismissed the initial framing as hype.
Also discussed on this episode: (2)
Safety (2)
- Interpretability researcher Liv Boote pushed back on Cammarata's thesis, arguing networks still store concepts in overlapping dimensions. Google DeepMind interpretability lead Neel Nanda noted language models hold far more memorized facts than available individual neurons.
- Seeing Clearly host Christian argues doomers and accelerationists form a self-correcting market feedback loop. Growing threat perceptions drive capital and technical talent into alignment research, compelling structural safety adjustments as model risks escalate.