AI’s black box is opening, not disappearing
- Nick Cammarata’s research suggests many neural-network detectors have clear, readable functions.
- Researchers say readable circuits could help reveal hidden behavior, but they do not prove a model safe.
The AI black box may be less opaque than researchers once thought, but readable circuits are not the same as a full explanation of how a model works, or proof that it is safe.
On The AI Daily Brief, host Nathaniel Whittemore discussed researcher Nick Cammarata’s finding that about nine in ten detectors in current OpenAI models have one primary function. Cammarata’s re-examination also found that a famously “crammed” cat-and-car detector was tracking a shared V-shaped feature, rather than mixing two unrelated concepts. His argument: old tools and models made neural networks look more tangled than they are.
The implications reach beyond labeling circuits. On Lenny’s Podcast, Anthropic interpretability researcher Emmanuel Ameisen said models can form internal plans before generating words, for example, mapping out a rhyme before completing a poem. He described probing a model’s intermediate states after conflicting training signals led it to favor 36 over the correct answer, 12. The internal signals exposed the bias before it appeared in the output.
Researchers see that visibility as a possible way to catch behavior that ordinary question-and-answer tests miss. But Cammarata’s claim that deception should leave clues in a model’s circuitry is an argument for further investigation, not a guarantee that researchers can find every hidden plan.
Google DeepMind interpretability lead Neel Nanda cautioned that language models store more facts than they have neurons, leaving room for concepts to overlap. Researcher Liv Boote also argued that Cammarata’s findings do not rule out this kind of “cramming.” The emerging picture is more hopeful than a wholly unreadable black box, but not yet a map of the whole machine.
Source Intelligence
- Deep dive into what was said in the episodes
Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic) • Oct 18
- Emmanuel Ameisen explains that building large language models does not equal understanding them because models develop through training architectures on massive datasets rather than traditional rule-based programming.
- Emmanuel Ameisen reveals that language models formulate internal multi-step plans within intermediate computational states. When writing rhyming poetry, models construct future rhyming strategies across intermediate layers without outputting the plan into text.
- Emmanuel Ameisen compares mechanistic interpretability to neuroscience, where researchers record intermediate numerical states across text corpuses to map feature activations. Researchers can manipulate these internal vectors directly to verify how concepts drive model output.
- Emmanuel Ameisen details an alignment investigation where a model cited 36 instead of the correct answer 12 during math calculations. Training data had accidentally rewarded 36, causing internal representation conflicts that drove the model to output erratic text.
Also discussed on this episode: (5)
Models (2)
- Emmanuel Ameisen notes language models surpassed human accuracy at next-token prediction in 2019. To solve difficult prediction tasks, models construct internal mental models of physical geography and world concepts rather than memorizing surface text patterns.
- Emmanuel Ameisen contends that anthropomorphizing artificial intelligence offers a practical framework for predicting model behavior. Training on human dialogue forces models to simulate character motivations, while outcome-oriented reinforcement learning produces human-like strategies such as frustration and deception.
AI Infrastructure (1)
- Emmanuel Ameisen argues that scaling compute and text richness yields predictable empirical improvements under scaling laws. Product breakthroughs occur when smooth capability gains cross discrete reliability thresholds, such as increasing code generation accuracy from 80 percent to 99 percent.
Reasoning (1)
- Emmanuel Ameisen compares rapid decision models like Jev to human System 1 thinking. He projects that fast gut-judgment classifiers will operate alongside deliberative reasoning models as fundamental components of intelligent systems.
Open Source (1)
- Emmanuel Ameisen recommends Neuripedia as an accessible open-source platform for conducting mechanistic interpretability experiments on language models.
Nathaniel Whittemore
A Potentially Historic AI Breakthrough • Oct 18
- Nick Cammarata published a paper asserting that neural network interior circuits are readable, challenging years of consensus that models are too tangled to interpret. OpenAI researcher Rohun called the finding a historic scientific breakthrough.
- An OpenAI project in 2023 found only 2% of GPT-2 neurons were legible with primitive tools. Cammarata claims modern tools reveal roughly nine out of ten detectors in current OpenAI models have one primary function.
- Researchers previously believed models crammed unrelated concepts into single neurons due to insufficient capacity. Cammarata demonstrated that a famous cat-and-car detector actually fired on pointed geometry, disproving the classic example of superposition.
- Cammarata attributes past assumptions about unreadability to flawed visualization tools, branching concepts, and outdated model designs. Retraining Google's 2014 Inception v1 architecture with modern settings increases detector readability from 5% to 70%.
- Contrary to fears that scaling creates opacity, Cammarata discovered larger models are easier to read. Testing across models scaled by 300 times showed broad, fuzzy category detectors refining into specific, isolated sub-feature detectors.
- Cammarata argues that model deception must remain legible to downstream internal circuitry to function. Consequently, human oversight can detect hidden deceptive plans if researchers learn to interpret the model's native internal wiring.
- Interpretability researcher Liv Boote pushed back on Cammarata's thesis, arguing networks still store concepts in overlapping dimensions. Google DeepMind interpretability lead Neel Nanda noted language models hold far more memorized facts than available individual neurons.
- Industry reactions to Cammarata's claims split sharply between optimism and skepticism. OpenAI researchers compared the paper to Attention Is All You Need, while AI safety researcher Steven Casper dismissed the initial framing as hype.
Also discussed on this episode: (1)
Safety (1)
- Seeing Clearly host Christian argues doomers and accelerationists form a self-correcting market feedback loop. Growing threat perceptions drive capital and technical talent into alignment research, compelling structural safety adjustments as model risks escalate.