AI weight editing may fix bugs, and weaken safeguards
Nov 9, 2026Monday, November 9, 2026Summary from 2 contributing podcasts.
- GPT 6 Astra improved a smaller model by directly editing a tiny fraction of its weights.
- The technique could help diagnose models, but it may also weaken safeguards and remains far from frontier-scale self-improvement.
MIT researchers demonstrated that one AI model can directly change another model’s weights, offering a possible way to fix specific problems without retraining. In the study “Sparks of Weight Space Programming,” GPT 6 Astra edited Qwen 3.5 0.8B.
Astra changed just 0.016% of the model’s weights, using 50 forward passes on practice problems. It found and removed a reasoning loop, raising the model’s reported math score from 7.9% to 48.1% and tripling its instruction-following score. The loop was already documented in Qwen’s model card.
That result does not show a general leap in mathematical ability. Wei Aizhou argued that the score increase came from fixing the loop, not from teaching the model to reason more broadly.
The researchers also reported that frontier models could infer a network’s purpose from its weights and write small game-playing networks without feedback from an environment. Google researcher Alex Krensel replicated the results but stressed the scale gap: these small networks are far from frontier models.
On Moonshots with Peter Diamandis, the discussion pointed to both uses and risks: weight editing might help researchers understand models or upgrade specific capabilities, but it could also remove safeguards from open models. That danger was raised as a possibility, not demonstrated in the experiment.
Nathaniel Whittemore, host of The AI Daily Brief, described the work as a possible mechanism for recursive self-improvement, while noting that models still cannot edit themselves at runtime. The study shows a targeted repair is possible; it does not show AI can improve frontier systems on its own.
Source Intelligence
- Deep dive into what was said in the episodes
Big Tech’s $3T AI Bet, Claude Writes 90% of Anthropic’s Code, AIs Editing AI Brains | EP #308 • Nov 4
- The MIT paper “Sparks of Weight Space Programming” showed a model editing another model's weights without retraining or backpropagation. Alex sees a path to interpretability and modular capability upgrades, while noting that weight editing could also remove safeguards from open models.
Also discussed on this episode: (19)
Elections (2)
- The panel argues AI agents could shape how people vote, with one speaker predicting most voters will consult agents by 2028. Local elections also matter because cities and states decide where data centers and autonomous vehicles operate.
- A speaker cited a 2015 Yale study arguing that public opinion rarely translates into US legislation. The panel also discussed shifting power toward local jurisdictions and special economic zones as a way to contain policy mistakes and let communities choose different rules.
Big Tech (1)
- Apple reported revenue of $114.5 billion, yet its stock fell 3% and its market value closed at $4.94 trillion. John Ternus said higher memory prices were squeezing margins, while Alex argued Apple should use its hardware and software to compete in AI.
AI Infrastructure (3)
- The panel floated thin-client phones and laptops that stream computing from data centers, with Starlink providing persistent connectivity. Alex argued consumer RAM could become a substitute for data-center capacity, but unreliable coverage and latency still limit the shift.
- Alphabet, Meta, Microsoft, Amazon, and Oracle disclosed more than $3 trillion in AI infrastructure commitments, including leases lasting 30 years. Google accounts for over $1 trillion and Meta for over $800 billion; the panel compared the total with roughly 10 Apollo programs.
- Lenders are demanding tougher terms as they question data-center delays and AI returns, while HSBC is exploring insurance for its data-center loans. OpenAI reportedly cut its funding round, and speakers pointed to insurance capital as a potential source of infrastructure financing.
Chips (1)
- Chip demand remains strong because AI use cases keep expanding, and one speaker said three-year-old H100s were rising in value. Salem estimated major new computing approaches could take at least three to five years to reach deployment, leaving a possible bubble hard to time.
Models (1)
- SemiAnalysis's GDPX benchmark tests agents on realistic expert work, including tasks lasting up to eight hours. Its 206 tasks span 11 fields; the leading models scored 12%, 10.9%, and 9.6%, with near-zero results on many finance and legal tasks.
Agents (1)
- OpenAI's LifeVal tests whether models can handle ordinary tasks such as finding childcare, budgeting, and ordering groceries through a browser. Peter called this practical assistance broadly useful; Alex questioned the benchmark's value because it measures OpenAI models on OpenAI's own test.
Coding (1)
- Peter cited Anthropic's filing that Claude usage inside the company grew 250-fold and Claude wrote over 90% of September 2026 code contributions. The filing also reported 45 times more daily code contributions per engineer than two years earlier.
Enterprise (1)
- Salem recommends companies below 50 employees rebuild aggressively, while larger firms move workflows into an AI-enabled system one at a time. The panel argued that leaders should track how quickly their systems improve, rather than only measuring current output.
Safety (1)
- A cyberattack remotely switched European supermarket refrigerators into defrost mode, spoiling food across eight countries; a similar wave hit over 500 French stores three weeks earlier. The panel warned that internet-connected industrial controls often retain outdated security.
Longevity (1)
- Peter said David Sinclair's gene-therapy approach uses three Yamanaka factors and could cost $500,000 to $2 million per treatment. Sinclair's AI-discovered SL100 molecules were estimated at $200 monthly, while an XPRIZE competition targets affordable healthspan gains.
Physics (1)
- Northwestern researchers held a positron for a month in a magnetic and electric trap near absolute zero. Alex said the work advances antimatter containment and precision tests of CPT symmetry, although spacecraft propulsion requires storing much larger quantities.
Space (1)
- NASA's Artemis plan calls for a 2027 crewed Earth-orbit mission to test docking with Blue Origin and SpaceX landers, followed by a 2028 lunar landing. The agency also plans a south-pole moon base, while Jared Isaacman framed China as the competing power.
Labor (1)
- Salem argued that employment has been mistaken for purpose: many workers already find little meaning in their jobs, and AI could free time for family, community, and science. The panel proposed local problem-solving bounties as a way to give people useful goals.
History (1)
- Alex described abundance as a possible new renaissance, driven by renewed investment in physical technologies after decades of slower progress. The panel linked past innovation to war and great-power competition, while warning that conflict can divert capital and attention from civilian advances.
Psychology (1)
- Dave argued empathy is valuable but media exposure can miscalibrate public decisions by making rare disasters feel common. Peter said news outlets emphasize fear because attention supports advertising, while Dave contrasted emotionally driven reactions with China's use of statistical analysis.
Robotics (1)
- Alex expects humanoid-robot supply to outpace current demand, requiring new uses such as building and maintaining data centers. Salem argued data centers can benefit nearby communities through access to compute, while Peter pointed to rising demand for electricians, welders, plumbers, and HVAC specialists.
The AI Daily Brief: Artificial Intelligence News and Analysis
Nathaniel Whittemore
AI Lurches Toward Recursive Self-Improvement • Nov 3
- An MIT study found GPT 6 Astra could improve Qwen 3.5 0.8B by editing its weights directly, without gradients or training. Astra identified reasoning loops and removed them, sharply raising math and instruction-following scores.
- The MIT researchers also reported that frontier models could infer a network’s purpose from its weights and write small game-playing networks without environment feedback. Alex Krensel replicated the results but stressed that edited models remain tiny compared with frontier systems.
- Wei Aizhou argued that the sixfold math gain came from fixing a looping bug, not broad mathematical improvement. The host says the result still suggests a possible mechanism for recursive self-improvement, while runtime self-editing remains beyond current models.
Also discussed on this episode: (13)
Models (1)
- Sundar Pichai said AI is already increasing Alphabet’s training and launch velocity, and expects that acceleration to continue over the next 12 months as models help improve AI development.
Agents (6)
- Elon Musk said GrokBot’s serving costs were unsustainable relative to revenue after users reported new usage limits. The host says the post could reflect product economics or market maneuvering, but the evidence does not establish which.
- GrokBot users objected that limits had changed without warning, while Musk’s earlier claims of rapid growth and revenue-generating demand clashed with his cost warning. Dots, Duet, and Hermes promoted themselves as alternatives for users considering a switch.
- Dax from OpenCode argues personal agents may be hard to monetize directly: consumers may not pay enough, leaving agents as top-of-funnel products whose business model remains unclear. The subsidy debate predates GrokBot and resurfaced during the 2025 AI bubble debate.
- Olivia Moore argues Meta Muse’s strongest opportunity may be small businesses: Meta can distribute agents through its apps, connect business tools, and turn existing advertising relationships into a broader role in SMB operations.
- SemiAnalysis’s GDPX benchmark ranks Opus 5.5 highest on economically valuable tasks, but Gemini 4 Carbon costs far less per task for a small performance gap. The results make model efficiency central to whether personal agents can become affordable.
- OpenAI’s LifeVal measures agents on everyday tasks such as finding childcare, budgeting, and ordering groceries. OpenAI reports improving success from GPT 5.2 to GPT 6 Astra and is targeting practical tasks relevant to free users.
Markets (3)
- Hedgy Markets says AI plans priced below token costs depend on investor subsidies; one analysis estimated a month of ChatGPT Pro use would cost about $14,000 at regular API prices. Nick Huber argued that subsidy withdrawal could raise prices, reduce usage, and crush valuations.
- The host expects API pricing to sort users into those satisfied by cheaper, less capable plans and power users willing to pay for valuable work. The host says API spending can create net-new value when it avoids human labor’s cost and management time.
- Anthropic’s chart showed AI leading 26% of research, drawing investor attention to RSI ahead of the company’s potential IPO. Jack Clark called RSI potentially the world’s most important technical trend; the chart was first published in September using August figures.
Safety (2)
- Anthropic’s updated S1A says the company may soon reach its RSI danger threshold and does not expect to safely build successive frontier generations with Claude acting fully autonomously. Anthropic is considering policy changes to preserve meaningful human oversight.
- The host says AI debate remains focused on whether RSI has arrived, while safety advocates are the main group describing what an RSI world might mean and some argue it should be illegal. The host calls for more attention to consequences than to defining the acronym.
Enterprise (1)
- The host argues enterprise adoption depends on more than model capability: companies still use a small fraction of available AI because corporate systems move slowly and adoption is difficult. That gap supports both a bearish case about weak demand and a bullish case about future growth.