The Frontier

Zico Kolter argues modern AI is conceptually simple, with core LLM training and RL code achievable in roughly 200-300 lines of Python.

Kolter chairs OpenAI's Safety and Security Committee, an oversight board that can delay model releases if safety evaluations are insufficient.

He says model safety does not automatically improve with scale, unlike capabilities. Making models robust requires explicit safety training and additional monitoring layers.

Kolter co-authored the 2023 GCG paper, which automated jailbreak generation and discovered universal, transferable attacks that worked across different models.

He categorizes AI risk into four areas: model mistakes, harmful use, societal/psychological effects, and loss-of-control scenarios.

Modern AI security is a multi-layered Swiss-cheese defense combining input/output classifiers, safety training, operational monitoring, and sandboxing for agents.

Kolter states AI agents introduce prompt injection risks by processing third-party data, requiring careful control over their permissions and access.

He believes reinforcement learning is the foundation of modern post-training, where models are trained on their own synthetic outputs selected by a reward signal.

Kolter is skeptical that transformer architecture was essential, arguing other sequence models would have scaled to similar capabilities given enough compute and data.

His startup, Gradient, provides third-party AI safety tools including automated red-teaming systems and custom safety models for enterprises.

He co-founded Gradient in 2023 after running a large agent red-teaming competition with 1.8 million attack attempts.

Kolter argues the key scientific discovery was that scaling simple architectures on vast text data produces coherent intelligence, not the specific engineering.