
How Amazon Turns Real Failures Into Better AI Models
How does Amazon build its agentic AI? Michael Giannangeli, Head of Product for Amazon Nova and Agentic AI, breaks down evals, RL gyms, and model routing. He also explains why the bottleneck in software has shifted away from engineering hours and what takes its place. In this video, we cover: The eval lifecycle: building from real failure modes, saturation, and why 100% means delete RL gyms: training models on real environments like migrations, DevOps, and pen testing Model routing, cost-per-token trade-offs, and why routing isn't solved The agent stack of an Amazon product lead: Claude Code, Codex, and Kiro Autonomous migrations, trust, and how much human-in-the-loop survives For engineers and product people building with AI agents who want to see how a frontier lab actually closes its feedback loops. Recorded at the AI4 conference 2026. Timestamps: 00:00:00 - Intro 00:00:36 - The Agents an Amazon Product Lead Uses Daily 00:03:36 - Why Nobody's Heard of Amazon Nova 00:04:55 - Model Costs and the Routing Problem 00:08:10 - Why Building Good Evals Is So Hard 00:10:05 - When Evals Saturate and Get Deleted 00:12:17 - Turning Real Failure Modes Into Hundreds of Evals 00:15:26 - Improving Models Without Training on Customer Data 00:18:26 - If Everyone Uses Agents, You Need Agents 00:20:22 - The Bottleneck Is No Longer Engineering Hours 00:23:20 - Ship Fast to Validate the Right Thing 00:26:44 - Staying at the Frontier Amid Constant Noise 00:29:37 - Spend 10-20% of Your Time Experimenting 00:32:54 - RL Gyms: How Models Learn From Failure 00:37:09 - Will Migrations Become Fully Autonomous? Guest: Michael Giannangeli - Head of Product, Agentic AI & Amazon Nova at Amazon #AmazonNova #AgenticAI #AIEngineering












