How AI Benchmarks Became Broken (And What's Replacing Them)

Logosofia no Dia a Dia · Apr 16, 2026 · 25m

AI labs announce breakthrough scores on benchmarks like MMLU and HellaSwag constantly — but how much do these tests actually tell us about real AI capabilities? This episode digs into the messy reality of AI evaluation: how benchmarks get contaminated by training data, why they saturate within years, what models are really learning when they ace them, and what newer approaches like SWE-bench and LMSYS Chatbot Arena are trying differently. It's a story about the gap between how we measure progress and what progress actually looks like. Episode 2239 — myweirdprompts.com/n/2239 (permanent short…