Why Million-Token Context Windows Can't Handle 3 Reasoning Steps

Logosofia no Dia a Dia · Apr 25, 2026 · 28m

The needle-in-a-haystack benchmark is saturated — every frontier model hits 99% on it. But that doesn't mean they can actually reason across long documents. This episode explores four benchmarks that replaced it: RULER, BABILong, NoCha, and Michelangelo. We break down why models that ace million-token retrieval tests collapse at 11,000 tokens on multi-hop reasoning, and what this gap between claimed and effective context windows means for anyone relying on AI for long-document analysis. Episode 2406 — myweirdprompts.com/n/2406 (permanent short link: myweirdprompts.com/243292)