Council of Models: How Karpathy Built AI Peer Review

Logosofia no Dia a Dia · Apr 12, 2026 · 27m

In November, Andrej Karpathy released llm-council, a deceptively simple system that treats language models like an academic council: four frontier models answer questions independently, then anonymously rank each other's responses, and a Chairman model synthesizes the results. The architecture packs deliberate design choices into just 800 lines of code—including a clever anonymization scheme, graceful error handling, and a multi-stage protocol that mirrors human expert panels. But does it actually achieve consensus, or just create a veneer of objectivity? This episode digs into the…