Latest / Logosofia no Dia a Dia
LLM Benchmarks Are Full of Noise: Statistical Rigor in AI Evals
Almost every model release blog post you've read has a statistical problem. When OpenAI, Anthropic, or Google claims their new model beats the previous one by two points on MMLU, that difference is often well within the noise floor. This episode gets into the weeds on power analysis, McNemar's test for paired evaluations, bootstrapped confidence intervals, and why the decimal-place precision in benchmark tables is a tell that something's wrong. We also break down the math behind Chatbot Arena's Elo ratings and explain why the rankings people obsess over may be essentially meaningless. If you…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.