Are Political Bias Benchmarks Actually Measuring Anything?

Logosofia no Dia a Dia · Apr 25, 2026 · 26m

The Political Compass Test is the go-to tool for measuring political bias in large language models — but a growing body of research suggests it's fundamentally broken. This episode unpacks why the PCT can mask bias rather than reveal it, then walks through four better alternatives: IssueBench's open-ended prompt approach, Stanford's perception-based user study, OpenAI's granular five-axis internal evaluation, and UT Austin's moral foundations framework. Along the way, we explore why measuring bias requires a reference point — and why picking what counts as "neutral" is itself a political act…