Latest / Logosofia no Dia a Dia
AI Grading AI: The Snake Eating Its Tail
The industry is scaling faster than humans can review, so we’ve turned to LLM-as-a-Judge to grade model outputs. But this creates a hall of mirrors: AI grading AI, often with a preference for verbosity and its own style. We explore the mechanics of single-point, pairwise, and reference-based scoring, and the hidden biases—like position and self-enhancement—that threaten to create a monoculture of identical models. Is this the future of evaluation, or a trap we can’t escape? Episode 2007 — myweirdprompts.com/n/2007 (permanent short link: myweirdprompts.com/508807)
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.