Building a Scorecard for Our Own AI Writers
Every episode of this show is written by an agentic pipeline, and for most of the archive that script-writing agent has been DeepSeek V4, with Gemini, Claude, and a stretch of OpenRouter randomization in the mix. So when Daniel asked how to build a bespoke benchmark for it, the answer couldn't be vibes and one careful listen. This episode works through the three things he wants measured — mid-episode repetition, lifelike dialogue between two named recurring hosts, and whether the retrieved sources actually make it into the script — and why no existing benchmark covers that combination…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.