Latest / Logosofia no Dia a Dia
Why Run One AI When You Can Run Two?
Inference latency is the biggest bottleneck for deploying large language models. This episode explores speculative decoding, a clever technique that uses a small draft model to predict tokens ahead of time, which a larger model then verifies in a single pass. Learn how methods like Medusa, EAGLE, and Mamba hybrids achieve 2-6x speedups without sacrificing quality, and why this matters for real-time AI applications and GPU economics. Episode 2065 — myweirdprompts.com/n/2065 (permanent short link: myweirdprompts.com/954868)
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.