The Attention Budget in Your Pocket

Logosofia no Dia a Dia · Sep 21, 2026 · 21m

Daniel wrote in with a sharp question: if punctuation restoration depends on surrounding context, why does dictating on a non-flagship phone feel like managing an invisible budget? This episode unpacks the attention tradeoff behind speech recognition — why streaming models punctuate badly, why full-batch transcription is too expensive on a phone, and how chunked inference with a bounded attention window sits in the middle. We look at Nvidia's NeMo toolkit and the att_context_size parameter, which lets you tune left and right context at inference time, and contrast it with Whisper's fixed…