Latest / Logosofia no Dia a Dia
How Transformers Learn Word Order: From Sine Waves to RoPE
Why do transformers need special tricks to understand word order? This episode dives into the math behind positional encoding—from the original sine waves to learned embeddings, ALiBi, and the modern RoPE standard. Learn how these methods enable massive context windows and why RoPE is now the go-to choice for models like Llama and GPT-4. Episode 2062 — myweirdprompts.com/n/2062 (permanent short link: myweirdprompts.com/816596)
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.