Heads vs Layers: How Model Merging Actually Works
When someone asks how open source enthusiasts combine two AI models into a hybrid, the intuitive answer — snap the attention heads together — turns out to be wrong in an interesting way. Heads are a granularity of control; layers are the unit of assembly. This episode unpacks what a transformer head actually is, from the query-key-value subspaces inside multi-head attention to the strange finding that some heads are actively interfering with the model's job, and then follows that thread into the weight-space arithmetic behind tools like mergekit. Along the way: task vectors, TIES trimming…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.