Working out
Notes
The entries state what a thing is. These state how I got there — the derivation taken slowly, the wrong turning that made the right one make sense, the question I could not answer and left standing.
4 notes.
Attention from scratch
4 partsbuilding the operation up from the problem it was invented to solve, one step at a time
01Why attention exists
The fixed-size bottleneck that broke sequence-to-sequence models, and the obvious fix nobody could take.
Building the attention weights
From "which positions matter" to softmax over scaled dot products, by elimination.
Reading an attention matrix
What the n×n matrix contains, what the pretty heatmaps show, and how much of it is an artefact.
The quadratic problem
Where the n² really comes from, and why the escape routes divide into exactly three kinds.