Colophon
How this thing is written
Every entry starts as a note taken while reading a paper or debugging something that didn't work. It gets promoted to an entry once the maths has been rederived by hand and the implementation runs on real tensors.
Entries that fail that bar stay stubs. There are 6 of them.
Conventions
NotationRow-vector convention throughout; batch and head dimensions elided unless they matter to the argument.
ComplexityStated for a single forward pass at sequence length n and model width d, ignoring constant factors.
CodePyTorch, minimal and unfused, written to be read rather than shipped. Fused variants are linked, not inlined.
StubsMarked inline. A stub means the maths is not yet rederived or the implementation has not been run.
CitationsPrimary sources only. Blog posts are cited when they are the primary source.
Stack
Literata for prose and equations · Space Mono for labels and code
Astro content collections in MDX · KaTeX for the maths · Shiki for the code
Tailwind for the tokens, TypeScript throughout, statically built
Notation follows Elhage et al. where the two overlap
Source and revision history kept in a private repository
Corrections
If an entry is wrong, it is wrong in a specific place. Send the entry path, the line, and what it should say.
corrections@grimoire →