Architectures / ffn
Feed-forward
Two thirds of a transformer’s parameters sit in a block that looks at one position at a time. It is where most of the model’s stored knowledge appears to live, and where gating turned out to be worth the extra matrix.
8 entries.
Entries
01.04.1O(n·d·s)01.04.2—01.04.3O(n·d²)01.04.4O(n·d)01.04.5—01.04.6O(√N·d)01.04.7O(n·d)01.04.8O(n·d²)
Activation Sparsitypromising
Most FFN neurons are zero for any given token; skip the rows they own.
Feed-Forward as Key-Value Memorycommon
Read the FFN as a lookup table: W₁ rows are patterns, W₂ columns are what gets written.
Feed-Forward Networksstandard
Two matrices and a nonlinearity, applied to each position alone.
GELUcommon
A smooth gate on the identity, and the activation lineage it belongs to.
Parallel Attention and FFNcommon
Run both branches off the same input instead of one after the other.
Product-Key Memorypromising
A million-entry lookup layer searched in √N time by factorising the key space.
Squared ReLUpromising
ReLU, then square: cheaper than a gate and far sparser than either.
SwiGLUstandard
Gated feed-forward block; a Swish gate multiplied into a linear branch.