Feed-forward

Two thirds of a transformer’s parameters sit in a block that looks at one position at a time. It is where most of the model’s stored knowledge appears to live, and where gating turned out to be worth the extra matrix.