Architectures / sparse
Sparse blocks
If only a fraction of the network runs for any given token, parameter count and compute cost decouple. The difficulty is not the idea but the routing — deciding which fraction, keeping the decision differentiable, and stopping every token from choosing the same expert.
8 entries.
Entries
01.05.1O(k·d²)01.05.2O(k·d²)01.05.3O(k·d²)01.05.4O(E)01.05.5O(c·n²·d)01.05.6O(k·d²)01.05.7O(k·d²/m)01.05.8O(k·d²)
Expert Capacity and Token Droppingcommon
Fixed buffers per expert, and what happens to the tokens that do not fit.
Expert Parallelismstandard
Experts on separate devices, and the two all-to-alls that connect them.
Expert-Choice Routingpromising
Invert the assignment: each expert picks its top tokens, not the reverse.
Loss-Free Load Balancingcommon
Balance experts with a per-expert bias on the router, not with a gradient.
Mixture-of-Depthspromising
Route tokens past whole blocks: conditional depth rather than conditional width.
Mixture-of-Experts Routingstandard
Top-k gate over expert FFNs with auxiliary load-balancing loss.
Shared and Fine-Grained Expertsstandard
Many small experts, plus a few that every token uses.
Sparse Upcyclingcommon
Turn a finished dense checkpoint into a mixture of experts by copying the FFN.