Shared and Fine-Grained Experts

Mixtral has 8 experts and routes to 2. DeepSeek-V3 has 256 and routes to 8, plus one that is always on. The second arrangement is not a scaled-up version of the first; it is a different theory of what an expert is.

Fine granularity

With 8 experts, each is a full-width feed-forward block and there are $\binom{8}{2} = 28$ possible routings. With 64 experts each a quarter of the width, the same FLOPs buy $\binom{64}{8} \approx 4.4 \times 10^9$ combinations.

FLOPs and parameters are both unchanged. What changes is the resolution of the routing decision — the model can compose a specialist combination per token instead of picking one of a handful of generalists.

The measured effect supports this. DeepSeekMoE reports that fine-grained experts show much lower activation overlap between unrelated inputs than coarse ones — that is, they genuinely specialise rather than all learning the same average function.

The shared expert

Some knowledge is needed by every token: English syntax, the shape of the residual stream, the boilerplate of being a language model. With pure routing, every expert has to learn it separately, because any expert might be the one selected.

That is a straightforward waste. The fix is to carve out one or two experts that are always active and never routed.

The shared expert absorbs the common knowledge; the routed ones are freed to hold what differs. It also has a useful side effect on stability — the layer is never a no-op for a token, so a dropped routing degrades the contribution rather than removing it.

import torch
from torch import Tensor, nn


class DeepSeekMoE(nn.Module):
    def __init__(self, dim: int, n_routed: int = 64, n_shared: int = 1, k: int = 6):
        super().__init__()
        # Fine-grained: each expert is a fraction of a normal FFN's width.
        hidden = int(dim * 8 / 3 / (n_routed / 8))

        self.shared = nn.ModuleList(FFN(dim, hidden) for _ in range(n_shared))
        self.routed = nn.ModuleList(FFN(dim, hidden) for _ in range(n_routed))
        self.gate = nn.Linear(dim, n_routed, bias=False)      # routed only
        self.k = k

    def forward(self, x: Tensor) -> Tensor:                   # [T, D]
        y = sum(expert(x) for expert in self.shared)          # never gated

        scores = self.gate(x).softmax(-1)
        weight, idx = scores.topk(self.k, dim=-1)
        weight = weight / weight.sum(-1, keepdim=True)        # renormalise the k

        for slot in range(self.k):
            for e, expert in enumerate(self.routed):
                mask = idx[:, slot] == e
                if mask.any():
                    y[mask] += weight[mask, slot, None] * expert(x[mask])
        return y

The renormalisation of the top-$k$ weights is the detail that differs between implementations. DeepSeek normalises after selection, so the routed contribution has consistent magnitude regardless of how confident the router was; Mixtral takes the softmax over the selected logits instead, which is nearly but not quite the same thing. Mismatching it against a checkpoint produces a model that is subtly and unfixably worse.

Fine granularity is not free in practice, only in FLOPs. More experts means more tokens to sort, more all-to-all messages under expert parallelism, and smaller matmuls that use the hardware less efficiently. Routing to 8 of 256 moves roughly four times the metadata of routing to 2 of 8, and the individual expert matmuls are a quarter the size — which is why implementations grouped by expert with a single batched matmul matter far more here than at coarse granularity.