Sparse Upcycling

Training a sparse model from scratch means discarding whatever dense checkpoints you already have. Since the architectures differ in exactly one block, that is an avoidable waste.

Upcycling: keep the embeddings, the attention, the norms. Replace each feed-forward block with $E$ copies of itself and a fresh router, then continue pre-training.

At step zero the model computes almost exactly what the dense model did — every expert is the same function, so which one is chosen does not matter, and with top-$k$ weights summing to one the block’s output is unchanged. The loss does not jump. Training then pulls the copies apart.

Symmetry breaking

Identical experts are a degenerate initialisation: the gradient to every copy is the same up to which tokens it received, and there is nothing to prefer one over another. What breaks it is the routing itself — the random router sends different tokens to different experts, so the copies see different data from the first step and diverge.

Some implementations add small noise to each copy instead, on the reasoning that explicit asymmetry beats waiting for the router to supply it. Skywork-MoE compared the two and found the noise unnecessary and occasionally harmful — it perturbs a well-trained block for no gain.

What it costs in quality

Komatsuzaki et al. are careful about the comparison, and the honest summary is a crossover. Upcycling beats continued dense training after roughly 10% of the original pre-training budget is spent on it. It also stays behind an MoE trained sparse from scratch, and the gap narrows with budget without closing.

Which is a statement about who you are, not about which method is better. Most organisations have a dense checkpoint and a fraction of the compute that produced it, and for them the crossover never arrives.

Skywork-MoE adds a second finding worth knowing: upcycled models show weaker expert specialisation than from-scratch ones, measurably so. The copies start in the same basin and do not travel far from it, so the diversity that makes sparsity worth having is partly foreclosed by the initialisation.

import copy

import torch
from torch import nn


def upcycle(model: nn.Module, n_experts: int = 8, k: int = 2, every: int = 2):
    """Replace every `every`-th FFN with n_experts copies of itself plus a router."""
    for i, block in enumerate(model.blocks):
        if i % every:
            continue

        dense = block.ffn
        experts = nn.ModuleList(copy.deepcopy(dense) for _ in range(n_experts))

        router = nn.Linear(model.dim, n_experts, bias=False)
        # Near-zero: the initial gate is uniform, so the layer starts out
        # computing what the dense block computed.
        nn.init.normal_(router.weight, std=1e-3)

        block.ffn = MoELayer(experts, router, k=k)
    return model

every = 2 is the common choice: alternate blocks become sparse and the rest stay dense. Converting every block multiplies the parameter count by $E$ and, in the reported ablations, buys less than converting half of them — the usual finding that sparsity is worth more in some layers than others, and that the middle of the network is where it pays.