SwiGLU

The original transformer’s feed-forward block is two matrices with a pointwise nonlinearity between them. A gated linear unit splits the first projection in two and multiplies one branch, passed through an activation, into the other.

At $\beta = 1$ the activation is SiLU, which is what every released implementation uses; $\beta$ is almost never learned.

Paying for the third matrix

A gated block has three weight matrices where the classic block has two, so holding the parameter count fixed against a $4d$ classic block means shrinking the hidden width by a factor of $\tfrac{2}{3}$.

In practice $d_{\text{ff}}'$ is rounded to a multiple of 128 or 256 so the matmul tiles evenly; the ratio you see in a config file is rarely exactly $8/3$ for that reason alone.

Shazeer’s own verdict on why gating helps is worth quoting for its honesty: the variants are offered “divine benevolence” as the explanation. The empirical result has held up across a decade of models regardless.

import torch.nn.functional as F
from torch import Tensor, nn


class SwiGLU(nn.Module):
    def __init__(self, d_model: int, multiple_of: int = 256):
        super().__init__()
        hidden = int(2 / 3 * 4 * d_model)
        # round up so the matmul tiles evenly
        hidden = multiple_of * ((hidden + multiple_of - 1) // multiple_of)

        self.gate = nn.Linear(d_model, hidden, bias=False)   # W1
        self.up = nn.Linear(d_model, hidden, bias=False)     # W3
        self.down = nn.Linear(hidden, d_model, bias=False)   # W2

    def forward(self, x: Tensor) -> Tensor:
        return self.down(F.silu(self.gate(x)) * self.up(x))

Fusing gate and up into one nn.Linear(d_model, 2 * hidden) and splitting the result is worth a few percent of step time, at the cost of a checkpoint layout that no longer matches anyone else’s. Biases are omitted throughout, as they are in every modern decoder.