Floating-Point Formats

A floating-point number is a sign, an exponent and a mantissa. The exponent sets how far apart the largest and smallest representable magnitudes are; the mantissa sets how finely spaced the values are within that span. A format is a decision about how to split a fixed bit budget between the two, and every such decision is a bet on which kind of error the workload can tolerate.

Why bf16 displaced fp16

fp16 came first and has more precision. It also has a dynamic range that gradient magnitudes routinely leave. A gradient of $10^{-8}$ is not small for a deep network; in fp16 it is zero, and the update it should have produced never happens. At the other end, an activation spike above 65 504 becomes infinity, and one infinity propagates to a NaN loss within a step.

The mitigation was loss scaling: multiply the loss by a large constant before the backward pass so that gradients land in fp16’s representable band, then divide it back out before the optimiser step. Dynamic loss scaling automates the constant by doubling it periodically and halving it whenever the gradients overflow — which means a training run that intermittently discards steps as a matter of course.

The precision loss is not free everywhere, and the exceptions are the reductions: sums over the feature dimension in normalisation, the softmax denominator, the optimiser’s second-moment accumulator. All of those are kept in fp32 in every serious implementation, which is what “mixed precision” actually means — the weights and activations are half precision, the accumulations are not.

fp8, and why it needs a scale

Four exponent bits reach 448. Activations in a large model exceed that routinely, so fp8 is not a format one simply casts into — every tensor carries a scale factor, and the value stored is $x / s$ with $s$ chosen so the tensor’s maximum lands near the top of the range.

The two variants split the work. e4m3 has more mantissa and less range, and holds weights and activations. e5m2 has more range and less mantissa, and holds gradients, which span more orders of magnitude and tolerate coarser steps. Getting this backwards is a real and common mistake.

Because $\max|x|$ is not known until the tensor exists, production implementations use delayed scaling: keep a short history of recent maxima, predict the next one, and correct if the prediction was too low. Recomputing the scale from the current tensor is exact and costs a full extra pass over it.

import torch
from torch import Tensor

FP8_E4M3_MAX = 448.0


def to_fp8(x: Tensor, amax: Tensor | None = None) -> tuple[Tensor, Tensor]:
    """Scale into range, then cast. Returns the tensor and its scale."""
    amax = x.abs().amax() if amax is None else amax
    # clamp: an all-zero tensor would otherwise divide by zero.
    scale = (amax / FP8_E4M3_MAX).clamp(min=1e-12)
    return (x / scale).to(torch.float8_e4m3fn), scale


def from_fp8(x: Tensor, scale: Tensor, dtype=torch.bfloat16) -> Tensor:
    return x.to(dtype) * scale

The matmul is where the win is. An fp8 tensor-core matmul runs at roughly twice the throughput of bf16 on Hopper and later, and — more importantly for serving — halves the bytes moved, which is the actual bound. The accumulator inside the kernel is still fp32; fp8 describes the operands, never the sum.

fp4 (e2m1) has six representable magnitudes. Per-tensor scaling is hopeless at that width, so the block moves to micro-scaling: a shared exponent per group of 16 or 32 elements, so the scale tracks local structure rather than the whole tensor. Blackwell implements this in hardware as MXFP4 and NVFP4.

Whether four-bit training works is genuinely open. Four-bit inference is settled — GPTQ and its successors are in production — and the distinction is that inference quantises a finished weight once, against a calibration set, while training has to keep gradients meaningful through a format with six magnitudes in it.