Fault Tolerance

A GPU has a mean time between failures measured in years. Sixteen thousand of them, plus the network, the storage and the power, gives a system MTBF measured in hours.

Llama 3’s 405B run is the best-documented case: 466 interruptions over 54 days, 419 of them unplanned, and about 78% traced to hardware — GPUs and HBM predominantly. Roughly one unplanned stop every three hours, sustained for two months.

Since every collective is a barrier, one dead rank stops the entire job. There is no partial progress.

The checkpoint interval

Differentiating gives the classical result, Young’s formula:

At a 3-hour MTBF and a 5-minute checkpoint, $T^* \approx 42$ minutes. The quantity that matters most is $C$, and it appears under a square root — so halving the checkpoint cost only improves the interval by 30%, but it also reduces the fixed overhead directly, which is where the real gain is.

Making the checkpoint cheap

A 405B model’s optimiser state is several terabytes. Written synchronously to shared storage, that is many minutes with every device idle.

Shard it. Under ZeRO each rank already holds a slice; each writes its own, in parallel, and the aggregate bandwidth is the cluster’s rather than one node’s.

Write asynchronously. Copy the state to host memory — fast, and the GPUs resume immediately — then flush to storage from a background thread. This turns minutes of idle time into seconds.

Keep it in memory. Some systems hold a checkpoint in a peer node’s RAM as well as on disk, so recovery from a single-node failure never touches storage.

What must be saved

Only the first two crash anything if wrong. The rest degrade the run quietly, which is why they are the ones that get missed.

import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict


def save(model, optimizer, loader, step: int, path: str) -> None:
    model_sd, optim_sd = get_state_dict(model, optimizer)
    state = {
        "model": model_sd,
        "optim": optim_sd,
        "step": step,
        "loader": loader.state_dict(),          # position in the stream
        "rng": torch.cuda.get_rng_state_all(),
    }
    # Sharded and parallel: every rank writes its own slice at once.
    dcp.save(state, checkpoint_id=f"{path}/step-{step}", async_save=True)


def resume(model, optimizer, loader, path: str) -> int:
    model_sd, optim_sd = get_state_dict(model, optimizer)
    state = {"model": model_sd, "optim": optim_sd, "step": 0,
             "loader": loader.state_dict(), "rng": None}
    dcp.load(state, checkpoint_id=path)

    torch.cuda.set_rng_state_all(state["rng"])
    loader.load_state_dict(state["loader"])
    return state["step"]

Detection. Most of the time saved is in noticing quickly. A hung collective produces no error — the job sits at a barrier indefinitely — so a watchdog on collective duration, and on per-rank step time, is what turns a silent hang into a fast restart.

Elastic training. torchrun --nnodes=min:max lets the job continue on fewer nodes after a failure and re-expand when they return. It changes the data-parallel degree and therefore the effective batch, so the learning rate should follow — usually ignored, and a small source of irreproducibility.

Bad batches. Not every interruption is hardware. PaLM’s authors report loss spikes that were not reproducible from the same checkpoint on different data, and their mitigation was to restart a few hundred steps back and skip the batches around the spike — a restart protocol for a problem that is not a failure at all.