Memory Fragmentation

CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has 79.15 GiB capacity; 61.20 GiB already allocated;
12.42 GiB free; 66.73 GiB reserved in total by PyTorch

Twelve gigabytes free and a two gigabyte allocation fails. The message is accurate and the explanation is in the gap between allocated and reserved.

The caching allocator

cudaMalloc synchronises the device and costs on the order of a hundred microseconds. A training step performs thousands of allocations, so PyTorch does not call it per tensor: it requests large segments from the driver and sub- allocates from them, returning freed blocks to its own pool rather than to CUDA.

The consequence is that free memory is free within a segment. A 2 GB request needs 2 GB contiguous inside one segment, and a pool holding thirty scattered 400 MB blocks cannot serve it however much it totals.

Under 10% is normal. Over 30% means the allocation pattern, not the model, is the problem.

The fixes, in order

expandable_segments:True. The allocator maps virtual address space and commits physical pages as needed, so a segment can grow instead of being fixed at creation. This removes most fragmentation for variable-shape workloads and is the single highest-value setting on this page.

Bucket the shapes. Pad sequence lengths to powers of two or to a small set of buckets. Fewer distinct sizes means blocks are reusable, which is the underlying cause addressed directly rather than worked around.

torch.cuda.empty_cache(). Returns unused segments to the driver. It is slow, it synchronises, and it is a symptomatic fix — useful between phases (train then evaluate) and harmful inside a training loop.

Allocate the large things first. At start-up, when the pool is empty, big allocations get clean segments. A KV cache or activation buffer claimed at initialisation and reused is not subject to any of this.

That last point is PagedAttention’s argument, applied to the KV cache specifically: fixed-size blocks with an indirection table, so a growing cache never needs contiguous space. It is an operating system’s answer to the problem — the vLLM paper measures 60–80% waste in the pre-paged allocators it replaces.

import torch


def memory_report() -> dict[str, float]:
    """The three numbers, in GB. Log them; the ratio is the diagnosis."""
    alloc = torch.cuda.memory_allocated() / 1e9
    reserved = torch.cuda.memory_reserved() / 1e9
    return {
        "allocated": alloc,
        "reserved": reserved,
        "fragmentation": (reserved - alloc) / reserved if reserved else 0.0,
        "peak": torch.cuda.max_memory_allocated() / 1e9,
    }


def capture_timeline(path: str = "mem.pickle", steps: int = 3):
    """Records every alloc and free with its stack. Open the pickle at
    pytorch.org/memory_viz — it names the tensors that actually dominate."""
    torch.cuda.memory._record_memory_history(max_entries=100_000)
    yield
    torch.cuda.memory._dump_snapshot(path)
    torch.cuda.memory._record_memory_history(enabled=None)

The snapshot viewer is the tool worth knowing about. It renders allocation lifetimes as a timeline with the allocating stack frame attached, which turns “something is using memory” into a named line of code in about a minute — as against the usual approach of bisecting the model by commenting things out.

Two things get misattributed to fragmentation.

A genuine leak. A tensor kept alive by a reference — appending loss rather than loss.item() to a list retains the entire graph. Here allocated grows monotonically, whereas fragmentation shows a stable allocated under a growing reserved.

A real shortfall. Sometimes the model does not fit and the allocator is blameless. If allocated at peak is close to capacity, the answer is on one of the other pages of this branch, not this one.