Activation Patching
Run the model twice on a minimal pair, then splice one activation from one run into the other and measure how far the output moves. The only interpretability primitive that establishes causation rather than correlation.
Construct a clean prompt and a corrupted prompt differing in one respect. Cache the activations of both. Then run the corrupted prompt while overwriting a chosen component with its clean value, and measure recovery of the metric of interest.
Direction matters. Denoising (clean → corrupt, as above) finds components sufficient to restore the behaviour; noising (corrupt → clean) finds components necessary for it. The two disagree whenever the circuit has redundant paths, and the disagreement is informative rather than a bug.
import torch
from contextlib import contextmanager
@contextmanager
def patch(module, cache, position):
"""Overwrite module output at one sequence position."""
def hook(_mod, _inp, out):
out = out.clone()
out[:, position] = cache[:, position]
return out
handle = module.register_forward_hook(hook)
try:
yield
finally:
handle.remove()
def patching_effect(model, clean, corrupt, module, position, metric):
with torch.no_grad():
clean_acts = {}
h = module.register_forward_hook(
lambda _m, _i, o: clean_acts.setdefault("a", o.detach())
)
base_clean = metric(model(clean))
h.remove()
base_corrupt = metric(model(corrupt))
with patch(module, clean_acts["a"], position):
patched = metric(model(corrupt))
return (patched - base_corrupt) / (base_clean - base_corrupt)The hook must clone before writing. Mutating the output tensor in place corrupts
the residual stream for every later reader in ways that do not raise an error and
do not look wrong in the metric. Always remove handles in a finally — a leaked
hook survives into the next experiment and quietly patches everything after it.