Filter by tag

Sort

Standing

Index

Every entry in the grimoire, including stubs. Filter in the margin, grep below.

/59/59

Activation Patchingstandard

Causal intervention swapping activations between clean and corrupt runs.

Interpretability

Adaptive moments with weight decay decoupled from the gradient.

optimisertrainingcore

Adapter Mergingstubcommon

Task arithmetic, TIES and DARE over independently trained deltas.

fine-tuningpeft

Additive Attentionstale

The original mechanism: a learned MLP scores each query–key pair.

attentionhistoryfoundations

Linear distance penalty on attention scores in place of position embeddings.

positionattention

Attention Sinksstandard

Softmax must sum to one, so heads need somewhere to dump mass; evict it and generation collapses.

attentioninferencelong-context

Attribution Patchingcommon

First-order approximation to patching via a single backward pass.

Interpretability

Block-Sparse Attentioncommon

Local windows plus a few global tokens, chosen before the scores are computed.

attentionsparselong-context

Causal Maskingstandard

The triangular mask that turns one forward pass into n training examples.

attentiontrainingcore

Chinchilla Scaling Lawscommon

Compute-optimal allocation between parameters and training tokens.

scalingtraining

Contamination Detectioncommon

n-gram and canary overlap tests between benchmark and pretraining data.

Continuous Batchingstandard

Iteration-level scheduling; requests join and leave the batch mid-flight.

Cross-Attentioncommon

Queries from one sequence, keys and values from another.

attentionconditioningmultimodal

Crosscodersstubpromising

Shared dictionaries fitted across layers or across model checkpoints.

Interpretability

Data Poisoning and Backdoorscommon

Corrupting training data so a trigger produces attacker-chosen behaviour.

attackdataprivacy

Direct Preference Optimisationstandard

Closed-form preference objective that removes the reward model.

Low-rank adaptation split into a magnitude and a direction.

fine-tuningpeft

Elo / Bradley-Terry Scoringcommon

Latent-strength model fitted to pairwise preference outcomes.

metricpreference

Feature Absorptionstubpromising

When one SAE latent swallows a more specific feature’s activation mass.

Interpretability

FlashAttentionstandard

Tiled, IO-aware attention kernel; recomputes softmax statistics online.

attentionkernelsmemory

Layer-wise second-order weight quantisation with Hessian-guided rounding.

Gradient Checkpointingstandard

Trade recompute for activation memory at segment boundaries.

Greedy Coordinate Gradientstandard

Discrete optimisation of an adversarial suffix against a refusal-trained model.

attackrobustnessalignment

Grouped-Query Attentionstandard

g key–value heads shared across h query heads; the dial between MHA and MQA.

attentionkv-cacheinference

Group-relative advantages from sampled completions; no value network.

Induction Headsstandard

A two-head circuit that completes [A][B]…[A] with [B]; the substrate of in-context learning.

Interpretability

Learning-Rate Schedulesstandard

Warmup, cosine decay, and the warmup–stable–decay alternative.

optimisertraining

Linear Attentionpromising

Drop the softmax, reassociate the matmuls, and decode in constant memory.

attentionefficiencyrecurrence

LLM-as-Judgecommon

Using a model to score model outputs, and the biases that come with it.

metricpreference

Logit Lenscommon

Project intermediate residual states through the unembedding matrix.

Interpretability

Low-rank additive update to frozen weight matrices during adaptation.

fine-tuningpeft

Membership Inferencecommon

Distinguish training members from held-out data via loss calibration.

Mixture-of-Experts Routingstandard

Top-k gate over expert FFNs with auxiliary load-balancing loss.

Model Extractioncommon

Query-only reconstruction of a victim model's decision function.

Multi-Head Attentionstandard

h attention operations in d/h-dimensional subspaces, summed back into the residual stream.

attentiontransformercore

Multi-Head Latent Attentionstubpromising

Cache a shared low-rank latent instead of per-head keys and values.

attentionkv-cacheinference

Multi-Query Attentionstale

Single shared K/V head across query heads to shrink the decode cache.

attentionkv-cache

Paged KV Cachestubstandard

Virtual-memory paging applied to the attention cache.

Path Patchingcommon

Restrict the intervention to a single computational path between components.

Interpretability

Perplexitycommon

Exponentiated mean negative log-likelihood over a held-out corpus.

Prefix Cachingstandard

Reuse KV blocks across requests that share a prompt prefix.

kv-cacheserving

Projected Gradient Descent Attackstandard

Iterative L_p-bounded perturbation maximising task loss.

attackrobustness

Prompt Injection Taxonomystandard

Direct, indirect and tool-mediated instruction hijack surfaces.

Proximal Policy Optimisationcommon

Clipped surrogate advantage objective with KL control to a reference.

4-bit NF4 base weights with paged optimiser state and LoRA adapters.

fine-tuningquantisation

RMSNormstandard

Re-scaling without re-centring; LayerNorm minus the mean subtraction.

normalisationcore

Rotary Position Embeddingstandard

Relative position injected as a per-pair rotation in complex subspaces.

positionattention

Sampling Strategiesstandard

Temperature, top-k, nucleus and min-p as successive truncations of the tail.

Scaled Dot-Product Attentionstandard

Content-based retrieval over key–value pairs with 1/√d_k logit scaling.

attentiontransformercore

Second-Order Preconditioningstubpromising

Shampoo, Muon and the orthogonalised-update family.

optimisertraining

Selective State Space Modelspromising

Linear recurrence with input-dependent dynamics; Mamba and the S4 line.

sequencerecurrencesparse

Sliding-Window Attentionstubcommon

Local band mask; receptive field grows linearly with depth.

attentionsparse

Sparse Autoencoderscommon

Overcomplete dictionary learning on activations for monosemantic features.

Interpretability

Speculative Decodingstandard

Draft model proposes tokens; target model verifies in one pass.

decodinglatency

Statistical Power for Evalsstandard

Error bars, paired tests and how many items an eval actually needs.

Steering Vectorscommon

Add a direction to the residual stream at inference to shift behaviour.

causalfeaturesalignment

Interpretability

Supervised Fine-Tuningstandard

Cross-entropy on completion tokens only; the step that makes a base model answer.

fine-tuningalignment

Gated feed-forward block; a Swish gate multiplied into a linear branch.

ZeRO Shardingstandard

Partition optimiser state, gradients and parameters across ranks.

distributedmemory

What the labels mean

Where a technique stands in current practice, reviewed as a set in 2026-09. These are judgements rather than facts, and they date — an entry with no label is one I have not formed a firm view on.

standard

Part of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.

29

common

Established and frequently the right choice, but competing with live alternatives rather than having settled the question.

21

promising

Promising and actively moving. The results are real but narrow — one lab, one model family, or one benchmark suite — and the picture may look different in a year.

6

stale

Load-bearing for understanding how the field arrived here, and replaced in practice by something on this list. Worth reading, not worth reaching for.

3