Filter by tag

Sort

Standing

Index

Every entry in the grimoire, including stubs. Filter in the margin, grep below.

/143/143

Activation Compressionpromising

Store the saved activations in fewer bits instead of recomputing them.

memorytrainingnumerics

Activation Memory Accountingstandard

How much a forward pass must keep, and which term dominates when.

memorytrainingcore

Activation Patchingstandard

Causal intervention swapping activations between clean and corrupt runs.

Interpretability

Activation Sparsitypromising

Most FFN neurons are zero for any given token; skip the rows they own.

ffninferencesparse

Adaptive moments with weight decay decoupled from the gradient.

optimisertrainingcore

Adapter Mergingstubcommon

Task arithmetic, TIES and DARE over independently trained deltas.

fine-tuningpeft

Additive Attentionstale

The original mechanism: a learned MLP scores each query–key pair.

attentionhistoryfoundations

Agent Frameworkscommon

Loops with tools: what the frameworks add, and how little of it is the hard part.

toolingorchestrationagents

Linear distance penalty on attention scores in place of position embeddings.

positionattention

Attention Sinksstandard

Softmax must sum to one, so heads need somewhere to dump mass; evict it and generation collapses.

attentioninferencelong-context

Attribution Patchingcommon

First-order approximation to patching via a single backward pass.

Interpretability

Batch Normalisationstale

Normalise each feature across the batch; keep running statistics for inference.

normalisationcoretraining

Block-Sparse Attentioncommon

Local windows plus a few global tokens, chosen before the scores are computed.

attentionsparselong-context

Causal Maskingstandard

The triangular mask that turns one forward pass into n training examples.

attentiontrainingcore

Chinchilla Scaling Lawscommon

Compute-optimal allocation between parameters and training tokens.

scalingtraining

Collective Communicationstandard

All-reduce, all-gather, reduce-scatter, all-to-all — the six primitives everything is built from.

distributedtrainingcore

Compute-Optimal versus Inference-Optimalstandard

Chinchilla minimises training cost. Nobody trains a model they only train once.

scalingtraininginference

Config-Driven Fine-Tuningcommon

Axolotl, LLaMA-Factory and the case for a YAML file instead of a script.

toolingtrainingfine-tuning

Contamination Detectioncommon

n-gram and canary overlap tests between benchmark and pretraining data.

Context Parallelismcommon

Split the sequence across devices, and pass the KV blocks around a ring.

distributedtraininglong-context

Continuous Batchingstandard

Iteration-level scheduling; requests join and leave the batch mid-flight.

CPU and NVMe Offloadingcommon

Park what is not in use on host memory or disk, bounded by PCIe rather than compute.

memorytraininginference

Cross-Attentioncommon

Queries from one sequence, keys and values from another.

attentionconditioningmultimodal

Crosscodersstubpromising

Shared dictionaries fitted across layers or across model checkpoints.

Interpretability

Data Parallelismstandard

Replicate the model, split the batch, average the gradients.

distributedtrainingcore

Data Poisoning and Backdoorscommon

Corrupting training data so a trigger produces attacker-chosen behaviour.

attackdataprivacy

Data-Constrained Scalingcommon

What repeated tokens are worth, and how many epochs before they are worth nothing.

scalingtrainingdata

DeepNormpromising

Post-norm with a scaled residual and depth-aware initialisation, to a thousand layers.

normalisationtraining

Direct Preference Optimisationstandard

Closed-form preference objective that removes the reward model.

Low-rank adaptation split into a magnitude and a direction.

fine-tuningpeft

Dynamic Tanhpromising

Replace the normalisation layer with a scaled tanh; no statistics at all.

normalisationtraining

Elo / Bradley-Terry Scoringcommon

Latent-strength model fitted to pairwise preference outcomes.

metricpreference

Emergent Abilitiescommon

Capabilities that appear abruptly with scale, and the argument that the abruptness is the metric.

scalingevaluation

Expert Capacity and Token Droppingcommon

Fixed buffers per expert, and what happens to the tokens that do not fit.

sparsetrainingscaling

Expert Parallelismstandard

Experts on separate devices, and the two all-to-alls that connect them.

sparsescalingtraining

Expert-Choice Routingpromising

Invert the assignment: each expert picks its top tokens, not the reverse.

Fault Tolerancecommon

At ten thousand devices something fails hourly; the run has to survive it.

distributedtraining

Feature Absorptionstubpromising

When one SAE latent swallows a more specific feature’s activation mass.

Interpretability

Feed-Forward as Key-Value Memorycommon

Read the FFN as a lookup table: W₁ rows are patterns, W₂ columns are what gets written.

ffninterpretabilitycore

Feed-Forward Networksstandard

Two matrices and a nonlinearity, applied to each position alone.

FlashAttentionstandard

Tiled, IO-aware attention kernel; recomputes softmax statistics online.

attentionkernelsmemory

Floating-Point Formatsstandard

What bf16, fp16, fp8 and fp4 each give up, and which failure follows.

numericstraininginferencecore

A smooth gate on the identity, and the activation lineage it belongs to.

Layer-wise second-order weight quantisation with Hessian-guided rounding.

Gradient Accumulationstandard

Simulate a large batch with several small ones, and the normalisation bug that follows.

memorytrainingcore

Gradient Checkpointingstandard

Trade recompute for activation memory at segment boundaries.

Gradient Clippingstandard

Bound the update norm so one bad batch cannot destroy a run.

optimiserstrainingcore

Greedy Coordinate Gradientstandard

Discrete optimisation of an adversarial suffix against a refusal-trained model.

attackrobustnessalignment

Group Normalisationcommon

Normalise over channel groups within one example; batch size drops out.

normalisationvision

Grouped-Query Attentionstandard

g key–value heads shared across h query heads; the dial between MHA and MQA.

attentionkv-cacheinference

Group-relative advantages from sampled completions; no value network.

The initialisation that makes a linear recurrence remember: online polynomial projection.

recurrencesequence

Hybrid Recurrent-Attentioncommon

A few attention layers among many recurrent ones, because the ratio beats either extreme.

recurrenceattentionlong-contextinference

O(n·d + n²·d/r)

Induction Headsstandard

A two-head circuit that completes [A][B]…[A] with [B]; the substrate of in-context learning.

Interpretability

Kaplan Scaling Lawsstale

The 2020 power laws, and the three choices that made them recommend the wrong model.

scalingtraining

KL Regularisationstandard

A leash to the reference policy, and the three ways it is estimated.

objectivestrainingalignment

Knowledge Distillationcommon

Train the small model on the big one’s distribution, not on the labels.

objectivestraininginference

LangChaincommon

The composition framework everyone starts with, and the argument about whether to keep it.

toolingorchestrationagents

LayerNormcommon

Re-centre and re-scale each activation vector over its own features.

normalisationcore

Learned Absolute Position Embeddingsstale

A lookup table indexed by position, trained like any other embedding.

positionattentioncore

Learning-Rate Schedulesstandard

Warmup, cosine decay, and the warmup–stable–decay alternative.

optimisertraining

Linear Attentionpromising

Drop the softmax, reassociate the matmuls, and decode in constant memory.

attentionefficiencyrecurrence

llama.cppstandard

A dependency-free C++ inference engine, and the GGUF format everything local uses.

toolinginferencequantisation

LlamaIndexcommon

A retrieval framework: ingestion, chunking, indexing and query pipelines.

toolingorchestrationretrieval

LLM-as-Judgecommon

Using a model to score model outputs, and the biases that come with it.

metricpreference

Logit Lenscommon

Project intermediate residual states through the unembedding matrix.

Interpretability

Low-rank additive update to frozen weight matrices during adaptation.

fine-tuningpeft

Loss-Free Load Balancingcommon

Balance experts with a per-expert bias on the router, not with a gradient.

LSTM and GRUstale

Gates and an additive cell path, so the gradient has somewhere to travel.

recurrencesequencecore

Masked Language Modellingcommon

Corrupt the input, predict what was removed; the objective that lost to next-token.

objectivestraining

Maximal Update Parameterisationcommon

Scale the initialisation and learning rates so the optimum transfers across width.

optimiserstrainingscaling

Membership Inferencecommon

Distinguish training members from held-out data via loss calibration.

Memory Fragmentationcommon

The allocator that says out-of-memory with gigabytes free, and what to do about it.

memorytraininginference

Memory-Efficient Optimiserscommon

Adam’s state is twice the model; factorise it, quantise it, or project it.

optimiserstrainingmemory

Mixed-Precision Trainingstandard

Half-precision compute, full-precision accumulation, and the master copy in between.

memorytrainingnumericscore

Mixture-of-Depthspromising

Route tokens past whole blocks: conditional depth rather than conditional width.

sparseinferencescaling

Mixture-of-Experts Routingstandard

Top-k gate over expert FFNs with auxiliary load-balancing loss.

Model Context Protocolcommon

A standard interface between a model host and the tools it can call.

toolingorchestrationagents

Model Extractioncommon

Query-only reconstruction of a victim model's decision function.

Multi-Head Attentionstandard

h attention operations in d/h-dimensional subspaces, summed back into the residual stream.

attentiontransformercore

Multi-Head Latent Attentionstubpromising

Cache a shared low-rank latent instead of per-head keys and values.

attentionkv-cacheinference

Multi-Query Attentionstale

Single shared K/V head across query heads to shrink the decode cache.

attentionkv-cache

Multimodal RoPEcommon

Splitting the rotary dimensions across time, height and width for interleaved image and video input.

positionattentionmultimodal

Next-Token Predictionstandard

Cross-entropy over the next token, and everything that follows from teacher forcing.

objectivestrainingcore

Nondeterminismstandard

Why the same prompt, the same weights and the same seed give different logits.

numericsinferenceevaluation

No positional encoding at all: the causal mask already carries the information.

positionattentionlong-context

A local model runner: pull a name, get a server, forget the flags.

toolinginference

Paged KV Cachestubstandard

Virtual-memory paging applied to the attention cache.

Parallel Attention and FFNcommon

Run both branches off the same input instead of one after the other.

ffnattentiontraining

Parallel Scanstandard

Compute a linear recurrence in log-depth instead of a loop.

recurrencesequencecore

Parallelism Compositionstandard

Four or five strategies at once, and the topology decides the order.

distributedtrainingscaling

Path Patchingcommon

Restrict the intervention to a single computational path between components.

Interpretability

PEFT and TRLstandard

The adapter library and the preference-training library, and how they compose.

toolingtrainingfine-tuning

Perplexitycommon

Exponentiated mean negative log-likelihood over a held-out corpus.

Pipeline Parallelismstandard

Layers on different devices, and the bubble that schedule design exists to shrink.

distributedtraining

Pre-Norm and Post-Normstandard

Where the normalisation sits relative to the residual add, and what each choice costs.

normalisationtrainingcore

Prefix Cachingstandard

Reuse KV blocks across requests that share a prompt prefix.

kv-cacheserving

Product-Key Memorypromising

A million-entry lookup layer searched in √N time by factorising the key space.

ffnsparsescaling

Projected Gradient Descent Attackstandard

Iterative L_p-bounded perturbation maximising task loss.

attackrobustness

Prompt Injection Taxonomystandard

Direct, indirect and tool-mediated instruction hijack surfaces.

Proximal Policy Optimisationcommon

Clipped surrogate advantage objective with KL control to a reference.

QK Normalisationcommon

Normalise queries and keys before the dot product to bound the attention logits.

normalisationattentiontraining

4-bit NF4 base weights with paged optimiser state and LoRA adapters.

fine-tuningquantisation

Recurrent Neural Networksstale

One hidden state, carried forward one step at a time, and why the gradient dies.

recurrencesequencecore

Relative Position Embeddingsstale

Learned vectors indexed by query–key offset, injected into the score rather than the input.

positionattention

Reward Modellingstandard

Fit a scalar reward to pairwise preferences, then watch the policy exploit it.

objectivestrainingalignment

RMSNormstandard

Re-scaling without re-centring; LayerNorm minus the mean subtraction.

normalisationcore

RoPE Scalingstandard

Extending a rotary model past its training length by rewriting the frequencies.

positionlong-contextattention

Rotary Position Embeddingstandard

Relative position injected as a per-pair rotation in complex subspaces.

positionattention

A transformer-shaped block whose token mixer is an exponentially decaying sum.

recurrencesequenceinference

Sampling Strategiesstandard

Temperature, top-k, nucleus and min-p as successive truncations of the tail.

Scaled Dot-Product Attentionstandard

Content-based retrieval over key–value pairs with 1/√d_k logit scaling.

attentiontransformercore

Project onto a learned-radius sphere: one scalar for the whole layer.

Scaling Law Methodologycommon

How the curves are fitted, and how the fits go wrong.

scalingevaluationtraining

Schedule-Free Optimisationpromising

Averaging in place of a decay schedule, so the run has no fixed horizon.

optimiserstraining

Second-Order Preconditioningstubpromising

Shampoo, Muon and the orthogonalised-update family.

optimisertraining

Selective Recomputationcommon

Recompute only the cheap-and-large activations, not the whole block.

Selective State Space Modelspromising

Linear recurrence with input-dependent dynamics; Mamba and the S4 line.

sequencerecurrencesparse

A serving runtime built around prefix reuse and constrained decoding.

toolinginferenceserving

Shared and Fine-Grained Expertsstandard

Many small experts, plus a few that every token uses.

sparsescalingcore

Sinusoidal Position Encodingstale

Fixed sinusoids of geometrically spaced frequencies, added to the token embeddings.

positionattentioncore

Sliding-Window Attentionstubcommon

Local band mask; receptive field grows linearly with depth.

attentionsparse

Sparse Autoencoderscommon

Overcomplete dictionary learning on activations for monosemantic features.

Interpretability

Sparse Upcyclingcommon

Turn a finished dense checkpoint into a mixture of experts by copying the FFN.

sparsetrainingscaling

Speculative Decodingstandard

Draft model proposes tokens; target model verifies in one pass.

decodinglatency

Squared ReLUpromising

ReLU, then square: cheaper than a gate and far sparser than either.

State Space Dualitypromising

Selective SSMs and linear attention are the same operator, written two ways.

recurrencesequenceattention

Statistical Power for Evalsstandard

Error bars, paired tests and how many items an eval actually needs.

Steering Vectorscommon

Add a direction to the residual stream at inference to shift behaviour.

causalfeaturesalignment

Interpretability

Stochastic Gradient Descentcommon

A step against the gradient, momentum on top, and why it loses to Adam on transformers.

optimiserstrainingcore

Supervised Fine-Tuningstandard

Cross-entropy on completion tokens only; the step that makes a base model answer.

fine-tuningalignment

Gated feed-forward block; a Swish gate multiplied into a linear branch.

T5 Relative Position Biascommon

One learned scalar per head per log-spaced distance bucket, added to the logits.

positionattention

Tensor Layout Conventionscommon

Axis order, pairing schemes and epsilon placement: arbitrary until weights exist.

conventionsinferencecore

Tensor Parallelismstandard

Split each matrix across devices; two all-reduces per layer buy the memory back.

distributedtraininginference

Test-Time Compute Scalingstandard

Spend more compute per question instead of more compute per model.

scalinginferenceobjectives

Training Compute Accountingstandard

Why a training run costs 6ND FLOPs, and where the six comes from.

scalingtrainingcore

Transformersstandard

The reference implementation of nearly every architecture, and the format everything reads.

toolingtrainingcore

Hand-written kernels and manual backward passes for fine-tuning on one GPU.

toolingtrainingfine-tuningmemory

Vector Databasescommon

Approximate nearest-neighbour search, and whether you need a separate system for it.

toolingorchestrationretrieval

The throughput server: paged KV cache, continuous batching, one endpoint.

toolinginferenceserving

Rotary embeddings with a per-dimension exponential decay, for stable extrapolation.

positionattentionlong-context

ZeRO Shardingstandard

Partition optimiser state, gradients and parameters across ranks.

distributedmemory

What the labels mean

Where a technique stands in current practice, reviewed as a set in 2026-09. These are judgements rather than facts, and they date — an entry with no label is one I have not formed a firm view on.

standard

Part of the default recipe. Present in most frontier models trained today, and the thing a new design departs from rather than argues for.

57

common

Established and frequently the right choice, but competing with live alternatives rather than having settled the question.

56

promising

Promising and actively moving. The results are real but narrow — one lab, one model family, or one benchmark suite — and the picture may look different in a year.

19

stale

Load-bearing for understanding how the field arrived here, and replaced in practice by something on this list. Worth reading, not worth reaching for.

11