AI Grimoire
Standing note
Kept open, revised often.
143 entries, 7 stubs.
No prose written by a model.

The AI Engineer's Grimoire

A field reference for how these systems actually work: attention and its cheaper cousins, training dynamics that fail quietly, inference kernels, the interpretability toolkit, and the attack surface that follows all of it.

Each entry begins with the maths and ends with something you can run. Nothing is explained twice.

Browse the index →Random entry

Categories

01
Architectures — attention variants, position schemes, normalisation, sparse blocks
5402
Training — optimisers, schedules, sharding, RL objectives, stability failures
4003
Fine-tuning — PEFT families, adapter merging, data curation, catastrophic drift
504
Inference — kernels, KV-cache layouts, quantisation, speculative decoding
805
Interpretability — circuits, SAEs, causal interventions, probing, feature geometry
906
Adversarial ML — evasion, extraction, poisoning, privacy leakage, agent hijacks
607
Evaluation — metrics, preference aggregation, contamination, statistical power
508
Miscellany — numerics, tensor conventions, the things every other shelf assumes
309
Tooling — what the software is for, when to reach for it, and what it hides
13

Notes

The entries state the result. The notes are the working out — read in order, or not at all.

Attention from scratch
building the operation up from the problem it was invented to solve, one step at a time
01 Why attention exists02 Building the attention weights03 Reading an attention matrix04 The quadratic problem05 Many heads at once06 What attention cannot do
Alignment from scratch
from "a base model completes text" to a policy trained against a learned reward, one honest step at a time
01 What alignment is asking for02 Where demonstrations run out03 Turning comparisons into a number04 The leash05 Removing the reward model
Making it fit
the memory arithmetic of training something larger than the machine you have
01 Counting the bytes02 Throwing work away03 Splitting the work04 When it all runs at once

Recent revisions

Activation Compression
Store the saved activations in fewer bits instead of recomputing them.
Training · revised 2d ago
O(L·B·T·d)
Activation Memory Accounting
How much a forward pass must keep, and which term dominates when.
Training · revised 2d ago
O(L·B·T·d)
Activation Sparsity
Most FFN neurons are zero for any given token; skip the rows they own.
Architectures · revised 2d ago
O(n·d·s)
Agent Frameworks
Loops with tools: what the frameworks add, and how little of it is the hard part.
Tooling · revised 2d ago
—