AI Grimoire
08 — siblings
numerics/
conventions/

Miscellany / numerics

Numerics

Every equation in this book is written over the reals and executed over a few thousand representable values. Most of the time the gap does not matter. The entries here are the cases where it does — where the format decides whether a run converges, and where the order of a reduction decides whether two identical prompts produce identical logits.

2 entries.

Entries

08.01.1
Floating-Point Formatsstandard
What bf16, fp16, fp8 and fp4 each give up, and which failure follows.
08.01.2
Nondeterminismstandard
Why the same prompt, the same weights and the same seed give different logits.