Second-Order Preconditioning
Update rules that precondition by a factored second-moment matrix rather than a diagonal, and the cheaper orthogonalisation approximations that have displaced them in practice.
Standing
PromisingPromising and actively moving. The results are real but narrow — one lab, one model family, or one benchmark suite — and the picture may look different in a year.
Real wall-clock wins reported at meaningful scale, and not yet the default. This is the most likely entry on this list to be relabelled within a year.
judged as of 2026-09 · what the labels mean
Status
This entry is a stub. The maths has not been rederived by hand, or the implementation has not been run on real tensors, so it is recorded here as an open question rather than an answer.
Read what follows as a pointer to the sources, not as a settled account.
Notes
Adam preconditions by a diagonal — one scalar per parameter, no knowledge of how parameters interact. Shampoo instead maintains a Kronecker-factored approximation to the full second-moment matrix, one factor per tensor axis, and the orthogonalised-update family approximates the resulting inverse root with a few Newton–Schulz iterations rather than an eigendecomposition.
The maths here has not been rederived by hand, and in particular the relationship between the Newton–Schulz coefficient choice and the conditioning of the update is recorded from the sources rather than checked. Treat this entry as a pointer.