Tree — 04

kv-cache/2

decoding/1

kernels/1

quantisation/1

Inference

Serving is a memory-bandwidth problem wearing a compute problem’s clothes. Kernels that avoid materialising intermediates, cache layouts that avoid re-reading them, and quantisation schemes that shrink what must be read at all.

Five entries. One is a stub and is marked as such — read them as open questions rather than answers.

Entries

Tiled, IO-aware attention kernel; recomputes softmax statistics online.

Layer-wise second-order weight quantisation with Hessian-guided rounding.

Multi-Query Attention

Single shared K/V head across query heads to shrink the decode cache.

Paged KV Cachestub

Virtual-memory paging applied to the attention cache.

Speculative Decoding

Draft model proposes tokens; target model verifies in one pass.