Tooling / serving
Serving and runtimes
Everything between a checkpoint on disk and a token on a wire. The split that matters is throughput against convenience: a datacentre server batching hundreds of concurrent requests is solving a different problem from a binary that runs a quantised model on a laptop, and the two families have almost no code in common.
4 entries.
Entries
09.01.1—09.01.2—09.01.3—09.01.4—
llama.cppstandard
A dependency-free C++ inference engine, and the GGUF format everything local uses.
Ollamastandard
A local model runner: pull a name, get a server, forget the flags.
SGLangcommon
A serving runtime built around prefix reuse and constrained decoding.
vLLMstandard
The throughput server: paged KV cache, continuous batching, one endpoint.