LLM Inference Optimization
Change how requests are batched and the same GPU can serve several times the throughput. This explains what continuous batching, speculative decoding, KV cache and context caching each save, and how far kernel libraries and sparse architectures can push it.
Continuous batching shrinks scheduling to one generation step; speculative decoding has a small model draft while the large one verifies a block at once. A KV cache reuses attention state, context caching keeps results for resent documents. MoE activates few experts per token, Kimi Linear cuts the KV cache at one million context by about 75%. Gains depend on workload: code accepts long drafts, open-ended writing often does not.
