Continuous Batch Processing is a dynamic scheduling method on the server side of large models: the system does not have to wait for all requests in the same batch to be generated before replacing the next batch; instead, after each generation step ends, it removes completed requests and adds new ones. It mainly improves GPU utilization and overall throughput, targeting service efficiency when multiple people call the model simultaneously.
Why static batch processing is prone to empty waiting
Traditional batch processors first assemble a set of requests and then execute them together. But the input and output lengths of generation tasks vary greatly: some answer with just a few dozen tokens, while others have to go hundreds of steps. After a short request is completed, its position may remain empty, while later requests can only wait in the queue for the entire batch to end.
Continuous batch processing narrows the scheduling granularity from "a complete request" to "current generation iteration." After each round of token generation, the scheduler reviews which sequences end and which can be entered, and forms the next batch together. Batch members change continuously, so it is often referred to as iterative scheduling or dynamic batch processing.
One dispatch requires handling three tasks simultaneously
- Throughput: Try to fill available computing resources to allow more requests to advance together.
- Latency: It's not just about waiting for larger batches, but keeping users who have already arrived waiting in long queues.
- Video memory: Each active sequence must retain its contextual state. The larger the batch, the higher the memory pressure on KV Cache.
Therefore, the batch limit is not always better as much as possible. The service framework also addresses filling waste of different lengths, preemptive strategies, prioritization, fairness, as well as input pre-padding and token-per-token decoding requiring varying computing power requirements.
What will users feel?
In concurrent and stable online services, continuous batch processing often reduces queue gaps and allows more users to serve the same set of GPUs. However, increased throughput does not mean every request is faster: if the scheduler pursues overload conditions, low-priority requests may wait longer, and single request latency may even increase.
During evaluation, the delay of the first token, the interval between subsequent tokens, the total number of tokens per second, and the high-rank latency should be observed separately. Low concurrency personal local deployments can hardly produce dynamic batches, potentially limiting the benefits; Whether for APIs, chat platforms, or multi-tenant inference clusters, it is the core infrastructure that determines cost and experience.
