On August 21, 2026, the RadixArk SGLang team and the Ant Ling Infra team announced the single-request decoding optimization results for Ling-3.0-flash on the LMSYS official blog. In a fixed environment with 4 NVIDIA Blackwell GPUs, TP4, bf16, and concurrency set to 1, the optimized NEXTN path increased average output throughput from 288 tok/s to 606 tok/s, and reduced average output token time from 3.33 milliseconds to 1.53 milliseconds.
Why is Batch 1 hard to optimize?
This result applies to a low-concurrency scenario where only one request is continuously decoded. Without large-scale requests to share startup and scheduling overhead, synchronization between CPU and GPU and small kernel startup delays directly reflect user wait times, making it harder to hide system overhead than high-throughput batch processing.
Performance isn't just about doubling with a single core
The team first removed the CPU sequence length synchronization that would freeze the host at every step, allowing the host preparation to overlap with GPU execution; Then, programmatic dependency initiation is used to connect MoE, routing, KDA, and total reduction paths to reduce gaps between small cores. The remaining optimizations also include operator fusion, KDA recalibration, and adjusting the routing gate and output head calculations from fp32 to bf16.
These changes collectively shortened critical paths. According to the official explanation, the BF16 router gate and output head are the largest single changes after the structural adjustment, bringing about a 10% improvement. The team did not focus solely on short-term peaks but evaluated stability using average time per token and acceptance length.
DSpark also included "guessing more tokens" into optimization
The same report also tested DSpark confidence scheduling speculative decoding. In a controlled comparison of the same machine, the same command, and 1000 requests, DSpark achieved an average of 1120 tok/s, 0.78 ms TPOT, and an average acceptance length of 9.95. Its advantage is not only faster in a single step, but also in the ability to submit more prediction tokens per validation.
But 1120 tok/s cannot be directly considered the universal speed for all tasks. Testing uses a fixed 8192 token input, 1024 token output, greedy decoding, and synthetic random loads; Tip content, output distribution, context length, sampling settings, and hardware all affect the acceptance rate of speculative tokens. The official statement also clearly treats 9.95 as the result under this workload, not as an inherent constant of the model.
What does it mean for deployment teams?
This optimization did not change model parameters or answer quality; what changed was how the inference system arranges host work, GPU kernels, and draft validation. Low concurrency latency often spans scheduling, communication, numerical accuracy, and speculative strategies, making it difficult to explain true speed based solely on graphics card computing power.
Reproduction experiments should lock hardware, SGLang version, input/output length, and decoding parameters, then compare TPOT, throughput, and correctness. Applying experimental numbers directly to different GPUs or business traffic easily leads to overly optimistic capacity estimates.
