Humming was announced as open source on October 21, 2026 by Ant Group and the vLLM community on the official vLLM blog. It is a general-purpose, high-performance kernel library for quantized inference, now hosted under the vllm-project organization on GitHub. The problem it tackles is not that any single operator is too slow, but that quantization formats have multiplied beyond what anyone can maintain one kernel at a time.
The quantization matrix exploded, and hand-writing kernels no longer scales
Weights and activations each vary in bit width, numeric format and scaling scheme: W4A16, W8A8, W4A8 and W4A4 are only the starting point, and even at 4 bits there is INT4 versus NVFP4, layered with group sizes and symmetric or asymmetric quantization. Every new combination traditionally meant another set of weight preprocessing, kernel selection logic and tests. Humming separates the quantization description from execution choices: formats exported by different quantization tools are first converted into one standard representation, then three configuration layers — layer, compute and tuning — assemble a kernel specialized for the target GPU and workload, compiled just in time. Compiling a single W4A16 kernel on an H200 drops from 23.35 seconds with CUTLASS to 1.82 seconds, cutting the cost of tuning iterations by an order of magnitude before any inference runs at all.
Read the baselines before the multipliers
In the published microbenchmarks, dense W4A16 reaches roughly 3 to 4 times the cuBLAS baseline at small batch sizes and about twice vLLM Marlin at large batches on H200; the MoE grouped path reaches around 5 times in some B300 cases. W8A8 runs at about 2.6x and 2.5x the corresponding DeepGEMM implementations on H20 and H200. The end-to-end test is more telling: serving DeepSeek-V4.1-Flash on eight H20 GPUs, switching MoE execution from Marlin W4A16 to Humming W4A8 improved overall throughput by 14% to 43% and cut time to first token by 25% to 32%, with output speed at concurrency 64 rising from 442 to 634 tokens per second. All figures come from the official blog's own test setup, and the margins will move with different hardware and models, but the direction is consistent: the storage advantage of low-precision weights is finally being converted into speed by the kernels.
Getting started: bundled with vLLM since 0.29
Humming supports every NVIDIA architecture from Turing onward, plus CUDA-compatible devices such as T-Head's Zhenwu 810E; the install package is only about 500 KB because it ships no precompiled kernels and generates them on demand. Since vLLM 0.29.0, CUDA installs already include Humming as a dependency, enabled at launch with --linear-backend humming and --moe-backend humming. One boundary is worth keeping in mind: tuning currently relies on device heuristics, and the team has put autotuning on its roadmap rather than in this release. For teams that are sensitive to inference costs and tired of juggling models in multiple formats, it deserves a place in evaluations now. Like the DeepSeek open-source inference framework DeepInfer covered on this site earlier, infrastructure open-sourcing of this kind is ultimately a bet on the long-term operations bill.
