llm-d 1.0 made its debut on October 20, 2026 at PyTorch Conference North America, with IBM Research announcing the release on its official blog. The Kubernetes-native distributed inference project is led by Red Hat, IBM and Google, which co-founded it in 2025 together with CoreWeave and NVIDIA; AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI have since joined as contributors. The point of 1.0 is not a faster benchmark number, but the move from inference that merely runs to inference that can be operated over the long term.
What 1.0 adds is operations, not another performance trick
The five techniques llm-d is known for — prefix-cache-aware routing, KV-cache offloading, batch inference optimization, prefill/decode disaggregation and wide expert parallelism — are unchanged in 1.0. What changed is the operational layer above them: a stable API, multi-tenant quality of service, zero-downtime upgrades, and flow control that keeps workloads with competing priorities from dragging each other down on shared infrastructure. Carlos Costa, Distinguished Engineer at IBM Research and an llm-d co-founder, puts it plainly: 1.0 is about running llm-d like any other critical service, not just posting impressive demo numbers.
One endpoint, different treatment for people and agents
Enterprise inference traffic is changing shape: behind a single endpoint there are people waiting for an instant reply from a coding assistant, and agents that fire bursts of requests, pause to use tools, and come back for another round. The flow control and fairness mechanisms in 1.0 decide explicitly who gets resources first at peak demand, and pin multi-turn requests to the same replica so prefix caches stay warm. In IBM's testing, the prefix-aware router captured 97% of available cache reuse on an agentic coding workload, and a self-hosted demonstration ran 3,000 software-engineering agents concurrently at one fifth to one tenth of the cost of the equivalent hosted service.
A vendor-neutral option for teams serving models themselves
llm-d does not replace inference engines such as vLLM or SGLang; it sits above them and adds distributed routing and operational capabilities, and the project is hosted as a CNCF sandbox project. For teams already serving models on their own Kubernetes clusters, 1.0 lowers the engineering bar for turning open-weight models into production services, and makes it possible to manage where compute goes and how much cache is actually hit. The limits deserve stating too: it still assumes Kubernetes operations skills, and small teams happily using a cloud provider's managed inference may not save enough to justify running the stack themselves.
