AI · October 23, 2026
llm-d 1.0 launches under CNCF with production-ready features for distributed AI inference
The llm-d project has released version 1.0, an open source software platform for distributed large language model inference now governed by the Cloud Native Computing Foundation. This release targets production environments, introducing capabilities for multi-tenant quality of service, zero-downtime operations, a stable configuration API, and agent-aware serving. The project was established in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA before transitioning to CNCF oversight. It is designed to assist organizations running inference on large-scale infrastructure, particularly as agentic AI workloads create unpredictable traffic patterns and heavy context management demands that differ from traditional inference tasks.
Version 1.0 emphasizes operational stability for production inference. The multi-tenant quality of service feature implements flow control that considers priority and fairness, allowing low-latency requests and high-traffic agent workloads to share the same fleet without resource contention. Zero-downtime operations support router high availability through hot standby and rolling updates, ensuring traffic continues during upgrades or planned maintenance. The system manages prefill and decode replicas as a single logical unit, enabling coordinated rollouts and lockstep upgrades that make disaggregated serving viable in production. A stable foundation is provided by configuration API v1 and a request path tested across multiple clouds and accelerators, offering a standard for organizational use.
Agent-aware serving is a key addition, utilizing session-aware routing to direct multi-turn conversations to nodes where context is already cached. This is combined with turn-priority scheduling and native support for the state of the Responses API, allowing agents to reuse context across turns and reducing redundant computation. The release builds on previous capabilities, including hardware heterogeneity support, prefix caching, load-aware routing, KV cache offloading, prefill and decode disaggregation, wide expert parallelism, and batch and asynchronous inference. The platform is compatible with multiple model servers, such as vLLM and SGLang.
Performance data from IBM Research indicates significant cost and efficiency improvements. In a recent use case involving an open frontier model for quantum error correction hardware design exploration, inference costs were 11.5 times lower than proprietary systems. The prefix cache hit rate was approximately 98 percent, with an average serving cost of 0.11 dollars per one million tokens. In another test using a benchmark workload, researchers deployed the GLM 5.2 open-weight mixture-of-experts model, which has approximately 753 billion parameters, on widely used NVIDIA GPUs. When running hundreds of parallel agentic sessions, the system achieved output rates of up to 6.6 million tokens per minute. It supported 3,000 concurrent coding agents without preemption. Based on current cloud rental rates, the cost per token was 5 to 10 times lower than equivalent commercial API prices, with the largest savings observed in input-heavy traffic typical of agentic workloads. A third test set showed a threefold increase in output throughput.
The llm-d community now includes more than 500 contributors from over 25 organizations. Partners have outlined specific integration plans. Google Cloud is incorporating llm-d 1.0 into Google Kubernetes Engine and enabling it on Google TPU and NVIDIA GPU. CoreWeave allows customers to run llm-d on CoreWeave Kubernetes Service alongside NVIDIA Dynamo. NVIDIA plans to use llm-d on NVIDIA Vera Rubin, while AMD supports it on AMD Instinct GPU. User organizations such as DigitalOcean, JetBrains, Mistral, Cohere, and Roblox are already using llm-d in their model serving systems. The community plans to continue development to support new hardware and models as they emerge.