Prime Inference was officially released by Prime Intellect on October 2, 2026, as an inference service for frontier open models. Across multiple data centers, on its own GPU infrastructure, it offers both serverless endpoints and reserved capacity, with production-grade SLAs, security and privacy by design, unified billing and team-level usage tracking, support for automatic failover across data centers, and an access method compatible with the OpenAI interface format.
Proven at home first, then sold to others
Prime Inference began as Prime Intellect's own internal platform, supporting large-scale reinforcement learning rollouts, synthetic data generation, evaluations and long-running coding agents, processing nearly one trillion tokens a day internally, and it has handled large-scale production deployments for customers since January 2026. Its first public deployment is GLM-5.3, launched on OpenRouter on September 22 as one of the fastest GLM-5.3 endpoints there, with a tool-calling error rate close to zero and 100% availability since launch.
Running on Blackwell, with a familiar software stack
It currently runs on NVIDIA Blackwell, with Vera Rubin to follow soon. On the software side, it combines NVIDIA Dynamo, vLLM, Mooncake and FlashInfer, works closely with Inferact and NVIDIA, and contributes improvements back upstream. For more on this kind of foundation, see What teams is vLLM right for? It is a high-performance inference foundation, not a chat product that works out of the box.
The engineering numbers tell the story: separating prefill and decoding onto different GPU groups cut p90 inter-token latency by nearly 40%; halving the per-step prefill token budget from 8K to 4K cut median queueing wait from 550 milliseconds to 110 milliseconds and reduced median time to first token by about 20%; and compressing the KV cache with NVFP4 shrank each row from 576 bytes to 352 bytes, raising cache capacity by about 50% on the same memory, from about 1.09 million tokens per decoder to 1.63 million tokens. At a target of 100 tokens per second per user, with a 1:4 prefill-to-decode ratio, each prefill group can serve 66 concurrent sessions.
For agent workloads, it is no longer just about compute
Agent workloads are defined by long contexts and multi-turn reuse of history, so the bottleneck in inference services more and more lies in caching and scheduling, not just compute itself. Open model weights do not mean a model can go straight into production; companies still need usage tracking, failover and an availability commitment someone is accountable for. Prime Inference turns all of that into a managed layer with SLAs, exactly the piece open models have been missing on the way to production.
