DeepSeek's inference framework DeepInfer was open-sourced on October 20, 2026: DeepSeek has placed the inference framework that powers its own large-scale production services in its official GitHub repository, deepseek-ai/DeepInfer, under the Apache-2.0 license. The first open-source release provides offline inference, and the only model currently supported is DeepSeek-V4.1-Flash.
What exactly has been released
The kernel layer bundles DeepSeek's own operators — FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect. The runtime supports attention data parallelism with balanced context parallelism, plus MoE expert parallelism with EPLB load balancing. On the decoding side it offers DSpark with confidence-scheduled verification and bounded replay for sliding-window attention. The asynchronous scheduler promises zero-overhead batch scheduling, and the feature list also includes batch-invariant inference, deterministic sampling and structured outputs constrained by regex, JSON Schema or EBNF grammars.
Performance, hardware and entry requirements
The published performance charts measure prefill on 4 SM100 GPUs and decoding on 64. Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, requiring GCC 13 and Python 3.12, plus CUDA 13 on the NVIDIA side and CANN 9.2 on the Ascend side. The framework was originally built mainly on vLLM, with additional inspiration from SGLang and TensorRT-LLM — and the DeepSeek-V4.1-Flash-1017 release covered on this site is currently the only model it serves.
What was left out is the hardest part to copy
The repository is candid about its limits: a complete inference service still needs a KV cache system, prefill/decode disaggregation (most of the required engine functionality is already implemented) and load balancing across data-parallel replicas — none of which is included in this release. In other words, outsiders get the engine and the gearbox, but the fuel lines and the dispatch center are still theirs to build. For teams that want to serve DeepSeek models themselves, this removes the need to tune kernels from scratch. For existing frameworks such as vLLM and SGLang, it is the first chance to compare, line by line, the trade-offs inside DeepSeek's production system: where batch invariance and determinism are worth their performance cost, and where asynchronous scheduling hides CPU overhead.
Why this release is worth watching
Over the past two years, DeepSeek's open-source output has mostly been model weights and individual kernel libraries; the framework layer was known by reputation only. Releasing the production framework as a whole lands first on inference cost: smaller teams deploying V4.1-Flash no longer have to start from a general-purpose framework's default configuration. It also lands on the Ascend ecosystem — DeepInfer treats the Ascend 950 as a first-class target, continuing the line of DeepSeek's earlier Ascend ports of its kernel libraries. A note of caution: the repository was created on October 16 and ships offline inference only, so it is still some distance from a production-grade online service. Whether it deserves migration effort will depend on the update cadence and issue responsiveness over the next few weeks.
