DeepSeek open-sourced DeepInfer, the Apache 2.0 engine behind its API. Here's what it needs, what's missing, and who should actually run it.
Vamsi Tallapudi
Manager, Architect Technology at Cognizant
DeepSeek open-sourced DeepInfer on October 20, 2026, the actual inference engine it runs in production to serve DeepSeek-V4.1-Flash to millions of API calls a day. Not a trimmed-down demo version. The real thing, under Apache 2.0, on GitHub.
That's a different move than what we're used to seeing. Companies open-source model weights all the time. They almost never open-source the serving stack that makes those weights cheap to run at scale. DeepSeek just did, and it tells you a lot about where the real competitive advantage in AI has shifted.
DeepInfer is the inference framework behind DeepSeek's API: the software layer that takes a trained model and actually serves it to users fast and cheap. It's built primarily on vLLM, with pieces borrowed from SGLang and TensorRT-LLM, then bolted together with DeepSeek's own kernels: FlashMLA, DeepGEMM, and DeepEP, all of which the company had already open-sourced separately.
On top of that it adds DSpark, DeepSeek's speculative decoding scheme. A small draft model proposes five tokens at once, a confidence head decides which ones are worth verifying, and the full model only double-checks the ones that need it. DeepSeek says this cuts verification compute by roughly 30%. There's also a prefill compression trick that pushes only the final 128 tokens of a prompt through the heavy decoder layers instead of the whole sequence, nearly halving prefill cost. Add asynchronous scheduling, batch-invariant inference, deterministic sampling, and structured outputs, and you get a 4-7% throughput bump at real production load. Small-sounding numbers that add up fast when you're running at DeepSeek's scale.
This is the part that should make people pause. DeepSeek says a single production cluster pushes through 155 trillion tokens a day, and that off-peak pricing still nets around $38 of revenue per GPU per hour. The README recommends a minimum of 64 GPUs for prefill and 64 for decode just to run efficiently. This isn't software meant for a spare A100 in your closet.
Those numbers matter beyond bragging rights. They're the closest thing we have to hard evidence that frontier-model inference is actually profitable, not just subsidized. The Hacker News thread on the release spent a lot of time on exactly this: most commenters agreed inference itself clears a profit, while training and R&D remain the money pit that has to be covered separately. One point came up repeatedly too: China's cheaper electricity and ongoing nuclear buildout give DeepSeek a structural cost edge that no amount of clever kernel engineering on the US side can fully erase.
Two platforms, day one: Nvidia's SM100 (Blackwell) data-center GPUs, and Huawei's Ascend 950 NPUs. No support for older Nvidia chips like Hopper or A100. You need the new silicon either way.
Putting a Chinese chip in the same official release as Nvidia, with first-class support rather than a side patch, is new. As The New Stack put it, Huawei gets "equal billing" with Nvidia here. Except it isn't quite equal in practice. Every published benchmark in the repo, every throughput and latency curve, comes from the Nvidia path. The Ascend 950 integration code is there and it compiles, but there's no performance data to back it up yet. Deterministic sampling and batch-invariant inference (both useful for reproducible outputs) are Nvidia-only for now too. So: equal billing in the announcement, not yet equal footing in the benchmarks.
| Feature | Nvidia SM100 | Huawei Ascend 950 |
|---|---|---|
| Inference support | Yes | Yes |
| Published benchmarks | Yes | No |
| Deterministic sampling | Yes | Not yet |
| Batch-invariant inference | Yes | Not yet |
| Minimum CUDA/CANN version | CUDA 13 | CANN 9.2 |
The requirements are specific, and if you don't match them, don't bother:
Setup is export DEEPINFER_ACCELERATOR=nvidia (or ascend), then ./build.sh. Once it's built, python examples/offline_inference.py pulls the DeepSeek-V4.1-Flash checkpoint from Hugging Face automatically and runs a sample job. Straightforward if you've got the hardware. Pointless if you don't.
This is the catch, and it's a real one, not a minor footnote. DeepInfer ships the compute engine, not the serving platform. Missing from the release:
Those three pieces are exactly what turns "fast inference code" into "a production API that handles real traffic." DeepSeek's reasoning, reported by SiliconANGLE, is that these components are too tightly wound into their internal infrastructure to pull out cleanly. Maybe. It's also convenient that the part they kept private is the part that's hardest to replicate and most directly tied to running at their specific scale. I don't think that's an accident, and I don't think it should be read as a free pass either way. Withholding infrastructure glue is normal, even for genuinely open projects.
It also only supports one model: DeepSeek-V4.1-Flash. If you're running anything else, DeepInfer isn't an option, full stop. For broader model coverage you're still looking at vLLM or SGLang.
People keep asking whether DeepInfer replaces vLLM. It doesn't, and comparing them head-to-head misses the point.
vLLM, SGLang, and TensorRT-LLM are general-purpose serving frameworks: you can drop almost any open model into them. DeepInfer is co-designed around exactly one model family, running on exactly two hardware targets, for exactly one workload (offline inference). It's narrower by design. What you get in exchange is a direct look at the specific tricks (DSpark, prefill compression, the kernel stack) that let DeepSeek serve V4.1-Flash at a cost per token that undercuts most of the market. TensorRT-LLm still edges out raw throughput on Nvidia hardware in isolation, by something like 10-30% in independent benchmarks, but it takes days to weeks to get a deployment tuned, versus hours for vLLM. DeepInfer sits in its own lane: not a drop-in replacement for either, more a transparency move than a competitor.
Be honest with yourself about which bucket you're in before you spend a weekend on this.
Use it if:
Skip it if:
I'd put DeepInfer in the "fascinating, mostly for infra teams" category rather than "something every developer should try this weekend." It's a genuinely useful release if you're building serving infrastructure at scale and want DeepSeek's actual production tricks instead of guessing at them from benchmark screenshots. For everyone else, it's a very well-documented reminder of why DeepSeek's API is cheap, and that reason turns out to be a lot of unglamorous engineering, not a single magic trick.
If you're trying to decide whether V4.1-Flash itself is worth building on in the first place, our breakdown of DeepSeek V4.1 Flash 1017 covers pricing and benchmarks against the rest of the field. And if cost-per-token is what you're actually optimizing for, check how it stacks up against DeepSeek's Search and Fetch API pricing for the tool-use side of the equation.
The questions people keep asking about DeepInfer, answered straight.
New AI tools, automation workflows, and course drops — straight to your inbox. Join 2,400+ builders.
Mistral's Voxtral 3 adds a LiveKit-based voice agent SDK with tool calling, starting at $0.04 a minute, plus new speech-to-text and TTS models.
tsrs is Cloudflare and VoidZero's Rust port of the TypeScript compiler: up to 8x faster, 70% less memory than TS 7.0. Here's the real story and how to try it.
Mistral's Timestral forecasts time series on a CPU in 10-140ms, no GPU. How to install and run it in Python (5M-209M models), plus the license's $20M catch.
Comments