An inference engine is the software layer that actually runs a large model: it takes the model weights and user requests, and handles queueing, compute scheduling, memory management and output. Without it, a model is just a pile of parameter files that cannot run; pick the wrong engine and the same GPU can run several times slower and cost several times more. Get one distinction straight first: the model decides what it knows, the inference engine decides how fast and how cheaply it runs.
What the three leading engines actually compute
vLLM, SGLang and TensorRT-LLM are currently the three most widely used open-source inference engines, and they all do the same job: packing the one-token-at-a-time generation process more densely and faster. Two core mechanisms matter. First, continuous batching: new requests no longer wait for the whole previous batch to finish — they join as they arrive, so the GPU stops idling on long tails. Second, PagedAttention, which manages the KV cache the way an operating system manages paged memory, sharply cutting memory fragmentation. SGLang adds RadixAttention on top, reusing repeated prompt prefixes across multi-turn conversations with a prefix tree; TensorRT-LLM relies on NVIDIA's compiler optimizations to squeeze operators to their fastest on its own GPUs.
Three common misunderstandings — stop mixing these up
The first is treating an inference engine as a chat product. It has no interface, no memory and no web search — there is still an application layer between it and the user, so installing one does not give you a conversational assistant. The second is thinking a different engine improves answer quality: the engine only governs speed and cost; with the same weights, answers stay essentially unchanged. The third is believing only huge models need one: once concurrency rises, even a small model can be dragged down by queueing and memory pressure — the engine's value shows as soon as requests pile up.
Which scenarios are worth an inference engine?
If you run your own API service, serve dozens or hundreds of concurrent requests, or want to squeeze several models onto one card, start with an inference engine; if you only run models locally now and then to write code, an out-of-the-box tool like Ollama is less hassle. When choosing, do not look only at benchmarks: first check whether it supports your model format and quantization, then memory footprint and latency, and only then community activity. To go deeper into the mechanics, start with the site's entry on What is continuous batching?, then read the hands-on analysis What teams are vLLMs suitable for?.