vLLM vs TGI vs llama.cpp

The five mental models

vLLM is usually the first name people reach for when they want high-throughput serving for open LLMs on GPUs. Its core strength is efficient memory management for the KV cache and serving many concurrent requests.

SGLang is a newer GPU serving runtime built around RadixAttention for prefix caching across requests, strong structured-generation support, and competitive throughput on long shared prefixes. Pick it when many requests reuse the same system prompt, tool schemas, or RAG context blocks.

Text Generation Inference, usually called TGI, is Hugging Face's production inference server. Its strength is the model ecosystem, deployment ergonomics, streaming APIs, batching, and integration with the Hugging Face world.

TensorRT-LLM is NVIDIA's optimized inference stack. Its strength is kernel-level performance on NVIDIA hardware, especially H100 with FP8, and tight integration with TensorRT. Pick it when you need maximum throughput on a fixed NVIDIA fleet and can invest in the build pipeline.

llama.cpp is the local inference workhorse. Its strength is running quantized models on CPUs, Apple Silicon, consumer GPUs, and machines that are not a classic datacenter GPU fleet.

Compare the default fit

This table is intentionally simple. Real deployments have details, but the default fit gives you a good starting point.

Feature matrix

Check the features your product actually needs before picking a winner on a generic benchmark.

SGLang vs vLLM

Both are strong GPU servers with continuous batching and OpenAI-style APIs. The fork is workload shape, not religion.

• Pick SGLang when a large fraction of input tokens are identical across requests: shared system prompts, repeated tool definitions, RAG chunks that many users hit, or agent harnesses with stable prefixes. RadixAttention is built for that reuse.
• Pick vLLM when prefixes are diverse, you need the broadest model and quantization coverage today, or your team already has vLLM deployment playbooks.
• Benchmark both on your real prompt distribution. Prefix-cache wins show up only when reuse is high enough to amortize cache management.

For structured generation (JSON, regex, grammar constraints), SGLang has been a common pick when constraint decoding is on the critical path. vLLM has been catching up fast; treat the matrix as a hypothesis to test, not a verdict.

Match runtime to bottleneck

If your bottleneck is multi-user GPU throughput, start with a runtime built around scheduling and KV cache efficiency. If your bottleneck is getting an open model deployed with fewer sharp edges, a server with strong model ecosystem support can save time. If your bottleneck is shipping a private local assistant, a small quantized model with llama.cpp may beat a server fleet you do not need.

There is no universal winner because the workloads are not the same. A voice agent, coding assistant, mobile summarizer, and offline desktop tool can all be "LLM serving" while needing very different tradeoffs.

Benchmark the same model

Comparisons get messy when each runtime is tested with a different model, quantization, prompt length, output length, batch size, or hardware target. Keep the model and traffic shape fixed before judging the runtime.

At minimum, record time to first token, output tokens per second, total tokens per second across all users, memory use, p95 latency, error rate, and startup time. Also record whether the runtime supports your needed features: streaming, structured output constraints, adapters, tensor parallelism, quantization, metrics, and health checks.

Prefer boring APIs

The runtime is an implementation detail your product should be able to change. Put a stable internal API in front of it. Normalize request fields, streaming events, errors, model names, timeout behavior, and token accounting.

This lets you test vLLM against TGI, move a local feature to llama.cpp, or fall back to a hosted API without rewriting product code. You still need to test behavior, but the blast radius is smaller.

Quick check