Measure LLM inference latency

The latency metrics glossary

These names show up in benchmarks, dashboards, and vendor docs. Use them consistently.

• TTFT (time to first token): wall-clock from request accepted to first output token available. Dominated by queue time plus prefill for long prompts. Measure at the client or API gateway if you care about user-perceived wait.
• TTFB (time to first byte): HTTP/web term for when the client receives the first byte of the response body. In streamed LLM APIs, TTFB often tracks TTFT, but it also includes TLS, proxy buffering, and SSE framing. Use TTFT for GPU and scheduler tuning; use TTFB when debugging gateways and CDNs. The Latency L05 lesson covers where they diverge.
• TPOT (time per output token): average decode time per generated token after the first one, in milliseconds per token. Roughly (total_latency - TTFT) / (output_tokens - 1) when output length > 1. This is the inverse of tokens per second for the decode phase.
• ITL (inter-token latency): the gap between consecutive streamed tokens as the user sees them. Similar to TPOT but measured per chunk on the wire; includes batching/scheduling jitter. Bad ITL feels like stuttering even when average TPOT looks fine.
• E2E (end-to-end latency): total time from send to final token (or connection close). Queue + prefill + full decode + any post-processing.

Log all four plus input/output token counts. TTFT points at prefill and queueing. TPOT and ITL point at decode and the scheduler. E2E is what product SLOs usually cite, but you cannot tune from E2E alone.

Use percentiles, not averages

Average latency hides the requests users complain about. In serving, the painful behavior lives in the tail: p95, p99, and the worst cases around traffic spikes or giant prompts.

A voice agent might feel good at p50 and broken at p95. A coding assistant might handle short prompts well and collapse on long files. Tail latency is where capacity problems show up first.

Benchmark with realistic shapes

A benchmark with 128 input tokens and 32 output tokens tells you almost nothing about a RAG support assistant that sends 6,000 input tokens and asks for 500 output tokens. The workload shape has to match the product.

Useful benchmark cases include short prompt and short output, long prompt and short output, short prompt and long output, and long prompt and long output. Add concurrency levels that reflect real traffic. Then record how TTFT, TPOT, ITL, tokens per second, memory, and error rate change.

For production confidence, run load tests that ramp concurrent users while holding prompt/output shape fixed. A single cold request on an idle GPU is a sanity check, not capacity planning. Watch p95 TTFT and p95 ITL under load; those are the numbers that break chat UX when traffic spikes.

Separate throughput from latency

Throughput asks how much work the system completes per second: requests per second or tokens per second across all users. Latency asks how long one user waits. You need both.

Batching can improve throughput while hurting individual latency if requests wait too long to join a batch. Low latency can also waste hardware if the server refuses to batch enough. This is the central serving tradeoff. The Latency & Throughput course covers continuous batching, chunked prefill, and scheduler knobs that move TTFT and ITL in opposite directions.

A practical trace shape

For each LLM call, log a compact event with request class, model, input tokens, output tokens, queue time, TTFT, decode duration, total latency, finish reason, and error status. If privacy rules allow it, store prompt categories or feature names, not raw user text.

That trace lets you answer real questions: Did latency rise because prompts got longer? Did a new prompt template double output length? Did tail latency spike only when concurrency rose? Did errors cluster around max context?

Quick check