AI compute / Inference economics
AI Inference GPU Sizing: Throughput, Latency and Token Cost
Peak tokens per second in a vendor benchmark is not the same as capacity at your latency target or actual cost per million delivered tokens.
Sections in this article
Two GPU vendors advertise 30,000 output tokens per second on different models. One is cheaper per hour. That is not enough information to choose an inference fleet: the tests may have used different prompt lengths, batching, precision, latency requirements and request patterns. A serving platform is purchased to deliver responses within a latency objective, not to produce the highest isolated throughput number.
Useful approach: first benchmark the exact model, runtime and prompt/output distribution at the desired time-to-first-token (TTFT) and inter-token latency (ITL). Measure sustainable aggregate output tokens/second per serving replica at that service level. Then determine replica count and allocate its hourly cost across actual delivered input and output tokens.
Replica sizing and cost-per-million-token estimator
Use measured, latency-qualified aggregate output throughput per full replica. A replica might be one GPU or a tensor-parallel group; its hourly cost must include everything serving that instance. The headroom control reserves capacity for bursts, failures and measurement uncertainty. This tool is a lower-bound capacity screen: prefill throughput, KV cache, routing, model sharding and request variability may require more replicas.
Compute-only serving approximation; validate real request mix and p95/p99 latency.
Cost per million total tokens and cost per million output tokens are different metrics. The estimator uses input + output tokens in the denominator, assuming a per-replica hourly charge; it does not apply separate input/output pricing. To compare against an API tariff, convert all bids to the same token mix and include the time period that hardware sits idle.
Measure throughput only at an acceptable TTFT and ITL
Measure time from request submission to first returned token. Longer prompts, queuing and prefill compute usually change the value. Report p95 and p99 as well as median.
Measure spacing between generated responses, not simply total tokens divided by wall-clock time. Request batching and scheduler contention can alter tail behavior.
Report total output tokens/sec while TTFT and ITL targets are met at the specified prompt distribution and concurrency—not an unloaded synthetic peak.
NVIDIA's GenAI-Perf reports token throughput, TTFT, ITL, request throughput and latency percentiles. Make test input length, output length, concurrency, streaming mode, precision, cache policy and model version explicit. The number to put into the calculator is throughput measured under the intended production quality constraint. If the benchmark shows 1,800 tok/s but only 900 tok/s at your p99 latency objective, budget from 900.
Memory fit and concurrency can override the calculator
Model weight size provides a first-order memory estimate: parameter count × bits per stored parameter / 8. A dense 70-billion-parameter model at 16 bits nominally needs about 140 GB for weights alone, before runtime buffers, KV cache, fragmentation and other allocations. Quantization and tensor parallelism change that requirement. Weight arithmetic cannot establish serving capacity by itself.
KV cache grows with concurrent sequence length and model architecture. A larger context window or greater active batching can hit the memory ceiling long before a nominal tok/s ceiling. Request count, GPU-memory residency, queueing and the mix of prompt and generated tokens should all be profiled. Prefill is not always exclusively compute-bound and decode is not universally limited only by memory bandwidth; hardware, batch size and model architecture matter.
| Benchmark field | Evidence to request | What it catches |
|---|---|---|
| Model and precision | Exact model revision, quantization and serving engine | Non-equivalent results |
| Load profile | Input/output length histogram, arrivals, concurrency | Unrepresentative test data |
| Performance | TTFT/ITL p95–p99, served tok/s | Throughput gained at unacceptable latency |
| Memory | Peak allocated memory and KV-cache pressure | Out-of-memory failures under long context |
| Economics | Power, hourly hosting charge and fleet availability | Underestimated delivered-token cost |
Illustrative sizing: 3 requests per second
At 250 generated tokens per request, demand is 750 output tokens/sec. Suppose one replica sustainably delivers 900 output tokens/sec while satisfying the latency target. Reserving 30% headroom limits planning throughput to 630 tokens/sec per replica; two replicas are the preliminary result.
The example is arithmetic using hypothetical inputs, not an H100 or any other GPU price quotation. If arrival rates are bursty, average RPS is insufficient for p99 latency sizing.
For a cloud rental, use billable instance hours even when traffic is low. For owned equipment, include depreciation, service, facility electricity and software licenses, but avoid charging electricity twice if the hourly replica figure already includes it.
A procurement test that avoids benchmark theater
- Define the model, target quality and p95/p99 TTFT and ITL before discussing GPU models.
- Replay realistic prompts and output lengths at low, average and burst load; do not rely only on a fixed single batch size.
- Measure achieved throughput and rejected/timed-out requests at the service objective.
- Check model-weight memory, KV cache, replica failover and cold-start time alongside throughput.
- Calculate effective cost per million delivered input/output tokens under the same time and energy accounting boundary.
Serving design should be revised when model versions, quantization or prompt distributions change. A transparent benchmark, rather than a universal token-speed claim, is what makes this estimate useful.
Sources and further reading
Primary frameworks and reference materials. Engineering decisions require site-specific validation; legal classifications require assessment of the actual use case.
- GenAI-Perf metrics and benchmarkingNVIDIA Triton Inference Server
- GenAI-Perf usage and concurrent workload examplesNVIDIA open-source performance analyzer