AI infrastructure / Training data pipeline
AI Training Storage: Capacity, Throughput and Checkpoints
The storage bottleneck depends on the data loader, cache and checkpoint strategy—not just the number of GPUs. Calculate three requirements separately.
Sections in this article
A storage vendor quotes 80 GB/s sequential throughput for a prospective 512-GPU cluster. The training team asks whether that is enough. The number cannot answer the question until the operator knows how data is sharded, how much the loader requests per step, what is already cached, and whether checkpoint writes compete with reads on the same storage system.
Start with three separate budgets: training-data read rate, checkpoint write rate and available capacity. A 500 TB array might meet the space target while missing its small-file rate. An all-flash system might be unnecessarily expensive if the active working set can be staged or cached. Neither observation means NFS, object storage or any specific parallel file system is inherently unsuitable.
Interactive AI storage sizing worksheet
The read estimate assumes all samples in a global step have the entered average size, and only the entered fraction must be fetched from the tier being sized. Cached data, transformations, prefetching and compression affect the true rate. The checkpoint estimate assumes a single simultaneous save of the entered size. Units are decimal MB, GB and TB.
Benchmark with a representative dataset; this model does not account for RAID, erasure coding, metadata operations or competing users.
The data-loader path is rarely a constant stream
Consider 4,096 samples of 4 MB processed each half-second. If 70% of the requested bytes are served by host, local SSD or other caches, the external tier must supply the remaining 30%. The simplified throughput requirement is (4,096 × 4 MB ÷ 1,000) ÷ 0.5 s × 0.30 = 9.83 GB/s. With no effective cache the same input would demand 32.77 GB/s from that tier.
This estimate describes average data-rate demand implied by the selected assumptions, not a guaranteed sustained benchmark. A dataset with millions of tiny files can be slowed by metadata lookups long before hitting that throughput. Video decoding and data augmentation may shift the bottleneck to CPU or GPU preprocessing. If data is generated or streamed directly in memory, the storage read rate may be lower than a naive per-step calculation suggests.
Holds the authoritative dataset. Track actual read bytes and metadata request patterns.
Buffers hot working sets, with space and refill constraints of its own.
Verify decode, prefetch and GPU waiting time with application-level profiling.
Before increasing flash budget, investigate whether access is sequential, shuffled, compressed, partitioned by worker or reusing a local cache. Switching storage technology will not repair an inefficient data-loader or slow preprocessing stage.
Checkpoint bandwidth is a second, independent bottleneck
A checkpoint can store model weights, optimizer state and other restart metadata. Its size varies with framework, precision, sharding, optimizer and configuration—not merely parameter count. If a 900 GB distributed checkpoint must finish in 180 seconds, required end-to-end writing throughput is 5 GB/s before protocol or safety headroom.
Do not assume the full training cluster always blocks synchronously until the complete checkpoint reaches durable storage. Some systems use asynchronous writing, local staging or overlapping transfers. In those cases, the write window and storage placement should represent the actual recovery-point objective and amount of memory or local disk needed while data moves in the background.
Test the actual checkpoint path, including flush, consistency and durability completion. A write to a volatile cache is not the same as a recoverable checkpoint.
For capacity, preserve enough full or incremental checkpoints for your retention policy. The calculator adds the selected checkpoint copies to the active dataset and then applies a capacity planning headroom. It deliberately does not derive raw drive count, since replication, erasure coding, filesystem reserves, snapshots and failure-domain placement are product-specific.
Select storage based on its measured failure mode
| Workload finding | Promising approach | What to validate |
|---|---|---|
| Large, sequential scans | Scale-out shared storage or efficiently staged object data | Sustained throughput and cost per delivered GB/s |
| Many small random files | Dataset packing, metadata scaling or client-side caching | Open/stat rate, average latency and worker scaling |
| Repeated epochs | Local NVMe staging or persistent read cache | Warm-cache fraction, preload time and eviction |
| Checkpoint bursts | Parallel write targets or local stage and flush | Checkpoint completion, restart and failure durability |
| Long-term archive | Object storage and lifecycle policy | Retrieval delay, protection, egress and storage costs |
Parallel file systems such as Lustre or GPFS, modern scale-out NAS and object-based designs can all support AI use cases under suitable configurations. NVIDIA GPUDirect Storage provides a direct data path for compatible devices and file systems, reducing CPU bounce-buffer overhead in supported scenarios. It does not mean every GPU training deployment requires NVMe-oF or that any RDMA network automatically bypasses all CPU software involvement.
Raw advertised bandwidth is frequently measured with large transfer sizes and favorable client concurrency. Ask how performance changes with your file count, average sample size, concurrency and access sequence. Check failure handling, client drivers, security, restore time, power and software licensing alongside throughput.
A repeatable acceptance test before ordering storage
- Capture a representative ten-minute or longer training trace, including I/O sizes, read patterns and cache hits.
- Run a cold-cache and warm-cache test; publish the gap instead of using one flattering result.
- Run real checkpoints while training reads continue; verify restart from completed artifacts.
- Increase concurrent workers until throughput plateaus and log GPU wait, metadata response and network utilization.
- Calculate a five-year cost model for capacity, performance licensing, optics, protection, maintenance and power.
A convincing architecture is one in which the slowest operation has been demonstrated, not merely predicted. The three calculator results are a useful shortlist filter; production approval depends on end-to-end measurements under an application workload.
Why storage sizing cannot be certified from a calculator
Capacity planning is not a safety certification or a guarantee that a dataset will sustain the required throughput. Compression ratios, small-file metadata calls, simultaneous tenants, network packet behavior, redundancy and drives under rebuild can dramatically alter observed service. A successful vendor demonstration should disclose the client count, test duration, working set, failure conditions and durability boundary. Validate those results with the same loader and checkpoint workflow that the cluster will run after handover.
Sources and further reading
Primary frameworks and reference materials. Engineering decisions require site-specific validation; legal classifications require assessment of the actual use case.