Reliability / Training recovery
GPU Cluster Failure Recovery: Checkpoints, Time and Cost
A checkpoint is insurance against lost training progress, but saving one costs compute time. Choose the interval from measured failure rates.
Sections in this article
A long-running training job saves its weights and optimizer state every six hours. A transient node failure arrives just before the next save, so almost six hours of progress must be repeated. Setting checkpoints every five minutes looks safer, but it can consume a substantial share of available storage bandwidth and computing time. The right interval depends on how often the job fails, how long a checkpoint takes and whether saves interrupt useful training.
Planning rule: the first-order Young/Daly approximation is a trade-off, not a mandatory universal cadence. The classical model balances checkpoint overhead against expected work lost between failures. Newer asynchronous saves, incremental checkpoints, correlated failures and shared storage contention require measured validation.
Checkpoint frequency and wasted GPU-hour estimator
Enter the duration of a full blocking checkpoint, the mean time between interruptions of the training job (not individual GPU nameplate MTBF), the fixed restart/restore delay, and effective GPU-hour cost. The estimate assumes independent random failures, checkpoint intervals longer than checkpoint duration and a stationary workload. If your save is asynchronous, its effective blocking cost may be substantially lower than total write time.
Simplified stationary-failure model; not a predictor of actual outages.
Why more frequent saving is not always better
C = blocking save time; M = job mean time between interruptions, in the same units. This is the first-order Young/Daly spacing model under its assumptions.
How much time is consumed by synchronous checkpoint operations across a long run.
With uniformly distributed interruption points, about half the checkpoint interval of useful progress is lost at each failure.
The calculator also adds fixed restore time divided by M as a rough unavailable-capacity share. This is a first-order sum of costs; it becomes less accurate when events overlap, interruptions are correlated, throughput degrades between failures or the waste fraction is large. Checkpoint periods shown are intervals between starts; the useful compute interval depends on the actual save strategy.
With a 5-minute blocking checkpoint and 24-hour job MTBF, the approximate interval is 120 minutes. The two core terms are each about 4.17%: one for saving and one for expected repeated work. A separate 45-minute recovery adds about 3.13% in this simple model. Your real job's measurement should replace those assumptions before accepting a saving policy.
Build a checkpoint that can actually restore
Measuring only write completion is insufficient. A usable checkpoint must represent a consistent training state and survive failures in the storage path. Restore testing should verify model weights, optimizer state, scheduler state, data-loader offsets or reproducible input position, and distributed framework requirements. A checkpoint “saved” according to a log but never restored during commissioning is still unverified protection.
| Recovery stage | Evidence to collect | Common failure |
|---|---|---|
| Detection and job stop | Failure timestamps and failed rank/node | Repeated attempts against a bad node |
| Scheduler and replacement | Assigned resources and requeue delay | No compatible spare capacity |
| Checkpoint integrity | Manifest, checksums, committed version | Partial save selected as latest |
| Distributed restore | Weights, optimizer, RNG/data state recovered | Job resumes from an inconsistent state |
| Progress validation | Completed step and loss curve after restart | Job restarts but produces no usable progress |
Some frameworks support elastic recovery with fewer nodes; others cannot resume without the original parallel topology or compatible tensor-sharding layout. Do not promise instant node replacement or automatic fabric reconfiguration for every training stack. Run an actual restore on a non-production replica and document the achieved RTO.
Avoid counting GPU-hour expense and energy twice
An example job using 512 GPUs at $2.50 per GPU-hour has a fully allocated compute rate of $1,280/hour. If one failure forces roughly 60 minutes of repeated progress and a 45-minute restart, that is about 105 minutes of capacity that did not create new progress, or approximately $2,240 per event. This excludes the periodic checkpoint writes, which the calculator's broader modeled share includes.
If you use a cloud hourly rental that includes power and facility cost, adding a separate “wasted electricity bill” would double count. If the rate is internal hardware depreciation only, disclose electricity and operations separately. Also separate opportunity cost from actual additional cash outlay.
Job failure rate is not necessarily the inverse of the sum of GPU component MTBF values. Application restart behavior, redundancy, fault correlation and the actual dependency graph influence the effective job failure rate. Base M on job histories when available; use component failure data only as an initial hypothesis.
An evidence-driven recovery acceptance runbook
- Establish a measured checkpoint duration under real network and storage concurrency, including tail time and effective training pause.
- Collect interruption timestamps for the job and calculate the observed interval between incidents; separate planned pauses and transient retries.
- Save checkpoints with integrity metadata and a committed-pointer scheme that cannot select half-written state.
- Test restore, verify resumed model progress and measure end-to-end time until the job again makes useful training steps.
- Compare at least three candidate intervals against observed storage contention, recovery behavior and consumed GPU-hours; change the policy only when measurements support it.
A good checkpoint strategy is one that can be explained to both the training team and facilities finance using the same measurement log—not a blanket “every 30 seconds” or “every hour” rule.
Sources and further reading
Primary frameworks and reference materials. Engineering decisions require site-specific validation; legal classifications require assessment of the actual use case.
- Checkpoint optimization and Young/Daly tradeoffsFuture Generation Computer Systems (2024), OSTI manuscript
- GenAI performance benchmarkingNVIDIA Triton Inference Server