Infrastructure commissioning / Handover
GPU Cluster Commissioning: Go-Live Testing Checklist
A rack can pass electrical and cooling acceptance yet fail a distributed training job. This checklist joins facilities testing to actual compute readiness.
Sections in this article
A deployment report says the UPS, CDU, and leaf switches have all passed their separate tests. The new GPU cluster then spends most of its first training job waiting on one faulty interconnect. The facility is technically available, but the promised compute capacity is not. This is the gap between equipment commissioning and workload-ready cluster acceptance.
For a useful go-live decision, prove three different things: the facility can safely support the intended load; the installed accelerators and network can execute the intended workloads; and operations teams know how to detect, isolate and recover from a problem. Each claim needs evidence from its owner.
Five gates between construction and production
Match installed node, GPU, NIC, firmware, switch, power feed and cooling connection to the approved design.
Verify power capacity, cooling duty, measurement points, local alarms and protections under the agreed duty cycle.
Run supported diagnostics for accelerators, memory, PCIe, NVLink and software stack while nodes are reserved.
Prove communication correctness, representative collective throughput and training throughput at scale.
Demonstrate isolation, operator escalation, rollback and ownership of every remaining exception.
ASHRAE's AI data center commissioning guidance distinguishes subsystem functional checks from Level 5 integrated testing. A GPU commissioning plan should build on those results rather than repeat invasive pressure or electrical failure tests that have already been completed and signed off.
A compact acceptance record for every test
Write the expected outcome before the first test. A record needs the configuration being tested, the responsible person, a pass criterion tied to vendor or application requirements, time stamps, measured outcome, deviations, links to logs and the retest decision. Without these items, a green icon may be impossible to reproduce after the next driver upgrade.
| Acceptance area | Representative test | Evidence to retain |
|---|---|---|
| Power | Observe actual or approved simulated training load changes on A/B feeds | IT load trace, EPMS events, UPS alarms, headroom |
| Cooling | Sustained representative GPU work at expected operating conditions | Inlet temperatures, liquid flow/temperatures, throttle flags |
| Single node | Supported DCGM diagnostics with the right node reservation | Run version, device inventory, severity and diagnostic log |
| Fabric | Collective communication test on intended topology | Correctness, achieved bandwidth, topology, retransmissions |
| Application | Representative short training or inference batch | Useful throughput, latency and errors against target |
| Recovery | Controlled node quarantine and job restart exercise | Detection, incident ticket, recovery time and sign-off |
Pass limits must come from project requirements and tested hardware, not from a universal rule such as “all links must run a 24-hour bit-error test” or “every load bank must step to 100% in five milliseconds.” Those might be chosen project tests, but unvalidated test conditions can themselves put equipment at risk.
Use workload-shaped tests rather than isolated green dashboards
Begin with node inventory and supported diagnostics. NVIDIA's DCGM documentation distinguishes active diagnostics from passive health monitoring. A health dashboard with no alarms does not prove that memory, power, PCIe and GPU collectives will withstand a representative workload. Conversely, a diagnostic can fail because of software permissions or conflicting work, not necessarily damaged hardware.
Diagnostic tests can load GPUs, memory, network and cooling infrastructure. Coordinate with the scheduler and only run the approved scope against resources taken out of service. Record DCGM version, selected tests and device coverage, because support varies by hardware and release. A command copied from a different cluster is not automatically safe or complete.
Next validate the intended multi-node topology. A single-node test can pass while a bad cable, misplaced NIC or congested fabric slows the job. Use the appropriate supported multi-node diagnostic or application benchmark with correctness checking. Compare both median and slowest-node behavior; one straggler can dominate synchronized collective completion time.
The engineer checks fabric error counters and topology mapping, swaps or isolates the suspect path, then reruns the same collective on the same node set. The original result is preserved as a failure; the later pass is documented as a retest, not used to erase the incident.
Meanwhile facilities specialists correlate EPMS and cooling telemetry with the test's time window. That correlation is useful only if clocks and measurement points are known. A thermal alarm reported after a training step should be investigated with sensor latency in mind, not automatically attributed to the model or workload scheduler.
What happens when the result is not a clean pass?
A failure should open an action with a clear owner, risk classification and retest requirement. Do not turn a mixed result into “accepted with notes” without a decision by the project authority. Hardware faults that affect data integrity, inadequate cooling under design load, unresolved electrical protection issues or loss of required redundancy can warrant a no-go decision.
All safety-critical gates pass; measured application results meet the approved acceptance envelope.
Only bounded noncritical exceptions remain, with documented restrictions, owners and a deadline.
Unresolved protection, thermal, integrity or recoverability issues; no workaround is assumed safe.
If only part of a cluster is accepted, record the precise boundary: rack IDs, servers, switch ports, thermal zone and approved maximum workload. A limited pilot can run within an accepted envelope, but it cannot be silently counted as acceptance of the full intended configuration.
Make operations ownership part of the test
Before releasing production scheduling, run a short operator walkthrough. Who receives a cooling alarm while the GPU team is offline? Who can drain a faulty node in the scheduler? Who decides whether an unexplained spike warrants pausing jobs? Where are the baseline measurements and rollback steps stored?
The final record should link commissioning reports, equipment inventory, software versions, benchmark conditions, training logs, vendor acceptance documents, contact roster and open exceptions. A signed handover needs the authorized facilities, IT and project owners—not simply the last engineer to run a test.
Commissioning is complete when the team can demonstrate the promised service safely, explain its known limits and take responsibility for recovery. That outcome matters more than the number of checkboxes on the original installation spreadsheet.
Sources and further reading
Primary frameworks and reference materials. Engineering decisions require site-specific validation; legal classifications require assessment of the actual use case.