Network operations / Troubleshooting
GPU Network Troubleshooting: BER, FEC and Congestion
A slow collective is a symptom, not proof of dirty optics. Use counter deltas and a short isolation sequence to find the failing layer.
Sections in this article
One rank in a 512-GPU training run falls behind during an All-Reduce. The dashboard shows a rising pre-FEC counter on one port, while another leaf switch shows a burst of PFC pause frames. Replacing optics across the data hall would be expensive and may fix nothing. The investigation must separate signal integrity, queue congestion, fabric configuration and host/application behavior.
First rule: compare error rates and counter changes over a measured time window, not isolated cumulative counter totals. Know which port and direction the measurement represents, the active link speed/FEC mode and the vendor's documented threshold. There is no universal pre-FEC BER number that makes every 400G or 800G link safe or unsafe.
Start with the observable signature, not an assumed root cause
Compare pre-FEC BER trend, corrected/uncorrected FEC, link flaps, optical power and temperature.
Review occupancy, ECN marks, PFC pause duration, output discards and active routes.
Verify negotiation, FEC compatibility, port speed, MTU, VLAN/partition and firmware parity.
Check process timing, GPU utilization, PCIe/NVLink errors, CPU, dataloader and congestion outside the measured fabric.
| Observed pattern | Initial hypothesis | Next corroborating measurement |
|---|---|---|
| Uncorrectable FEC and link resets | Possible signal integrity or hardware defect | Per-lane BER, optics diagnostics, connector state, replacement test |
| Corrected FEC increases; no service symptoms | Potentially normal operation or degrading margin | Rate relative to vendor limits and historical baseline |
| High ECN and PFC under bursts | Congestion or head-of-line effects | Per-priority queue fill, duration, incast source, offered load |
| Link negotiated below expected speed | Configuration, cable/module or physical limitation | Port speed, FEC mode, lane width and supported transceiver matrix |
| All NIC counters healthy, one slow rank | Application, PCIe, storage or scheduling bottleneck | Profiler, host counters, storage latency and GPU step timing |
BER and FEC: do not confuse correction with outage
Pre-FEC BER describes errors before forward error correction. Corrected FEC counters show that the code is repairing received errors; the expected rate varies with line encoding, optical/electrical design and FEC scheme. Uncorrectable FEC generally warrants closer investigation because affected data cannot be repaired by that layer. Still, the consequences depend on link protocol and whether the system recovers or retransmits through its own mechanism.
A single pre-FEC BER threshold such as 10⁻⁵ should not be applied to all links, and a rise in corrected FEC is not on its own proof that training stalls. Verify lane-specific specification and vendor thresholds. Capture rates over a representative interval and compare with neighboring healthy links of the same module type under similar thermal load.
mlxlink -d DEVICE -p PORT -cNVIDIA's mlxlink documentation identifies -c as the query for physical counters and BER information. Replace DEVICE and PORT with values discovered on the local system, and check tool/adapter compatibility. This is a diagnostic example, not a configuration-changing command.
Digital optical monitoring—temperature, laser bias, transmit and receive optical power—may help localize deteriorating links, but readings must be compared against the module's own acceptable range. A clean-looking module is not proof of a healthy fiber path; likewise, a dirty connector does not prove an unrelated queue-overflow symptom. Physical cleaning requires approved laser safety, inspection practices and a maintenance window where necessary.
PFC, ECN and oversubscription are different signals
When offered traffic exceeds egress capacity, buffers can accumulate packets. Ethernet fabrics using RoCE may deploy explicit congestion notification (ECN) and priority flow control (PFC), while InfiniBand uses different flow-control behavior. A high PFC count is not automatically a “lossless network success”: excessive pause durations can spread congestion or interfere with other traffic. The same network can experience congestion without any optical error.
Measure ingress/egress bandwidth and burst duration at the actual leaf, uplink and receiver. Look for incast and simultaneous checkpoint writes.
Track buffer occupancy, ECN marking, PFC events/duration and discard counters per priority and per port.
Compare ECMP hashing, adaptive routing, the negotiated topology and the intended oversubscription ratio under real collectives.
A 1:1 aggregate downlink-to-uplink ratio does not guarantee zero contention: synchronized many-to-one patterns and uneven path selection can congest individual egress ports. Do not change PFC or ECN thresholds on a production fabric solely because a graph looks high. Benchmark and stage every change, and retain a rollback configuration.
A five-step investigation that preserves evidence
- Correlate the slow job and time window. Identify rank, host, job phase and first symptom. Record network changes and workload placement.
- Map the physical path. Identify NIC, port, switch hop, optic and network partition. Confirm actual negotiated width, speed and FEC configuration.
- Capture counters twice. Compare optical/FEC deltas and congestion/queue deltas across the same time interval. Use comparable baseline and unaffected ports.
- Isolate one variable. If physical errors localize to a path, inspect connector and module under procedure. If queue signals dominate, test workload distribution or routes in staging. If neither appears, profile the host and application.
- Retest and restore. Repeat the same collective, duration and load after intervention. Save logs and explicit before/after metrics, then close the ticket with a falsifiable root-cause statement.
A supported port diagnostic is preferable to speculative commands copied across different NIC generations. Some hardware exposes BER and FEC in separate utilities; some counters roll over or reset on a port flap. Preserve counter reset times when calculating rates.
A useful incident record is a comparison, not a narrative
| Field | Example value type | Why it matters |
|---|---|---|
| Symptom | All-Reduce p99 duration by job and rank | Defines the measurable performance impact |
| Link | Host/NIC/port, leaf port, optic SKU and firmware | Makes the failure reproducible |
| Counter window | Before and after values plus interval | Prevents interpreting lifetime totals as active faults |
| Change applied | Connector reinspection, tested module swap or staged network config | Separates hypothesis from intervention |
| Exit criteria | Vendor error limits and restored application benchmark | Shows the fix addressed the actual symptom |
Network teams should not relabel every slow training step “a fabric problem.” When FEC, queue and speed measurements remain normal, move to CPU scheduling, storage, inter-GPU links and application profiling. The purpose of the runbook is to find what changed and what evidence rules competing explanations out.
Sources and further reading
Primary frameworks and reference materials. Engineering decisions require site-specific validation; legal classifications require assessment of the actual use case.
- mlxlink physical counters and BER command referenceNVIDIA Networking Documentation
- GPU inference and model benchmarking telemetryNVIDIA Triton Inference Server