AI networks / Physical-layer design
GPU Cluster Network Cabling: Fiber, Optics and Costs
Before comparing fiber prices, count the actual network ports, verify link lengths and know what each optic draws. That is where cabling budgets go wrong.
Sections in this article
A cluster procurement spreadsheet has a line for switches and another for network adapters. The optics and cabling line still says “TBD.” At 400G and 800G, that omission can affect budget, lead time, power distribution, cooling and the entire physical layout. The useful starting point is not a fiber preference; it is an end-to-end link inventory.
Do this first: list every server-to-leaf and leaf-to-spine port, the expected path length, the exact supported transceiver form factor and which side supplies each optic. Two identical 400G ports are not proof that two modules with different signaling or connectors will interoperate.
Copper, multimode or single-mode: match the interface
| Medium / solution | Typical engineering use | Important question |
|---|---|---|
| Passive DAC copper | Short, supported links in or near a rack | Is the exact cable and length approved by both endpoints? |
| Active copper / AEC | Longer copper reach where supported | How much equalization power and space does the assembly need? |
| Active optical cable (AOC) | Preterminated optical link with integrated endpoints | Can it be repaired or must the entire assembly be replaced? |
| Multimode OM4 / OM5 | Supported short-reach optics in a data hall | Does the particular 400G/800G optic meet the distance and connector spec? |
| Single-mode OS2 | Longer reach and many structured cabling use cases | Which optic type, fiber count and loss budget are required? |
There is no universal rule that 45 meters requires single-mode fiber. Cisco's 400G QSFP-DD portfolio lists specific multimode options supporting tens of meters and other options designed for 100 meters, while 400G DR4 single-mode supports 500 meters and FR4 can reach 2 kilometers. The usable distance comes from the exact module and fiber specification, not just the headline 400G rate.
Connector count also matters. Parallel DR4 and SR-style optics can use multi-fiber MPO arrangements, whereas FR4 designs can use duplex LC interfaces. Mixing those assumptions in a bill of materials results in missing fiber pairs and wrong patch panels.
Worked example: 64 servers with two 400G ports each
Consider a simplified, illustrative AI cluster with 64 servers. Assume each server has two 400G external fabric ports, and both ports terminate on leaf switch ports. That means 128 physical server-to-leaf links. If both ends use pluggable optical modules, the links require 256 transceiver endpoints. This example does not assume that each GPU requires an individual physical port.
If, only for budgeting, each module consumes 9 W at the relevant load, those endpoints consume 2,304 W (2.304 kW) combined. That is not a vendor claim for every transceiver. Cisco's published 400G QSFP112 module data show models with a maximum power rating around this level, but another form factor, generation or vendor can differ. A switch's thermal budget must account for what its chassis specification includes.
Before leaf/spine uplinks, switch electronics, cooling overhead and any copper-link substitutions.
Do not forget the leaf-to-spine links: their count depends on topology, port speeds, oversubscription and redundancy requirements. Build a separate bill for them. “512 GPUs” by itself does not determine a valid link count.
Check the optical link budget before calling the cable good
A short physical path can still fail if connectors are contaminated, excessive patch points are introduced or the wrong optic and fiber pair are selected. The loss budget should account for fiber attenuation, connectors, splices, engineering margin and the module's published transmit/receive requirements. Use the vendor's supported reach as a necessary screen—not proof that every installed path passes.
Before plugging in a fiber, use qualified inspection tools and cleaning procedures. IEC 61300-3-35:2022 specifies visual inspection of connector end faces. The standard explicitly notes that visual inspection does not replace attenuation or return-loss testing. Include both in your acceptance plan where relevant.
A procurement bill that avoids hidden optic costs
The purchasing list should have at least six groups: supported transceivers (both ends), cable assemblies or structured fibers, patch panels/cassettes, installation labor, spare modules and validation equipment. Also budget electrical power, spares inventory and fiber tray capacity. Confirm whether an AOC failure means replacing a whole cable and whether third-party optics are supported under the network equipment warranty.
The final physical-layer handover
Save the as-built link map, optic model and firmware, connector cleaning/inspection reports, attenuation testing where specified, optical diagnostics, error counts and a baseline under representative workload. Clean connector faces and replace damaged parts before trying to mask physical errors with congestion tuning.
Decision: choose the lowest complete lifecycle-cost media that meets the exact interface, reach, reliability and refresh requirements. It may be copper, multimode or single-mode depending on the actual links. A one-size-fits-all cabling rule is not an engineering design.
Sources and further reading
Primary frameworks and reference materials. Engineering decisions require site-specific validation; legal classifications require assessment of the actual use case.