Networking and optics

Training a large model is a conversation among thousands of GPUs. At every step they exchange the results of their work, and none can move on until the slowest exchange is finished. The network that carries that conversation decides how much of a cluster's arithmetic turns into a trained model, and it is a large and growing share of what a cluster costs.

This piece explains the hierarchy that rules cluster design, the copper and fibre that make it up, the contest between the two main network technologies, and why optics have become one of the most closely watched parts of the supply chain.

The pyramid that rules cluster design

Data moves at very different speeds depending on where it travels. Fastest of all is the high-bandwidth memory on the GPU package itself. Next is the copper domain inside a rack, where GPUs talk to one another directly. Slowest, for each GPU, is the fibre network that joins racks into a cluster. Each step down the pyramid gives up a large share of the bandwidth.

Cluster architecture therefore comes down to one rule: keep the traffic as low in the pyramid as possible. The ways of splitting a model across GPUs map onto it directly. The chattiest, which split individual layers or route work between experts in a mixture-of-experts model, stay inside the copper domain. The bulkier, less frequent exchanges, which pass work along a pipeline or average results across copies of the model, cross the fibre.

That rule is the whole strategic logic of NVIDIA's rack-scale systems. By enlarging the copper domain from a single server to a whole rack, they pull the most bandwidth-hungry traffic off the network, which is why rack-scale systems train large mixture-of-experts models much faster than the same chips wired as separate servers.

Scale-up: copper inside the rack

Inside the rack, NVIDIA's NVLink behaves more like shared memory than a network. GPUs read and write directly into one another's memory, with no packets and no software stack in the way. Switch trays in the middle of the rack give every GPU a full-speed path to every other, over thousands of copper cables in the rack's backplane.

Copper wins on every count over a couple of metres: it needs almost no power for conversion, it costs little, and it has no transceivers to fail. Its limit is physics. As signalling speeds rise, the distance copper can carry a clean signal shrinks towards a metre. That is why the rack became the unit of computing, and why the coming generations pack more GPUs into one chassis rather than stretching the cables.

Scale-out: the fabric between racks

Between racks runs a lossless network in which every GPU has its own network card, so that data can move straight from one GPU's memory to another's. Each new GPU generation has come with a faster card, and each step up in speed raises the cost of the fabric.

Two technologies compete for this fabric. InfiniBand, now sold by NVIDIA, is the incumbent for the largest training runs: its transport never drops packets, it routes around congestion, and its switches can perform some of the arithmetic of combining results inside the network, which cuts the traffic needed. Ethernet has closed much of the gap. Tuned implementations, NVIDIA's own Spectrum-X and platforms built on Broadcom's switch chips, now deliver comparable effective bandwidth, and the Ultra Ethernet Consortium has standardised the modern feature set. Hyperscalers and cost-conscious neoclouds lean towards Ethernet for its ecosystem and multi-vendor pricing; customers who care most about time to performance still tend to choose InfiniBand.

The layout is standardised pragmatism: a rail-optimised fat tree, in which matching GPUs in each server connect to the same switches, so that the commonest exchanges cross the fewest hops.

Optics: where the money and the watts went

Beyond a couple of metres, every link is light. A pluggable transceiver at each end of a fibre converts electrical signals into laser light and back. The traditional design includes a digital signal processor that cleans the signal and draws considerable power. Each GPU drives several transceivers, across the training fabric, the front-end network that connects the cluster to the outside world and the storage network. Multiplied across a cluster, optics take a noticeable share of both its power and its capital cost, and transceivers fail often enough to create a steady maintenance load.

Two responses are scaling. Linear pluggable optics remove the signal processor where the distance allows, which saves power. Co-packaged optics go further and move the conversion into the switch package itself, removing thousands of pluggable modules and the failures that come with them. NVIDIA has begun shipping co-packaged switches, and the switch end of each link is moving first; the network cards in the servers remain pluggable for now.

Fibre itself

The fibre has changed too. Older datacentres used multimode fibre, with a wide core in which light takes many paths, driven by cheap lasers. As line rates climbed, those many paths smeared the pulses together and cut the usable distance below the length of a row of racks. AI datacentres have standardised on single-mode fibre, whose narrow core carries one path, so reach becomes a matter of the laser and the loss budget. Silicon photonics has brought the cost of single-mode optics down. At the scale of a large cluster, with connectors in the hundreds of thousands, something as simple as cleaning each connector before it is mated measurably affects how many links work first time.

Why the network reaches the contract

A network that drops packets, congests or fails stalls the whole training job, so network quality shows up directly in a cluster's goodput. Buyers have learned to write it into their contracts, specifying the bandwidth a cluster must deliver for collective operations at given scales, alongside classical uptime. Two clusters with identical GPUs can differ widely in useful output, and much of the difference comes from the network and the people who tune it.

What it means for investors

The fabric is a meaningful share of a cluster's capital cost, and that share rises with each doubling of network-card speed. The suppliers span NVIDIA's networking business, Arista and Broadcom-based platforms for switching, and optics makers such as InnoLight, Coherent, Eoptolink and Fabrinet. The shift to co-packaged optics and faster cards moves value between them: towards whoever controls the switch package, and away from the separate pluggable module. Each piece of the stack, from the grid connection to the fibre, is now sized to the GPU generation it serves.

Sources

• AI Infrastructure Primer 2026, Compute Terminal: the chapters "Networking: from chip to fibre", "GPU productivity: installed is not productive" and "The value chain", with their dated sources.
• Ultra Ethernet Consortium specification, as summarised in the Primer.