F
Member
Two topologies dominate GPU cluster network design: the classic fat-tree, and rail-optimized (also called rail-aligned) which became popular with the 8-GPU-per-node generation. Most comparisons online are qualitative. This post puts numbers on the tradeoff.
1. Traffic pattern first
Everything follows from which collective you run most.
AllReduce (data parallel): every GPU talks to every other GPU, roughly equal volumes. Wants full bisection bandwidth.
All-to-All (expert parallel / MoE): each GPU sends a different chunk to every other GPU. Wants full bisection, and is much more sensitive to congestion.
AllGather / ReduceScatter (ZeRO, tensor parallel across nodes): nearest-neighbour in a ring, but the ring spans the whole cluster.
If your workload is pure data-parallel AllReduce, rail-optimized is excellent. If you run MoE with expert parallelism, fat-tree is safer.
2. What rail-optimized actually does
In an 8-GPU node, GPU i connects to NIC i, and NIC i of every node connects to the same leaf switch (rail i). So:
Rail 0: all GPU 0s in the cluster behind leaf switch 0
Rail 1: all GPU 1s behind leaf switch 1
...
Within a rail, communication is one hop. Across rails, traffic has to go up to spine.
Why this is efficient: the dominant collective, Ring AllReduce, forms a ring where each GPU talks to its two neighbours. If the ring is built rail-aligned, most of that traffic stays within a rail. The result is that you can build a 1:1 oversubscribed rail layer and a heavily oversubscribed spine layer, and still get good AllReduce performance.
3. The numbers
Consider 512 GPUs (64 nodes x 8), 400G per GPU:
Fat-tree, non-blocking:
Leaf: 64 ports per switch, 32 down / 32 up
Leaves needed: 512 / 32 = 16
Spines needed: 16 leaves x 32 up / 64 ports = 8 spines
Total switches: 24
Optics: 512 node links + 512 leaf-spine links = 1024 x 400G
At 400G pluggable, roughly 1024 x 12 W = 12.3 kW of optics
Rail-optimized, 1:1 within rail, 4:1 spine oversubscription:
Leaves: 8 rails, each with 64 down (64 nodes) — need 8 leaf switches at 64 ports
Spines: 8 leaves x 8 up / 64 = 1 spine... in practice 2 for redundancy
Total switches: 10
Optics: 512 node links + 64 leaf-spine links = 576 x 400G
Roughly 6.9 kW of optics
So rail-optimized cuts switch count by more than half and optics power by 44%, at the cost of cross-rail bandwidth.
4. Where rail-optimized hurts
MoE / All-to-All: cross-rail traffic explodes, and your 4:1 oversubscribed spine becomes the bottleneck. Step time can double.
Tensor parallel across nodes: if you place TP ranks across rails, every layer pays the cross-rail penalty.
Rebalancing / checkpointing: storage traffic is often cross-rail.
Mitigation: map ranks to rails deliberately. If your parallelism strategy puts TP within a node and DP across nodes, rail-optimized is nearly free. If TP spans nodes, you need more spine bandwidth.
5. What I would actually build
For a first cluster under 256 GPUs, running dense models with data parallelism:
Rail-optimized, 1:1 rail, 2:1 spine. Simple, cheaper, and the failure domain is smaller.
For 512+ GPUs, or any MoE workload:
Fat-tree, or a rail-optimized design with 2:1 spine. Do the math on your actual collective mix before committing.
For a research cluster where the workload will change:
Fat-tree. You pay for flexibility, but you will not be re-cabling in a year.
6. One thing that matters more than topology
Per-port visibility. A rail-optimized cluster with per-port BER and FEC counters, sampled every 10 seconds, will outperform a fat-tree cluster where you only see aggregate switch counters — because you will actually find the bad link before it ruins a two-week training run.
Topology gets you the theoretical number. Observability gets you the achieved one.
-- F-tone Networks, Optical Interconnect Engineering
Technical resources: https://www.f-tone.com
1. Traffic pattern first
Everything follows from which collective you run most.
AllReduce (data parallel): every GPU talks to every other GPU, roughly equal volumes. Wants full bisection bandwidth.
All-to-All (expert parallel / MoE): each GPU sends a different chunk to every other GPU. Wants full bisection, and is much more sensitive to congestion.
AllGather / ReduceScatter (ZeRO, tensor parallel across nodes): nearest-neighbour in a ring, but the ring spans the whole cluster.
If your workload is pure data-parallel AllReduce, rail-optimized is excellent. If you run MoE with expert parallelism, fat-tree is safer.
2. What rail-optimized actually does
In an 8-GPU node, GPU i connects to NIC i, and NIC i of every node connects to the same leaf switch (rail i). So:
Rail 0: all GPU 0s in the cluster behind leaf switch 0
Rail 1: all GPU 1s behind leaf switch 1
...
Within a rail, communication is one hop. Across rails, traffic has to go up to spine.
Why this is efficient: the dominant collective, Ring AllReduce, forms a ring where each GPU talks to its two neighbours. If the ring is built rail-aligned, most of that traffic stays within a rail. The result is that you can build a 1:1 oversubscribed rail layer and a heavily oversubscribed spine layer, and still get good AllReduce performance.
3. The numbers
Consider 512 GPUs (64 nodes x 8), 400G per GPU:
Fat-tree, non-blocking:
Leaf: 64 ports per switch, 32 down / 32 up
Leaves needed: 512 / 32 = 16
Spines needed: 16 leaves x 32 up / 64 ports = 8 spines
Total switches: 24
Optics: 512 node links + 512 leaf-spine links = 1024 x 400G
At 400G pluggable, roughly 1024 x 12 W = 12.3 kW of optics
Rail-optimized, 1:1 within rail, 4:1 spine oversubscription:
Leaves: 8 rails, each with 64 down (64 nodes) — need 8 leaf switches at 64 ports
Spines: 8 leaves x 8 up / 64 = 1 spine... in practice 2 for redundancy
Total switches: 10
Optics: 512 node links + 64 leaf-spine links = 576 x 400G
Roughly 6.9 kW of optics
So rail-optimized cuts switch count by more than half and optics power by 44%, at the cost of cross-rail bandwidth.
4. Where rail-optimized hurts
MoE / All-to-All: cross-rail traffic explodes, and your 4:1 oversubscribed spine becomes the bottleneck. Step time can double.
Tensor parallel across nodes: if you place TP ranks across rails, every layer pays the cross-rail penalty.
Rebalancing / checkpointing: storage traffic is often cross-rail.
Mitigation: map ranks to rails deliberately. If your parallelism strategy puts TP within a node and DP across nodes, rail-optimized is nearly free. If TP spans nodes, you need more spine bandwidth.
5. What I would actually build
For a first cluster under 256 GPUs, running dense models with data parallelism:
Rail-optimized, 1:1 rail, 2:1 spine. Simple, cheaper, and the failure domain is smaller.
For 512+ GPUs, or any MoE workload:
Fat-tree, or a rail-optimized design with 2:1 spine. Do the math on your actual collective mix before committing.
For a research cluster where the workload will change:
Fat-tree. You pay for flexibility, but you will not be re-cabling in a year.
6. One thing that matters more than topology
Per-port visibility. A rail-optimized cluster with per-port BER and FEC counters, sampled every 10 seconds, will outperform a fat-tree cluster where you only see aggregate switch counters — because you will actually find the bad link before it ruins a two-week training run.
Topology gets you the theoretical number. Observability gets you the achieved one.
-- F-tone Networks, Optical Interconnect Engineering
Technical resources: https://www.f-tone.com