M
Member
Every cluster review has a slide with MFU on it. Almost none of them show the arithmetic that connects a physical-layer number - optical bit error rate - to that percentage. It is a three-term conversion, and it is worth doing because it tells you whether to care about a link that "passes" its BER spec.
Term one: BER to event rate
A 400G link at BER 1E-12 produces one errored bit every 1e12 bits, which is 2.5 seconds at 400 Gbps. At 1E-9 the same link produces roughly 2500 errored bits per second. Both of those numbers can appear on the same link under different temperatures, which is why a link that passes at the bench fails in row 12 of a hot hall.
Term two: errored bits to retransmits
Modern fabrics use FEC in front of the MAC: RS(544,514) corrects up to 15 symbols per codeword, and its practical pre-FEC limit sits around 2E-3 to 4E-3 depending on the implementation and the interleaving. Below that, errors are absorbed and you see nothing but a slowly rising FEC counter. Above it, codewords fail, the packet is dropped, and one of two things happens:
- RoCEv2 / IB with reliable transport: a retransmit with an RTO measured in hundreds of microseconds to a millisecond, plus the receiver window stalls.
- Lossy or UDP-based collective: silent corruption that shows up later as a NaN or a divergence you spend a day on.
The useful rule: track pre-FEC BER, not just post-FEC. A link at pre-FEC 1E-5 is "clean" by post-FEC accounting and is also 100x closer to the FEC cliff than one at 1E-7. Both report zero errors.
Term three: retransmits to MFU
All-reduce in a ring or tree is synchronous: the step ends when the slowest participant finishes. So a link that stalls periodically taxes every GPU in the job, not just the one behind that port. Concretely: a 3-second training step over a 200 Gbps-per-GPU fabric, with one link injecting a 1 ms stall every 200 ms, loses 5 ms per 1 s of step time - 0.5% of MFU - and the same numbers applied to the lossy case, where a corrupted reduce forces a restart from the last checkpoint, are single-digit-percent losses.
Where this gets missed
- Looking at bandwidth and ignoring error rate. A 400G port that delivers 380G of goodput and one that delivers 380G with 300 retransmits per second look identical in a goodput graph and very different in a step-time graph.
- Reading module DDM once, at 25 °C. Case temperature is the variable that matters: an 800G OSFP at 14-16 W in a 1U 32-port switch is a few hundred watts of heat in one rack unit, and the module case routinely runs 20-25 °C above inlet. A link that shows BER 1E-12 at 25 °C can be at 1E-7 at 70 °C case temperature without any DDM alarm firing, because the alarm threshold is usually set for damage, not for performance.
- Blaming the NIC. Host-side counters (discards, out-of-order) rise after the fabric already lost something. Start at the module's pre-FEC counter, then the switch port, then the NIC.
A diagnostic order that has worked
1. Snapshot pre-FEC BER for every port at idle and under load, at the same time of day. Take the top 5% and label them; do not fix them yet.
2. Correlate those ports with rack position and inlet temperature. A cluster of high-BER ports in one rack usually means airflow, not bad modules.
3. Only then swap optics. Swapping before checking temperature teaches you nothing and usually moves the problem to another port.
Open question: for a 3-second step time, what retransmit rate do you treat as "must fix today" - one per minute, one per second, or is it only about the tail latency of the slowest collective? I would like to calibrate that threshold against real jobs rather than against module specs.
Disclosure: we supply 400G/800G optics and AOC for clusters, at F-tone Networks (f-tone.com); the numbers here come from link characterization work, not from a customer deployment.
Term one: BER to event rate
A 400G link at BER 1E-12 produces one errored bit every 1e12 bits, which is 2.5 seconds at 400 Gbps. At 1E-9 the same link produces roughly 2500 errored bits per second. Both of those numbers can appear on the same link under different temperatures, which is why a link that passes at the bench fails in row 12 of a hot hall.
Term two: errored bits to retransmits
Modern fabrics use FEC in front of the MAC: RS(544,514) corrects up to 15 symbols per codeword, and its practical pre-FEC limit sits around 2E-3 to 4E-3 depending on the implementation and the interleaving. Below that, errors are absorbed and you see nothing but a slowly rising FEC counter. Above it, codewords fail, the packet is dropped, and one of two things happens:
- RoCEv2 / IB with reliable transport: a retransmit with an RTO measured in hundreds of microseconds to a millisecond, plus the receiver window stalls.
- Lossy or UDP-based collective: silent corruption that shows up later as a NaN or a divergence you spend a day on.
The useful rule: track pre-FEC BER, not just post-FEC. A link at pre-FEC 1E-5 is "clean" by post-FEC accounting and is also 100x closer to the FEC cliff than one at 1E-7. Both report zero errors.
Term three: retransmits to MFU
All-reduce in a ring or tree is synchronous: the step ends when the slowest participant finishes. So a link that stalls periodically taxes every GPU in the job, not just the one behind that port. Concretely: a 3-second training step over a 200 Gbps-per-GPU fabric, with one link injecting a 1 ms stall every 200 ms, loses 5 ms per 1 s of step time - 0.5% of MFU - and the same numbers applied to the lossy case, where a corrupted reduce forces a restart from the last checkpoint, are single-digit-percent losses.
Where this gets missed
- Looking at bandwidth and ignoring error rate. A 400G port that delivers 380G of goodput and one that delivers 380G with 300 retransmits per second look identical in a goodput graph and very different in a step-time graph.
- Reading module DDM once, at 25 °C. Case temperature is the variable that matters: an 800G OSFP at 14-16 W in a 1U 32-port switch is a few hundred watts of heat in one rack unit, and the module case routinely runs 20-25 °C above inlet. A link that shows BER 1E-12 at 25 °C can be at 1E-7 at 70 °C case temperature without any DDM alarm firing, because the alarm threshold is usually set for damage, not for performance.
- Blaming the NIC. Host-side counters (discards, out-of-order) rise after the fabric already lost something. Start at the module's pre-FEC counter, then the switch port, then the NIC.
A diagnostic order that has worked
1. Snapshot pre-FEC BER for every port at idle and under load, at the same time of day. Take the top 5% and label them; do not fix them yet.
2. Correlate those ports with rack position and inlet temperature. A cluster of high-BER ports in one rack usually means airflow, not bad modules.
3. Only then swap optics. Swapping before checking temperature teaches you nothing and usually moves the problem to another port.
Open question: for a 3-second step time, what retransmit rate do you treat as "must fix today" - one per minute, one per second, or is it only about the tail latency of the slowest collective? I would like to calibrate that threshold against real jobs rather than against module specs.
Disclosure: we supply 400G/800G optics and AOC for clusters, at F-tone Networks (f-tone.com); the numbers here come from link characterization work, not from a customer deployment.

