F
Member
A training run that is 8% slower than the benchmark usually gets blamed on the framework, the dataloader, or the GPU thermals. In our experience, a meaningful share of these cases trace back to a handful of optical links running at BER 1e-8 instead of 1e-12. Nothing is down. Nothing throws an error. The run is just slower.
This post explains the mechanism and gives a measurement procedure.
1. How a 1e-8 BER becomes a step-time penalty
At 400G per link, 1e-8 BER means roughly 4 wrong bits per second per link. Those bits corrupt Ethernet frames, which fail CRC and get dropped.
With FEC (RS544,514 on 400G), most of that is corrected. But FEC correction is not free in the tail: bursts that exceed the correction budget become uncorrectable, and an uncorrectable codeword means a dropped frame.
A dropped frame in RoCEv2 triggers one of two things:
A PFC pause, which propagates backpressure
A congestion-control reaction (ECN mark, or timeout)
Either one stalls the collective. A single stalled link in a Ring AllReduce makes every other rank wait, because the ring is only as fast as its slowest member.
Quantitatively: a 100 microsecond stall every 10 seconds is a 0.001% loss. But collectives amplify it. If the affected link is on the critical path of a 126-step ring at 64 ranks, and the stall happens during a step, the whole step waits.
2. Why it looks like "random slowness"
Three properties make this hard to find:
It is intermittent. The link is fine most of the time.
It is asymmetric. Usually one direction of one link, so A/B swap tests pass.
It does not show in aggregate counters. A switch reporting 99.99% of ports clean looks healthy.
The signal is in per-port FEC counters, which most operators do not scrape.
3. Measurement procedure
Step 1: establish that it is network-side
Compare step time across different parallelism configurations. If the penalty disappears when you shrink to a single node, it is cross-node.
Compare against a run with a different rank-to-node mapping. If the penalty follows specific ranks, you have a link or a node.
Step 2: scrape per-port counters
On the switch: FEC corrected codewords, FEC uncorrected codewords, CRC errors, per port, per direction.
On the host NIC: rx_crc_errors, rx_missed_errors, and RoCE counters (out_of_sequence, packet_seq_err).
Sample every 10 seconds. Five-minute polling averages away the signal.
Step 3: correlate with step time
This is the part people skip. Plot FEC corrected codewords per second against step time over the same window. If the correlation is visible, you have your answer.
Step 4: physical layer verification
ethtool -m on both ends. Look at:
Rx power against the sensitivity floor, not against zero
Module temperature (anything sustained above 70 C is suspect)
Bias current, compared to the baseline taken at install
Then clean the end faces, and re-measure. This resolves a surprising fraction of cases.
Step 5: swap, one variable at a time
Cable, then module, then port. Never two at once.
4. Thresholds we use
FEC corrected, idle: should be flat. Any growth means the link is dirty.
FEC corrected under load: some growth is normal, but rate matters. A doubling per day is a failing link.
FEC uncorrected: must be zero. One uncorrectable codeword is a dropped frame.
Pre-FEC BER: better than 1e-9 for a "clean" link. Between 1e-9 and 1e-7, keep it but watch it. Worse than 1e-7, expect application impact.
5. The organizational problem
The reason this persists is that it crosses team boundaries. The network team sees clean ports. The ML team sees slow steps. Nobody owns the correlation.
The fix is boring: put FEC counters and step time on the same dashboard, scraped at the same interval. Once both are visible side by side, the argument takes ten minutes instead of two weeks.
6. A checklist
1. Per-port FEC and CRC counters, 10-second sampling, retained 30 days
2. DOM baseline at install, weekly trend comparison
3. Pre-FEC BER acceptance test before the cluster enters production
4. End-face inspection as part of install, not troubleshooting
5. Step time and FEC counters on one dashboard
The link that is "up" is not necessarily the link that is working.
-- F-tone Networks, Optical Interconnect Engineering
Technical resources: https://www.f-tone.com
This post explains the mechanism and gives a measurement procedure.
1. How a 1e-8 BER becomes a step-time penalty
At 400G per link, 1e-8 BER means roughly 4 wrong bits per second per link. Those bits corrupt Ethernet frames, which fail CRC and get dropped.
With FEC (RS544,514 on 400G), most of that is corrected. But FEC correction is not free in the tail: bursts that exceed the correction budget become uncorrectable, and an uncorrectable codeword means a dropped frame.
A dropped frame in RoCEv2 triggers one of two things:
A PFC pause, which propagates backpressure
A congestion-control reaction (ECN mark, or timeout)
Either one stalls the collective. A single stalled link in a Ring AllReduce makes every other rank wait, because the ring is only as fast as its slowest member.
Quantitatively: a 100 microsecond stall every 10 seconds is a 0.001% loss. But collectives amplify it. If the affected link is on the critical path of a 126-step ring at 64 ranks, and the stall happens during a step, the whole step waits.
2. Why it looks like "random slowness"
Three properties make this hard to find:
It is intermittent. The link is fine most of the time.
It is asymmetric. Usually one direction of one link, so A/B swap tests pass.
It does not show in aggregate counters. A switch reporting 99.99% of ports clean looks healthy.
The signal is in per-port FEC counters, which most operators do not scrape.
3. Measurement procedure
Step 1: establish that it is network-side
Compare step time across different parallelism configurations. If the penalty disappears when you shrink to a single node, it is cross-node.
Compare against a run with a different rank-to-node mapping. If the penalty follows specific ranks, you have a link or a node.
Step 2: scrape per-port counters
On the switch: FEC corrected codewords, FEC uncorrected codewords, CRC errors, per port, per direction.
On the host NIC: rx_crc_errors, rx_missed_errors, and RoCE counters (out_of_sequence, packet_seq_err).
Sample every 10 seconds. Five-minute polling averages away the signal.
Step 3: correlate with step time
This is the part people skip. Plot FEC corrected codewords per second against step time over the same window. If the correlation is visible, you have your answer.
Step 4: physical layer verification
ethtool -m on both ends. Look at:
Rx power against the sensitivity floor, not against zero
Module temperature (anything sustained above 70 C is suspect)
Bias current, compared to the baseline taken at install
Then clean the end faces, and re-measure. This resolves a surprising fraction of cases.
Step 5: swap, one variable at a time
Cable, then module, then port. Never two at once.
4. Thresholds we use
FEC corrected, idle: should be flat. Any growth means the link is dirty.
FEC corrected under load: some growth is normal, but rate matters. A doubling per day is a failing link.
FEC uncorrected: must be zero. One uncorrectable codeword is a dropped frame.
Pre-FEC BER: better than 1e-9 for a "clean" link. Between 1e-9 and 1e-7, keep it but watch it. Worse than 1e-7, expect application impact.
5. The organizational problem
The reason this persists is that it crosses team boundaries. The network team sees clean ports. The ML team sees slow steps. Nobody owns the correlation.
The fix is boring: put FEC counters and step time on the same dashboard, scraped at the same interval. Once both are visible side by side, the argument takes ten minutes instead of two weeks.
6. A checklist
1. Per-port FEC and CRC counters, 10-second sampling, retained 30 days
2. DOM baseline at install, weekly trend comparison
3. Pre-FEC BER acceptance test before the cluster enters production
4. End-face inspection as part of install, not troubleshooting
5. Step time and FEC counters on one dashboard
The link that is "up" is not necessarily the link that is working.
-- F-tone Networks, Optical Interconnect Engineering
Technical resources: https://www.f-tone.com