广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

From optical BER to MFU: the three-term arithmetic that never makes it onto the slide

Started by marcuslin · 2026-11-18 21:37 北京时间 · 11 views · 2 replies
M Member
Every cluster review has a slide with MFU on it. Almost none of them show the arithmetic that connects a physical-layer number - optical bit error rate - to that percentage. It is a three-term conversion, and it is worth doing because it tells you whether to care about a link that "passes" its BER spec.

Term one: BER to event rate
A 400G link at BER 1E-12 produces one errored bit every 1e12 bits, which is 2.5 seconds at 400 Gbps. At 1E-9 the same link produces roughly 2500 errored bits per second. Both of those numbers can appear on the same link under different temperatures, which is why a link that passes at the bench fails in row 12 of a hot hall.

Term two: errored bits to retransmits
Modern fabrics use FEC in front of the MAC: RS(544,514) corrects up to 15 symbols per codeword, and its practical pre-FEC limit sits around 2E-3 to 4E-3 depending on the implementation and the interleaving. Below that, errors are absorbed and you see nothing but a slowly rising FEC counter. Above it, codewords fail, the packet is dropped, and one of two things happens:
- RoCEv2 / IB with reliable transport: a retransmit with an RTO measured in hundreds of microseconds to a millisecond, plus the receiver window stalls.
- Lossy or UDP-based collective: silent corruption that shows up later as a NaN or a divergence you spend a day on.

The useful rule: track pre-FEC BER, not just post-FEC. A link at pre-FEC 1E-5 is "clean" by post-FEC accounting and is also 100x closer to the FEC cliff than one at 1E-7. Both report zero errors.

Term three: retransmits to MFU
All-reduce in a ring or tree is synchronous: the step ends when the slowest participant finishes. So a link that stalls periodically taxes every GPU in the job, not just the one behind that port. Concretely: a 3-second training step over a 200 Gbps-per-GPU fabric, with one link injecting a 1 ms stall every 200 ms, loses 5 ms per 1 s of step time - 0.5% of MFU - and the same numbers applied to the lossy case, where a corrupted reduce forces a restart from the last checkpoint, are single-digit-percent losses.

Where this gets missed
- Looking at bandwidth and ignoring error rate. A 400G port that delivers 380G of goodput and one that delivers 380G with 300 retransmits per second look identical in a goodput graph and very different in a step-time graph.
- Reading module DDM once, at 25 °C. Case temperature is the variable that matters: an 800G OSFP at 14-16 W in a 1U 32-port switch is a few hundred watts of heat in one rack unit, and the module case routinely runs 20-25 °C above inlet. A link that shows BER 1E-12 at 25 °C can be at 1E-7 at 70 °C case temperature without any DDM alarm firing, because the alarm threshold is usually set for damage, not for performance.
- Blaming the NIC. Host-side counters (discards, out-of-order) rise after the fabric already lost something. Start at the module's pre-FEC counter, then the switch port, then the NIC.

A diagnostic order that has worked
1. Snapshot pre-FEC BER for every port at idle and under load, at the same time of day. Take the top 5% and label them; do not fix them yet.
2. Correlate those ports with rack position and inlet temperature. A cluster of high-BER ports in one rack usually means airflow, not bad modules.
3. Only then swap optics. Swapping before checking temperature teaches you nothing and usually moves the problem to another port.

Open question: for a 3-second step time, what retransmit rate do you treat as "must fix today" - one per minute, one per second, or is it only about the tail latency of the slowest collective? I would like to calibrate that threshold against real jobs rather than against module specs.

Disclosure: we supply 400G/800G optics and AOC for clusters, at F-tone Networks (f-tone.com); the numbers here come from link characterization work, not from a customer deployment.
L Member
Good breakdown, and the third term is where most reviews quietly stop. Two things I would add, both of which change the arithmetic rather than restate it.

First, a compact way to get from pre-FEC BER to "how close are we to the cliff", instead of comparing raw BER numbers across links.

RS(544,514) with 10-bit symbols gives about 544/10 = 54 symbols per codeword, 15 of which are correctable. For independent errors the mean number of errored symbols per codeword is roughly lambda = 544 * BER (54 symbols times a symbol error probability of about 10*BER). So:

- BER 1E-7 -> lambda ~5.4E-5, effectively zero codeword failures.
- BER 1E-5 -> lambda ~5.4E-3, still absorbed; the counter creeps.
- BER 1E-3 -> lambda ~0.54, still under the 15-symbol budget but no longer comfortable.
- The published 2E-3 to 4E-3 pre-FEC limits correspond to lambda around 1 to 2, i.e. a codeword that is routinely carrying one or two bad symbols.

The catch: optical errors are bursty, not independent. A CDR slip or a DGD excursion dumps a run of consecutive symbols into one codeword, so lambda measured from average BER understates the FEC load considerably. That is exactly why the practical limit sits two orders of magnitude below where a Poisson model would put it. Consequence for measurement: track the per-codeword error pattern or at minimum the pre-FEC counter delta per fixed interval, not just the BER average. A link whose counter increments in clumps of tens is in worse shape than one incrementing steadily at the same average rate.

Second, the third term deserves the amplification factor written in. For ring all-reduce the step is gated by the slowest participant, and a stalled hop blocks the ring twice per step, so the effective cost is roughly stall_fraction * 2 for a ring and (n-1)/n * stall_fraction for a tree, applied to the whole job rather than to one GPU. Practical form:

effective_bw = goodput_bw * (1 - retransmit_rate * stall_duration * amplification)

With a 1 ms stall every 200 ms and amplification 2, that is 1% of effective bandwidth before any loss of goodput shows up - invisible on a bandwidth graph, visible on step time.

What I would put on the weekly sheet, per switch, rather than a one-off bench run:

1. Pre-FEC counter delta per lane over a fixed 10 s window at production load, plus module case temperature at the same instant. Growth in one, flat in the other, is a healthy link; growth in both is a thermal margin problem.
2. Retransmit or PFC-pause counts per port, normalised per minute. This is the number that connects to term three and it is the one nobody graphs.
3. Step-time p50 and p99 alongside MFU. The gap between them is the stall tax, and it is usually far larger than the MFU delta suggests.

One caveat on your 0.5% figure: it assumes stalls land uniformly. In practice they cluster when a rack warms up in the afternoon, which is why the same job can show 0.5% in the morning and several percent after 15:00 - and why a single midday measurement is not evidence of a clean fabric.
A Member
关于 3 秒步长的“今天必须修”阈值,我们的做法是看重传对步长的影响而不是只看重传率本身:单次重传停顿 1ms、放大系数 2,步长 3 秒里每 200ms 一次停顿就吃掉约 1% 的有效带宽,MFU 掉一个点以上就值得当天处理。换算成速率,大致是每步超过 15 次重传、或每分钟重传导致 p99 步长比 p50 高出 5% 以上,就进当日清单。更关键的是突发性:均匀分布的重传可以靠流水线吸收,集中在某几步的重传会直接拉长同步等待,p99 步长一跳就是信号。我们会把 pre-FEC BER、重传数和步长 p99 画在同一张时间轴上,同一时段三者一起抬头,基本能定位到某条链路或某个机架,而不是全网排查。
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.