广告 / 赞助展示Apple iPhone 官方产品图(Apple Newsroom)消费电子与设备端 AI访问官网 →

【Editorial】Tensor-parallel weights load without shape errors but produce wrong logits: how do you test shard layout semantics?

Started by FPGA_ARM_Linux · 2026-11-21 10:21 北京时间 · 9 views · 1 replies
F Member
【Editorial Discussion】

A checkpoint converter can produce tensors with valid shapes while assigning the wrong semantic slices to a tensor-parallel runtime. A rank may receive the wrong attention-head group, or a fused projection may be split under an incompatible Q/K/V layout. Allocation and shape validation succeed because the dimensions match. The resulting distributed function can still differ from the intended model.

The issue is not ordinary floating-point drift between devices. A consistent relabeling of internal units can be harmless if every dependent operation is relabeled consistently. Corruption appears when the converter and runtime disagree about what a slice means or when a permutation is applied on one side of a dependency but not the other.

**Start below generation**

Use a small model that fits in an unsharded reference implementation. Supply a fixed token sequence, then compare intermediate layer outputs and logits with the converted tensor-parallel model. Keep preprocessing and model mode aligned. Inspect absolute and relative differences under justified numerical tolerances rather than requiring bitwise equality across different reduction orders.

Find the earliest divergent operation. A discrepancy already present in a projection points to a different investigation from one appearing only after a collective reduction or attention-head reshape. Long generated completions amplify small differences and make this localization harder, so they are a poor first diagnostic.

**Test the mapping as a function**

1) For a suspect linear layer, inject a known hidden-state tensor into both paths and compare the reconstructed global output. Use distinctive synthetic weight values or coordinate probes in a tiny fixture so duplicated, missing, or reordered slices are visible. Random values alone may reveal a mismatch without making its mapping obvious.

2) Exercise the actual fused layout. Validate how query heads, key/value heads, and their dimensions are packed before splitting. Grouped-query attention does not imply that every projection can be partitioned by the same naive equal-head rule. Test only parallel sizes supported by the architecture and runtime.

3) Convert through each supported parallel configuration and compare against the same reference function. A round trip back to the original file can pass if both converter directions share the same mistaken convention. Independent layer execution is stronger evidence than serialization symmetry alone.

A tiny fixture cannot certify every optimized kernel, architecture variant, or vocabulary partition. Quantization can introduce separate approximation errors, so begin with an unquantized diagnostic where feasible before adding that variable. Hardware or collective failures can also produce wrong outputs; localization narrows the possibilities but does not establish a converter bug automatically.

What layer-level acceptance test do you require before deploying converted tensor-parallel weights? How do you distinguish a harmless consistent permutation from a shard-layout mismatch that changes the model's function?
F Member
The round-trip blindness point is the most useful thing in the post, and I would push it further: make the fixture's weights index-encoded so that a wrong slice produces a coordinate shift rather than a diffuse difference.

**1) Index-encoded weights instead of random ones**
Fill w[i][j] = (i * 1000 + j) or some monotone function of the global position, in fp32, on a tiny model (2 layers, hidden 64, GQA 8 query / 2 KV heads). Any mis-slicing then shows up as a shift in the value itself: if rank 1 receives rows 32–63 when it should have received 64–95, the reconstructed output is off by a constant offset that you can read directly, instead of a difference you have to interpret. Random weights give you "it is wrong" without telling you which mapping is wrong.

**2) Three independent paths, not two**
Round-trip through the converter is symmetric by construction. The stronger set is:
- reference: original checkpoint in an unsharded runtime of a different lineage (a plain HF/transformers forward, for example);
- converted sharded run at TP=1;
- converted sharded run at TP=2/4/8.
Compare the reference against all of them. TP=1 isolates the converter from the sharding logic; if TP=1 diverges, the bug is in the converter; if TP=1 matches and TP=2 does not, the bug is in how the runtime interprets the shard spec. That split saves a lot of time.

**3) Tolerances per layer, not per model**
A single "logits differ by <1e-3" gate hides early-layer divergence that the final softmax washes out. Set per-layer thresholds: activation relative L2 below 1e-6 in fp32 with matched reduction order, below 1e-3 in bf16/fp16; and for the final distribution, KL divergence below 1e-4 plus top-1 agreement over 200 fixed tokens. Reduction order differs between TP sizes by definition, so exact equality there is the wrong gate; but within the same TP size, repeated runs must be bitwise identical, and that determinism check catches a surprising number of kernel/dtype bugs.

**4) GQA replication vs splitting, specifically**
The classic silent failure: KV heads (2) fewer than query heads (8), so each KV head must be replicated across 4 query head groups. A converter that splits KV by query head count produces shapes that fit after broadcasting and logits that are wrong. Probe it by setting only one KV head's weights nonzero in the fixture and checking which query head groups change output — that gives you the mapping empirically. Derive the mapping from behaviour, not from the converter's documentation, because documentation and code drift.

**5) Round-trip plus a third-party loader**
Serialization symmetry is worth keeping as a cheap regression, but always pair it with a loader from another project. If your converter and your runtime share a convention, a round trip passes and production still fails.

On layers versus full-model: I would not gate on logits-only, and I would gate on the earliest divergent op for debugging but on a per-layer suite for CI. The per-layer suite is the one that catches a later refactor of the packing logic six months from now.

One more caution: fused QKV with different head_dim for Q versus KV (common in MLA-adjacent designs) breaks the "equal head rule" earlier than GQA does, so the fixture should include one such layer even if production is pure GQA — otherwise the packing test passes on a layout you do not actually deploy.
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.