【Editorial Discussion】

Mixed-precision training may detect nonfinite gradients, skip an optimizer update, and adjust the loss scale. If the training loop advances a learning-rate schedule on every attempted batch anyway, the learning rate can move forward while the weights do not receive the corresponding updates. A resume or comparison experiment can then follow a different trajectory despite apparently matching batch counts.

That mismatch is not automatically a scheduler bug. Some schedules deliberately follow consumed examples or tokens; others are defined over successful optimizer updates. The defect is an unstated or inconsistently implemented clock. A log field called “step” is inadequate when several different events share that name.

**Name the clocks separately**

Distinguish microbatches processed, accumulation windows attempted, optimizer updates accepted, and examples or tokens consumed. Also distinguish an attempted window from a partially completed accumulation. Decide which event drives warmup, decay, evaluation cadence, and checkpoint metadata. Those choices can differ, but they should not drift accidentally between the training loop and its recovery path.

Use a tiny deterministic model and a fixed input sequence. In a controlled test, inject a nonfinite gradient into one accumulation window and verify that the loss-scaling mechanism rejects that update. Record the learning rate actually used at each accepted update, not only the rate printed after the scheduler was advanced.

**Compare the next successful update**

If the declared policy is update-based, compare against a reference run that starts from identical model and optimizer state and performs the same accepted updates without advancing the schedule for the rejected attempt. If the policy is token-based, the expected learning rate may intentionally differ; test against that declared token count instead. Do not silently switch policies to make the two runs match.

Observe acceptance through an explicit signal or instrumented control path where available. A zero parameter delta alone does not prove an update was skipped: an accepted update can have zero gradients or otherwise leave a parameter unchanged. Likewise, calling an optimizer wrapper does not establish that its underlying update executed.

Then checkpoint immediately after a rejected window and resume. Confirm that model state, optimizer state, schedule counters, loss-scale state, and accumulation status restore a consistent next action. In distributed training, verify that ranks agree on rejection before any rank commits an update; otherwise the issue is no longer just a shifted schedule.

This experiment does not diagnose why overflow occurred or establish the best schedule for convergence. Retrying the same data versus discarding the rejected window changes the learning process and needs its own explicit rule. Stochastic operations can also complicate exact trajectory comparisons, so begin with a controlled diagnostic before returning to the full training stack.

Which event does your training loop call a step? What acceptance test would catch a skipped update advancing the wrong clock, especially across accumulation and checkpoint recovery, without assuming every schedule must be update-based?