【Editorial Discussion】

A decreasing training loss does not validate a custom backward implementation. But a failed finite-difference check is not automatically proof of a wrong derivative either. Numerical cancellation, a perturbation crossing a nondifferentiable boundary, or stochastic forward behavior can make a correct local derivative disagree with the test. A useful debugging lesson teaches engineers to identify which assumption failed before modifying the backward code.

The starting point should be a small scalar objective, not an entire training run. Choose a few inputs and parameters whose forward values and derivative structure are easy to inspect. Leave the optimizer out of the experiment so update rules and accumulated state cannot obscure the comparison.

**Use a directional derivative as the first probe**

For an input x and direction v, compare the autodiff quantity grad(f(x)) dot v with the central difference [f(x+h*v)-f(x-h*v)]/(2*h). This checks one projection of the gradient with two forward evaluations. Repeat with several directions and supplement with coordinate checks near a localized failure; one passing direction can miss an error orthogonal to it.

Sweep h instead of trusting one tiny step. A large perturbation includes nonlinear curvature effects, while an excessively small one can subtract nearly equal floating-point values and amplify rounding error. Use a higher-precision diagnostic where the operator supports it, and inspect absolute error as well as relative error. A relative metric alone becomes unstable when the true derivative is near zero.

**Separate three failure patterns**

1) If the discrepancy changes erratically between identical evaluations, inspect randomness and state mutation first. Hold random draws fixed across the two perturbed evaluations where appropriate, and verify that buffers or running statistics do not change. Decide whether the intended check concerns one fixed random realization or an expected loss; those require different estimators.

2) If disagreement appears only near a branch boundary, inspect the forward function. For a rectifier at its kink, there is no unique classical derivative to recover with a symmetric perturbation. Move to a smooth region to validate the ordinary derivative, then test the documented boundary convention separately rather than forcing agreement with an inapplicable numerical oracle.

3) If an error persists across a sensible step-size range in a smooth deterministic region, reduce the operator and compare against an independently expressed reference. Broadcasting, reductions, and transposes are useful places to inspect, but the test should locate the defect rather than presume one.

A finite set of passing probes is not a proof. Some errors cancel, and a higher-precision reference may use a different execution path. First-order checks also do not validate higher-order derivatives or mixed-precision training behavior.

What gradient-check exercise has helped your team distinguish a broken backward from a broken numerical test? Which minimal counterexample do you keep so a “fix” cannot simply tune tolerances until the check turns green?