Two 100-step training runs printed the same loss to four decimal places at every step. Their weights differed at every step. The only change was the reduction orderThe order in which values are added together. used to add the per-example float32 gradients.
The largest weight difference stayed below 1.2 × 10−7. Both runs learned essentially the same regression function. The failure was in exact replayRerunning the same steps to check whether they lead to the same state., which is the property needed when a developer wants to reproduce a specific numerical event rather than merely reach a similar final score.
Why exact replay deserves its own test
NVIDIA’s October 2026 Megatron Core determinism post distinguishes independent reproducibility from checkpoint-resume reproducibility. Its validation scope fixes the hardware, software, recipe, and parallelism configuration. It also recommends finding the first numerical disagreement instead of waiting for visible loss divergence. The local study below turns those two replay contracts into executable checks.
A random seed controls a source of randomness. It does not specify every arithmetic operation or the state of an interrupted optimizer. PyTorch’s reproducibility guidance similarly separates random-number control from deterministic algorithm selection and warns against expecting identical results across releases or platforms.
The local test holds those environmental questions still. The dataset, initial weights, batch-sampling seed, optimizer settings, and Python process environment remain fixed. Each intervention changes one replay-relevant part of the program, except the optimizer-reset case, which deliberately resets both its moments and its update counter.
A training loop small enough to inspect
The CPU experiment trains a 24-weight linear regressor with Adam for 100 updates. It uses 512 synthetic rows and samples 64 without replacement per update. The target contains a linear signal, a small sinusoidal component, and noise. This makes the optimization nontrivial without introducing a neural network’s additional implementation choices.
Training state uses float32. Evaluation loss is computed in float64 over the fixed dataset, then formatted to four decimal places for the logging comparison. The experiment stores both the full evaluation value and the formatted string.
For every update, the program retains ordered fingerprints of the gradient contributions, the reduced gradient, the weights, and the complete replay state. A shape-and-dtype-aware SHA-256 digest summarizes array bytes. The checkpoint validation also compares final arrays directly, so matching hashes are not the sole basis for claiming an exact restart.
There are five comparisons against an uninterrupted reference:
- Start again from the same full initial state.
- Save after step 40 and restore the complete checkpoint.
- Reverse the addition order of the same gradient contributions.
- Resume the step-40 weights but reset Adam’s moments and update counter.
- Restore the optimizer and weights, but reset the sampler’s random-number state to its original seed.
The first mismatch arrives before the loss tells you
The forward and reverse reductions receive identical ordered per-example gradient arrays at step 1. Their reduced gradients differ in bytes. Their updated weights then differ. This establishes a local origin for the disagreement instead of merely noticing that two later models are different.
Floating-point addition rounds intermediate values. Changing the order changes which intermediate sums get rounded. NVIDIA’s floating-point guide explains this non-associativity and how mathematically equivalent evaluation orders can produce different results. Here, both orders are deterministic in isolation. Reversing one of them produces a different deterministic trajectory.
Across the full run, all 100 weight comparisons fail exact equality, while all 100 four-decimal loss strings match. The reference final loss is 0.0202684791; the reversed-order loss is 0.0202684778. The loss comparison misses changes that a byte-level weight check catches.
This is also why the earliest mismatch is more useful than the largest mismatch. If the investigation begins at step 80, both runs already enter the update with different weights. At step 1, the matching gradient inputs and differing reduction output identify the changed operation directly.
A checkpoint has to resume a process
The full checkpointSaved values needed to continue computation from a particular point. contains weights, both Adam moment arrays, the optimizer update count, and the batch sampler’s generator state. The data and configuration are held fixed and identified separately. The archive stores arrays without pickle; a JSON companion stores the scalar and generator state.
Restoring that checkpoint reproduces all 60 remaining steps exactly. Saving it to disk, loading it again, and finishing training also produces byte-identical final weight and moment arrays, with matching sampler state and update count.
The partial restorations show why weights alone are insufficient. Resetting Adam changes the largest observed weight coordinate by 0.1605 relative to the reference at some point after restart. Resetting only the sampler’s random-number stream reaches a maximum difference of 0.02873. Both first diverge at step 41, immediately after the checkpoint boundary.
The sampler-reset run ends with slightly lower loss than the reference: 0.02016255 versus 0.02026848. That result is useful because it prevents a convenient but incorrect interpretation. The failed replay did not have to look worse on the final metric. It followed a different batch sequence.
For a real trainer, the state inventory can be larger: a learning-rate scheduler, gradient scaler, data-loader cursor, distributed sampler, and framework-specific random generators may all influence the next update. This lab has no such components, so its checkpoint cannot validate their recovery.
What to carry into a larger training stack
Turn independent replay and resumed replay into separate tests. Compare the first operation that receives matching inputs but emits differing outputs. Save the state that determines the next update, not just the object eventually exported for inference.
A replay test makes a changed kernel or restart path comparable against a fixed numerical reference. Logging precision can then be chosen for observability rather than treated as evidence of exact state recovery.
Extending these checks to a distributed trainer would add another measurement: the cost of preserving exact replay. That comparison needs a fixed hardware configuration, parallelism strategy, precision format, and checkpoint path. The notebook provides a compact reference implementation for the two replay contracts.
Sources and reproduction
- NVIDIA: Scale Bitwise-Deterministic Pretraining with Megatron Core, October 6, 2026
- PyTorch: Reproducibility
- NVIDIA: Floating Point and IEEE 754
Read the notebook with its code and saved outputs, or download the editable Jupyter notebook, Python source, and reproducibility package.