An agent's newest experiences often look much like the ones immediately before them. A robot moving through one corridor or a game agent spending time in one region collects a local sequence, not a freshly randomized training set. Updating from that sequence can concentrate many consecutive changes on the same narrow situation.
The original deep Q-learning Atari paper stores transitions in replay memory and samples them for updates. Reuse reduces the coupling between the current interaction sequence and the training order. Because stored transitions may come from earlier behavior policies, the method learns off-policy. Playing Atari with Deep Reinforcement Learning
The 2013 Atari paper reports outperforming earlier approaches on six of its seven games and its human-expert comparison on three. Replay is one component of that agent. Neither those game results nor a shuffled list alone isolates replay's causal contribution; this article instead tests one specific mechanism, the training order induced by a fixed buffer.
Inspect a buffer before drawing a learning curve
Make a buffer with twelve transition IDs. IDs 0 through 5 come from regime A, followed by IDs 6 through 11 from regime B. The labels can stand for two regions of state space; they are not rewards.
In collection order, ten of the eleven adjacent pairs have the same regime label. The only cross-regime boundary is between IDs 5 and 6. A learner that updates sequentially sees a long run of A-like evidence followed by a long run of B-like evidence.
The code creates a seeded random permutation of those same twelve transitions and counts adjacent same-regime pairs again. It keeps every transition exactly once, isolating the ordering effect from extra reuse or missing samples. This is a without-replacement teaching example, whereas a practical replay learner can sample minibatches repeatedly from a moving buffer.
The expected same-regime fraction for a uniformly permuted adjacent pair in this balanced buffer is 5/11. After choosing one transition, only five of the remaining eleven share its regime. The observed fraction in one finite shuffle can differ. One seed is a reproducible illustration, not a statistical experiment.
Replay does not manufacture coverage
A random sample from a buffer containing only corridor A cannot teach the agent about corridor B. It can reduce temporal ordering patterns in available data, but it cannot repair missing transitions. Buffer composition and exploration still matter.
Nor does replay make updates independent in every sense. Samples can repeat, transitions share trajectories, bootstrapped targets depend on the current value estimates, and the buffer itself was collected by related policies. Calling replay data “independent and identically distributed” without qualification hides these dependencies.
For Q-learning, a nonterminal target has the familiar form reward + gamma × max_a Q(next_state, a). Replaying an old transition recomputes that target using the learner's current estimates. It does not simply memorize the numeric target that existed when the transition was collected. Terminal transitions require their own handling because they lack a continuation value.
To evaluate learning effects fairly, match environment interactions and update counts, specify buffer capacity and warmup, and report several seeds. Otherwise an apparent replay benefit may just reflect more optimization work. The adjacency count below is a diagnostic of sample order, not evidence of higher return or improved stability.
Replace one seed with all possible balanced label orders
There are C(12, 6) = 924 ways to arrange six A labels and six B labels. The extension enumerates every one and computes the same adjacent-match statistic. This is exact for label order: each label sequence corresponds to the same number, 6! × 6!, of permutations of the distinct transition IDs. Equal weighting over label sequences therefore matches a uniform permutation of identities.
The mean is exactly five matching pairs out of eleven, agreeing with the pairwise calculation 5/11. Only two arrangements have ten matching pairs: all A followed by all B, or the reverse. Their combined probability is 2/924, about 0.2165%. The original seeded shuffle gives four matches; it is one ordinary member of this distribution, not the evidence on which the conclusion rests.
Buffer composition changes the answer. If A contributes a transitions and B contributes b, the expected same-regime adjacent fraction is [a(a − 1) + b(b − 1)]/[N(N − 1)], with N = a + b. A highly imbalanced buffer remains locally repetitive even after a perfect shuffle. The explorer shows that curve for all thirteen A/B compositions of a twelve-transition buffer.
This exact enumeration is a diagnostic for sampling code, not a learning experiment. A tabular learner study would still need matched environment interactions and update budgets. But a sampler that fails the expected adjacency check on this controlled buffer should be fixed before attributing a downstream learning curve to an algorithmic improvement.