Imagine learning to remain on the centerline of a simulated path by watching an expert that never leaves it. The training set says “at offset zero, take action zero.” That can be perfectly accurate and still fail to answer the first interesting question: what should happen after an unexpected displacement?
DAgger addresses the changing state distribution in imitation learning. It rolls out a policy, obtains expert actions for states encountered during that rollout, aggregates those labeled examples with earlier data, and trains again. Its general formulation can mix expert and learner actions during collection. DAgger, Section 3 and Algorithm 3.1.
DAgger's contribution is a no-regret reduction with guarantees under its assumptions, supported by imitation and sequence-labeling experiments. Its data aggregation changes the distribution on which the learner receives expert labels. The lookup-table example tests that distributional mechanism without conflating it with neural-network fitting or a claim about physical driving.
A tiny world with a missing recovery rule
Our original simulator uses an integer lateral offset x. An action is −1, 0, or +1; the next offset equals the current offset plus the action and a disturbance. The expert moves one unit toward zero whenever the offset is nonzero. This is deliberately a bounded arithmetic environment, not a vehicle model.
The initial learner is a lookup table containing only {0: 0}. For unseen states it chooses zero. During an eight-step rollout, a +1 disturbance arrives at step two and a −1 disturbance at step five. Starting at zero, the learner reaches offset +1 after the first disturbance. It has no labeled recovery action there, so it remains displaced until the later disturbance happens to cancel the error.
Notice that more copies of the original centerline example would not change the lookup table. The problem is coverage, not optimization effort. A dataset can grow in row count without growing in the kinds of decisions it teaches.
Keep labels and executed actions separate
The code records each state before executing the learner's action. After the rollout, it asks the expert for the desired action at every recorded state and adds those pairs to the table. The action that produced a state is not automatically the correct label for that state. Confusing the two would teach the learner to imitate its own mistakes.
On the next rollout, the table knows how to correct +1. That correction also changes which later states appear. In particular, the negative disturbance can now expose offset −1 rather than merely canceling an existing positive offset. A later aggregation pass can add the missing opposite-direction recovery action. This is the mechanism worth watching: learning one correction can reveal another region of the state space.
The reported error is the sum of absolute post-transition offsets. In the executed example it moves from 3 to 4 to 2: fixing one recovery direction temporarily worsens the total before the opposite direction is learned. This is a diagnostic of these exact rollouts, not a statistical estimate of performance. The script prints visited states and the accumulated table so the change remains inspectable. No optimizer or neural network is needed for this illustration.
Test beyond the collected states
The final check begins at offset two. If that state never appeared during collection, the default action still applies. A successful centered rollout does not establish recovery from arbitrary offsets. The figure overlays the successive offset traces and tests a range of initial offsets separately.
A larger simulation study could use a function approximator, vary disturbances and initial offsets, and hold out entire disturbance sequences. Expert queries should be counted as a resource, not treated as free. The lesson here is more specific than “collect more data”: collect correct labels for the situations that the learner itself makes likely, and then test whether that coverage extends to the situations you actually care about.
Exhaust the small test distribution
A single training rollout can hide both improvement and missing coverage. The extension evaluates each of the three learned tables on every length-six sequence of disturbances drawn from {−1, 0, +1}. There are 3⁶ = 729 sequences for each initial state. Starting offsets range from −3 through +3, and all sequences receive equal weight. This is an exhaustive finite stress test, not an estimate of how frequently real disturbances occur.
From a centered start, mean accumulated absolute offset falls from about 6.9712 to 5.9300 to 4.0000 across the three tables. The final table has no unknown-state decisions in those centered tests. Once it knows how to correct ±1, every next state after a bounded unit disturbance remains in {−1, 0, +1}. That closed set explains the coverage result; it is not mysterious generalization.
Starting at ±2 or ±3 breaks the argument. The lookup table still has no recovery action there, and its default action can leave the trajectory displaced until a disturbance happens to move it into a labeled region. The heatmap shows those initial states separately instead of averaging them into the favorable centered result.
The training trace itself remains nonmonotonic: errors 3, 4, then 2. Correcting the first positive displacement exposes a negative state that the previous rollout never visited. Distinguishing that newly exposed failure from a regression in the known correction is the useful debugging step. State coverage, expert-query count, and held-out conditions should accompany a single aggregate imitation-loss number.