Rewarding an agent for approaching a goal seems harmless until the agent discovers that collecting the approach reward is better than finishing. Before using a bonus to make learning easier, check whether it changes which behavior the task actually prefers.
Ng, Harada, and Russell identify potential-based shaping as a policy-preserving reward transformation under the paper's assumptions. Its transition bonus has the form gamma × Phi(next_state) − Phi(current_state). The discount factor belongs inside the bonus; an ordinary difference of potentials is not interchangeable with it in a discounted task. Reward-shaping paper
Ng, Harada, and Russell's result is stronger than one successful shaped reward: potential-based transformations provide the general policy-invariance guarantee, and the paper also establishes necessity when no additional assumptions about the MDP are available. The complete-path calculation below exposes the cancellation that makes the guarantee possible.
Compare two routes before training anything
Construct a tiny deterministic environment with start S, intermediate state M, and terminal goal G. One action goes directly from S to G and receives reward 1. Another goes from S to M for reward 0, then M to G for reward 1. Use discount gamma = 0.9.
The direct route returns 1. The detour returns 0 + 0.9 × 1 = 0.9, so the original task prefers finishing immediately.
Now add a bonus of 2 whenever the agent enters M. The detour returns 2.9, beating the direct route's 1. The learner would be correct to take the detour under this modified reward. Calling that behavior a learning failure misses the specification error.
Instead choose potentials Phi(S) = −2, Phi(M) = −1, and Phi(G) = 0. The direct transition gains a shaping term of 2, producing return 3. The detour's first shaped reward is 0 + 0.9 × (−1) − (−2) = 1.1. Its second is 1 + 0.9 × 0 − (−1) = 2. Discounting gives 1.1 + 0.9 × 2 = 2.9.
Both paths gained exactly 2 relative to their original returns. Their ranking stays the same.
The cancellation is the useful test
Summing discounted shaping terms along a finite trajectory cancels the intermediate potentials. The remaining adjustment is −Phi(start) + gamma^T × Phi(terminal). Setting terminal potential to zero makes the adjustment independent of the route in this episodic example.
This suggests a practical unit test before running a learning algorithm: enumerate a few complete paths and compare shaped-minus-original returns. If paths from the same start receive different offsets, inspect discounting, terminal handling, or the proposed bonus.
A time-limit truncation is especially easy to mishandle. Treating an unfinished state as a true terminal state can change the boundary term. The correct treatment depends on the modeled task and how continuation values are bootstrapped.
The accompanying code evaluates routes exactly; it does not claim faster learning or run a gridworld training experiment. Preserving the optimal policy says nothing by itself about finite-sample speed, exploration coverage, or approximation error. A poor potential can still provide unhelpful guidance during learning.
Make the tempting detour arbitrarily long
The extended test adds zero through twelve self-loops at M before the terminal transition. Every self-loop has zero original reward. The original discounted return decreases as the goal is delayed. Potential-shaped return decreases in exactly the same order because every complete path still gains the same +2 offset. The code checks that invariant for all thirteen paths within a tolerance of 10⁻¹².
The arbitrary entry bonus behaves differently. It pays two units on every transition back into M, including a self-loop. With twelve extra loops, the original return is about 0.254187, while the bonus-augmented return is about 15.170455. The agent can improve its stated objective by postponing the goal. A learner that discovers this is exploiting the reward definition, not miscomputing the discount.
The test also evaluates the tempting but incorrect replacement Φ(next) − Φ(current). Its discounted path offsets are no longer constant. The right unit test is not whether the shaped scores look plausible or whether one favored trajectory still wins. It is whether the transformation has the boundary cancellation required for the intended guarantee.
Finally, ending the return calculation after S → M leaves an offset of 1.1 rather than 2. The extra term is γΦ(M), because M is not the true zero-potential terminal. For truncated rollouts, continuation values and terminal flags must agree with the task definition. Otherwise an implementation can use the correct local formula and still optimize the wrong finite-horizon objective.