What PPO Clipping Actually Limits

If PPO uses a clip range of 0.2, can an action's probability move by more than 20%? Yes. The clippingA training-objective rule that stops rewarding some overly large policy changes. modifies a training objective; it does not project every updated policy back into a guaranteed probability-ratio interval.

PPO's clipped surrogate compares the ordinary probability-ratio-weighted advantage with a clipped version and takes the smaller value. The paper uses this construction while performing multiple minibatch optimization epochs on collected data. The sign of the advantage determines which direction loses additional incentive beyond the clipping threshold. PPO paper, Section 3

Read one term carefully

Let r be new_probability / old_probability for a sampled action, A its estimated advantage, and epsilon = 0.2. The objective term is min(rA, clip(r, 0.8, 1.2)A). The objective is maximized.

For A = 2 and r = 1.4, the unclipped term is 2.8 while the clipped term is 2.4, so the result is 2.4. Increasing that action's probability further provides no extra benefit through this individual term.

But if r = 0.6, the terms are 1.2 and 1.6, so the objective keeps 1.2. Reducing the probability of a positively advantaged action remains penalized. Clipping does not flatten both sides symmetrically.

Now use A = −2. At r = 0.6, the terms are −1.2 and −1.6, and the result is −1.6. The incentive to keep decreasing the action's probability has flattened. At r = 1.4, the result is −2.8: increasing the probability of a negatively advantaged action still makes the objective worse.

The minimum is essential. Clipping r first and multiplying by A without retaining the unclipped alternative would create a different objective.

A counterexample to the hard-bound interpretation

Take an old two-action policy [0.5, 0.5] and a candidate policy [0.9, 0.1]. For the first action, r = 1.8, clearly outside [0.8, 1.2]. With advantage 1, its clipped objective term is still perfectly well-defined: 1.2.

The code calculates the full old-to-new KL divergence for these two distributions, approximately 0.510826 nats. This candidate is not claimed to be the outcome of a particular optimizer run. It demonstrates that the clipping formula itself does not reject or make an out-of-range policy impossible.

Actual updates involve many samples, shared parameters, optimizer state, and potentially additional loss terms. A change encouraged by one sample can move another sample's probability after its own term has flattened. Unseen states have no directly evaluated clipping term at all.

For a training experiment, log an explicit policy-change statistic alongside return, clip fraction, learning rate, and update epochs. A small-looking clip parameter cannot replace those observations. The CPU example evaluates objective values only; it provides no control-task performance result.

from math import log

epsilon = 0.2

def clipped_objective(ratio, advantage):
    clipped_ratio = min(1 + epsilon, max(1 - epsilon, ratio))
    return min(ratio * advantage, clipped_ratio * advantage)

for ratio in (0.6, 0.8, 1.0, 1.2, 1.4):
    positive = clipped_objective(ratio, 2.0)
    negative = clipped_objective(ratio, -2.0)
    print(f"r={ratio:.1f}: A=+2 -> {positive:.1f}, A=-2 -> {negative:.1f}")

old = [0.5, 0.5]
new = [0.9, 0.1]
assert abs(sum(old) - 1) < 1e-12
assert abs(sum(new) - 1) < 1e-12
ratio = new[0] / old[0]
kl = sum(p * log(p / q) for p, q in zip(old, new))
assert ratio > 1 + epsilon
assert abs(clipped_objective(ratio, 1.0) - 1.2) < 1e-12
assert kl > 0.5
print(f"candidate_ratio={ratio:.2f}, objective={clipped_objective(ratio, 1):.2f}")
print(f"KL(old||new)={kl:.6f} nats")

r=0.6: A=+2 -> 1.2, A=-2 -> -1.6
r=0.8: A=+2 -> 1.6, A=-2 -> -1.6
r=1.0: A=+2 -> 2.0, A=-2 -> -2.0
r=1.2: A=+2 -> 2.4, A=-2 -> -2.4
r=1.4: A=+2 -> 2.4, A=-2 -> -2.8
candidate_ratio=1.80, objective=1.20
KL(old||new)=0.510826 nats