Python starts when you choose Run.
What PPO Clipping Actually Limits
The clipped surrogate removes incentives in one direction at a time. It does not enforce a hard probability-ratio or KL bound.
Original objective-value and categorical-KL calculation; no PPO training, optimizer run, or control benchmark.
The code uses Python 3 and the standard library. Saved output is from the checked CPU run.
r=0.6: A=+2 -> 1.2, A=-2 -> -1.6 r=0.8: A=+2 -> 1.6, A=-2 -> -1.6 r=1.0: A=+2 -> 2.0, A=-2 -> -2.0 r=1.2: A=+2 -> 2.4, A=-2 -> -2.4 r=1.4: A=+2 -> 2.4, A=-2 -> -2.8 candidate_ratio=1.80, objective=1.20 KL(old||new)=0.510826 nats
Sources
- Proximal Policy Optimization Algorithms: https://arxiv.org/pdf/1707.06347
Python runs locally in this page. Saved results and figures are visible before execution.