{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# What PPO Clipping Actually Limits\n",
        "\n",
        "The clipped surrogate removes incentives in one direction at a time. It does not enforce a hard probability-ratio or KL bound.\n",
        "\n",
        "Original objective-value and categorical-KL calculation; no PPO training, optimizer run, or control benchmark.\n",
        "\n",
        "The code uses Python 3 and the standard library. Saved output is from the checked CPU run.\n"
      ],
      "id": "ppo-clipping-probability-ratio-0"
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "source": [
        "from math import log\n",
        "\n",
        "epsilon = 0.2\n",
        "\n",
        "def clipped_objective(ratio, advantage):\n",
        "    clipped_ratio = min(1 + epsilon, max(1 - epsilon, ratio))\n",
        "    return min(ratio * advantage, clipped_ratio * advantage)\n",
        "\n",
        "for ratio in (0.6, 0.8, 1.0, 1.2, 1.4):\n",
        "    positive = clipped_objective(ratio, 2.0)\n",
        "    negative = clipped_objective(ratio, -2.0)\n",
        "    print(f\"r={ratio:.1f}: A=+2 -> {positive:.1f}, A=-2 -> {negative:.1f}\")\n",
        "\n",
        "old = [0.5, 0.5]\n",
        "new = [0.9, 0.1]\n",
        "assert abs(sum(old) - 1) < 1e-12\n",
        "assert abs(sum(new) - 1) < 1e-12\n",
        "ratio = new[0] / old[0]\n",
        "kl = sum(p * log(p / q) for p, q in zip(old, new))\n",
        "assert ratio > 1 + epsilon\n",
        "assert abs(clipped_objective(ratio, 1.0) - 1.2) < 1e-12\n",
        "assert kl > 0.5\n",
        "print(f\"candidate_ratio={ratio:.2f}, objective={clipped_objective(ratio, 1):.2f}\")\n",
        "print(f\"KL(old||new)={kl:.6f} nats\")\n"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "r=0.6: A=+2 -> 1.2, A=-2 -> -1.6\n",
            "r=0.8: A=+2 -> 1.6, A=-2 -> -1.6\n",
            "r=1.0: A=+2 -> 2.0, A=-2 -> -2.0\n",
            "r=1.2: A=+2 -> 2.4, A=-2 -> -2.4\n",
            "r=1.4: A=+2 -> 2.4, A=-2 -> -2.8\n",
            "candidate_ratio=1.80, objective=1.20\n",
            "KL(old||new)=0.510826 nats\n"
          ]
        }
      ],
      "id": "ppo-clipping-probability-ratio-1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Sources\n",
        "\n",
        "- Proximal Policy Optimization Algorithms: https://arxiv.org/pdf/1707.06347\n"
      ],
      "id": "ppo-clipping-probability-ratio-2"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}