{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Reliability Beyond One Successful Run\n",
        "\n",
        "All-k success and best-of-k success answer different questions. Exact estimators and synthetic trials show why task heterogeneity matters.\n",
        "\n",
        "Reproduction of pass^k metric on synthetic stochastic outcomes, not τ-bench agent scores.\n",
        "\n",
        "The code uses Python 3 and the standard library. Saved output is from the checked CPU run.\n"
      ],
      "id": "tau-bench-pass-k-reliability-0"
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "source": [
        "from math import comb\n",
        "from random import Random\n",
        "rng = Random(34)\n",
        "probabilities = {'uniform': [0.8] * 20, 'split': [0.99] * 10 + [0.55] * 10}\n",
        "counts = {}\n",
        "for name, probs in probabilities.items():\n",
        "    counts[name] = []\n",
        "    for task, probability in enumerate(probs):\n",
        "        successes = 0\n",
        "        for trial in range(10):\n",
        "            booking = {'id': task, 'status': 'pending'}\n",
        "            if rng.random() < probability:\n",
        "                booking['status'] = 'confirmed'\n",
        "            successes += booking['status'] == 'confirmed'\n",
        "        counts[name].append(successes)\n",
        "\n",
        "def score(cs, k, all_success=True):\n",
        "    assert 1 <= k <= 10\n",
        "    terms = [comb(c if all_success else 10-c, k) / comb(10, k) for c in cs]\n",
        "    return sum(terms) / len(terms) if all_success else 1 - sum(terms) / len(terms)\n",
        "\n",
        "def interval(cs, k):\n",
        "    boot_rng = Random(340 + k)\n",
        "    values = sorted(score([boot_rng.choice(cs) for _ in cs], k) for _ in range(500))\n",
        "    return values[12], values[487]\n",
        "\n",
        "for name, cs in counts.items():\n",
        "    print(name, 'successes_per_task', cs)\n",
        "    for k in (1, 2, 5):\n",
        "        lo, hi = interval(cs, k)\n",
        "        print(f'k={k} pass^k={score(cs,k):.3f} pass@k={score(cs,k,False):.3f}',\n",
        "              f'task_bootstrap_95pct=[{lo:.3f},{hi:.3f}]')\n",
        "assert abs(score([8], 2) - 28/45) < 1e-12\n"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "uniform successes_per_task [6, 10, 7, 8, 7, 10, 6, 9, 8, 5, 8, 9, 6, 8, 6, 9, 7, 7, 10, 7]\n",
            "k=1 pass^k=0.765 pass@k=0.765 task_bootstrap_95pct=[0.705,0.825]\n",
            "k=2 pass^k=0.589 pass@k=0.941 task_bootstrap_95pct=[0.491,0.694]\n",
            "k=5 pass^k=0.295 pass@k=1.000 task_bootstrap_95pct=[0.153,0.443]\n",
            "split successes_per_task [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 5, 5, 7, 4, 5, 5, 5, 6, 3, 7]\n",
            "k=1 pass^k=0.760 pass@k=0.760 task_bootstrap_95pct=[0.650,0.870]\n",
            "k=2 pass^k=0.629 pass@k=0.891 task_bootstrap_95pct=[0.471,0.802]\n",
            "k=5 pass^k=0.511 pass@k=0.994 task_bootstrap_95pct=[0.309,0.713]\n"
          ]
        }
      ],
      "id": "tau-bench-pass-k-reliability-1"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Sources\n",
        "\n",
        "- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains: https://arxiv.org/abs/2406.12045\n"
      ],
      "id": "tau-bench-pass-k-reliability-2"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}