Reliability Beyond One Successful Run notebook

Python starts when you choose Run.

Reliability Beyond One Successful Run

All-k success and best-of-k success answer different questions. Exact estimators and synthetic trials show why task heterogeneity matters.

Reproduction of pass^k metric on synthetic stochastic outcomes, not τ-bench agent scores.

The code uses Python 3 and the standard library. Saved output is from the checked CPU run.

from math import comb
from random import Random
rng = Random(34)
probabilities = {'uniform': [0.8] * 20, 'split': [0.99] * 10 + [0.55] * 10}
counts = {}
for name, probs in probabilities.items():
    counts[name] = []
    for task, probability in enumerate(probs):
        successes = 0
        for trial in range(10):
            booking = {'id': task, 'status': 'pending'}
            if rng.random() < probability:
                booking['status'] = 'confirmed'
            successes += booking['status'] == 'confirmed'
        counts[name].append(successes)

def score(cs, k, all_success=True):
    assert 1 <= k <= 10
    terms = [comb(c if all_success else 10-c, k) / comb(10, k) for c in cs]
    return sum(terms) / len(terms) if all_success else 1 - sum(terms) / len(terms)

def interval(cs, k):
    boot_rng = Random(340 + k)
    values = sorted(score([boot_rng.choice(cs) for _ in cs], k) for _ in range(500))
    return values[12], values[487]

for name, cs in counts.items():
    print(name, 'successes_per_task', cs)
    for k in (1, 2, 5):
        lo, hi = interval(cs, k)
        print(f'k={k} pass^k={score(cs,k):.3f} pass@k={score(cs,k,False):.3f}',
              f'task_bootstrap_95pct=[{lo:.3f},{hi:.3f}]')
assert abs(score([8], 2) - 28/45) < 1e-12

uniform successes_per_task [6, 10, 7, 8, 7, 10, 6, 9, 8, 5, 8, 9, 6, 8, 6, 9, 7, 7, 10, 7]
k=1 pass^k=0.765 pass@k=0.765 task_bootstrap_95pct=[0.705,0.825]
k=2 pass^k=0.589 pass@k=0.941 task_bootstrap_95pct=[0.491,0.694]
k=5 pass^k=0.295 pass@k=1.000 task_bootstrap_95pct=[0.153,0.443]
split successes_per_task [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 5, 5, 7, 4, 5, 5, 5, 6, 3, 7]
k=1 pass^k=0.760 pass@k=0.760 task_bootstrap_95pct=[0.650,0.870]
k=2 pass^k=0.629 pass@k=0.891 task_bootstrap_95pct=[0.471,0.802]
k=5 pass^k=0.511 pass@k=0.994 task_bootstrap_95pct=[0.309,0.713]

Sources

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains: https://arxiv.org/abs/2406.12045

Python runs locally in this page. Saved results and figures are visible before execution.

Download full experiment notebook