Reliability Beyond One Successful Run

A system that solves a task once in ten attempts may be useful when every answer can be checked and retried. It is a different proposition when the same task must succeed every time a user asks. Those two requirements lead to different metrics, even when they use the same run outcomes.

All attempts versus at least one attempt

τ-bench defines pass^kThe chance that every one of k attempts at the same task succeeds. as the probability that all k independent trials of a task succeed, averaged across tasks. Its pass@kThe chance of finding at least one successful attempt among k tries at the same task. comparison instead asks whether at least one trial succeeds. Given n observed trials with c successes, the paper uses C(c, k) / C(n, k) for pass^k and 1 − C(n−c, k) / C(n, k) for pass@k. τ-bench, Section 3

The combinatorial expression counts subsets of observed trials. With eight successes among ten runs, there are C(10, 2) = 45 pairs, of which C(8, 2) = 28 contain two successes. The all-two estimate is therefore 28/45, about 0.622. The at-least-one estimate is 44/45, about 0.978. Reporting one in place of the other changes the story dramatically.

For k = 5, the same task's all-success estimate is 56/252, about 0.222. Raising the empirical 0.8 success rate to the fifth power gives 0.328 instead. The subset estimator avoids the finite-sample bias of that plug-in calculation under independent, identically distributed trials. It requires k no greater than the number of observed runs.

Preserve the task structure

The demonstration creates twenty synthetic booking records and ten trials per record. A successful scripted trial changes a local record to confirmed; a failed one leaves it pending. One policy succeeds with probability 0.8 on every task. Another has probability 0.99 on ten tasks and 0.55 on the remaining ten. A seeded pseudorandom generator produces the outcomes; no real booking is attempted.

Before sampling, the first policy's mean success probability is 0.8, compared with 0.77 for the second. But their theoretical all-five probabilities are approximately 0.328 and 0.501. Strong performance concentrated on a subset of tasks can raise all-k success even when the one-run average is lower. This is why a reliability curve should accompany per-task results rather than replace them.

The executed fixture gives the uniform policy pass^1 = 0.765 and pass^5 = 0.295; the split policy gives 0.760 and 0.511. These are seeded synthetic sample estimates, not benchmark measurements. The script calculates estimates for k equal to 1, 2, and 5. Its percentile intervals resample entire tasks, preserving each task's ten-run success count. Resampling all two hundred binary outcomes as though they came from one homogeneous task would erase the heterogeneity this comparison was designed to expose.

Interpret the interval narrowly

These bootstrap intervals describe variation from resampling this synthetic task set, conditional on its recorded trials. They do not cover every source of deployment uncertainty, and twenty tasks provide limited resolution. Correlated retries, changing policies, or shared environment failures also violate the simple independent-trial interpretation.

As k grows, all-k success is nonincreasing, while best-of-k success is nondecreasing. The opposite directions reflect the different questions: repeatability across every attempt, or finding one success among several. Keeping the metrics separate is essential even when they are calculated from the same trials.

from math import comb
from random import Random
rng = Random(34)
probabilities = {'uniform': [0.8] * 20, 'split': [0.99] * 10 + [0.55] * 10}
counts = {}
for name, probs in probabilities.items():
    counts[name] = []
    for task, probability in enumerate(probs):
        successes = 0
        for trial in range(10):
            booking = {'id': task, 'status': 'pending'}
            if rng.random() < probability:
                booking['status'] = 'confirmed'
            successes += booking['status'] == 'confirmed'
        counts[name].append(successes)

def score(cs, k, all_success=True):
    assert 1 <= k <= 10
    terms = [comb(c if all_success else 10-c, k) / comb(10, k) for c in cs]
    return sum(terms) / len(terms) if all_success else 1 - sum(terms) / len(terms)

def interval(cs, k):
    boot_rng = Random(340 + k)
    values = sorted(score([boot_rng.choice(cs) for _ in cs], k) for _ in range(500))
    return values[12], values[487]

for name, cs in counts.items():
    print(name, 'successes_per_task', cs)
    for k in (1, 2, 5):
        lo, hi = interval(cs, k)
        print(f'k={k} pass^k={score(cs,k):.3f} pass@k={score(cs,k,False):.3f}',
              f'task_bootstrap_95pct=[{lo:.3f},{hi:.3f}]')
assert abs(score([8], 2) - 28/45) < 1e-12

uniform successes_per_task [6, 10, 7, 8, 7, 10, 6, 9, 8, 5, 8, 9, 6, 8, 6, 9, 7, 7, 10, 7]
k=1 pass^k=0.765 pass@k=0.765 task_bootstrap_95pct=[0.705,0.825]
k=2 pass^k=0.589 pass@k=0.941 task_bootstrap_95pct=[0.491,0.694]
k=5 pass^k=0.295 pass@k=1.000 task_bootstrap_95pct=[0.153,0.443]
split successes_per_task [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 5, 5, 7, 4, 5, 5, 5, 6, 3, 7]
k=1 pass^k=0.760 pass@k=0.760 task_bootstrap_95pct=[0.650,0.870]
k=2 pass^k=0.629 pass@k=0.891 task_bootstrap_95pct=[0.471,0.802]
k=5 pass^k=0.511 pass@k=0.994 task_bootstrap_95pct=[0.309,0.713]