When benchmark averages reverse an agent ranking notebook

Python starts when you choose Run.

When benchmark averages reverse an agent ranking

An 83% overall score can lose to 80% under equal family weighting. The crossover can be calculated rather than argued over.

Aggregation and bootstrap demonstration on authored labels; not AgentBench model evaluation.

"""When benchmark averages reverse an agent ranking
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
from random import Random
counts = [('lookup', 80, 76, 64), ('editing', 10, 6, 8),
          ('planning', 5, 1, 4), ('recovery', 5, 0, 4)]
rows = {name: [(int(i < a), int(i < b)) for i in range(n)]
        for name, n, a, b in counts}

def macro(data, policy):
    return sum(sum(row[policy] for row in group) / len(group)
               for group in data.values()) / len(data)

for policy, name in enumerate(('A', 'B')):
    micro = sum(row[policy] for group in rows.values() for row in group) / 100
    print(name, f'micro={micro:.4f}', f'macro={macro(rows,policy):.4f}')
for name, n, a, b in counts:
    print(name, f'A={a}/{n}', f'B={b}/{n}')
for weight in (0, 0.25, 0.5, 0.75, 1):
    a = 0.95 * weight + ((0.6 + 0.2 + 0) / 3) * (1-weight)
    print(f'lookup_weight={weight:.2f}', f'A={a:.3f}', 'B=0.800')

rng = Random(35)
differences = []
for _ in range(1000):
    sample = {name: [rng.choice(group) for _ in group] for name, group in rows.items()}
    differences.append(macro(sample, 0) - macro(sample, 1))
differences.sort()
print('macro_A_minus_B_bootstrap_95pct',
      f'[{differences[24]:.3f},{differences[974]:.3f}]')
crossover = (0.8 - 0.8/3) / (0.95 - 0.8/3)
print('lookup_weight_crossover', f'{crossover:.4f}')
assert abs(macro(rows, 0) - 0.4375) < 1e-12

RESULTS=dict(families=[dict(name=n,tasks=t,A=a,B=b,A_rate=a/t,B_rate=b/t) for n,t,a,b in counts],paired_rows=rows,bootstrap_differences=differences,crossover=crossover,micro={'A':.83,'B':.80},macro={'A':macro(rows,0),'B':macro(rows,1)},sensitivity=[dict(lookup_weight=i/100,A=.95*i/100+(.8/3)*(1-i/100),B=.8) for i in range(101)])
assert abs(crossover-32/41)<1e-12
print('Analytic crossover: 32/41')

A micro=0.8300 macro=0.4375
B micro=0.8000 macro=0.8000
lookup A=76/80 B=64/80
editing A=6/10 B=8/10
planning A=1/5 B=4/5
recovery A=0/5 B=4/5
lookup_weight=0.00 A=0.267 B=0.800
lookup_weight=0.25 A=0.438 B=0.800
lookup_weight=0.50 A=0.608 B=0.800
lookup_weight=0.75 A=0.779 B=0.800
lookup_weight=1.00 A=0.950 B=0.800
macro_A_minus_B_bootstrap_95pct [-0.506,-0.203]
lookup_weight_crossover 0.7805
Analytic crossover: 32/41

Python runs locally in this page. Saved results and figures are visible before execution.

Download full experiment notebook