Selective tool calls: finding the cost threshold notebook

Python starts when you choose Run.

Selective tool calls: finding the cost threshold

A carry-detection fixture makes the error–latency trade-off explicit, including the point at which a perfect calculator stops being worthwhile.

Synthetic routing utility calculation; no learned Toolformer, model confidence, or measured tool latency.

"""Selective tool calls: finding the cost threshold
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
TASKS = [(a, b) for a in range(10, 20) for b in range(20, 25)]

def limited_solver(a, b):
    tens = 10 * (a // 10 + b // 10)
    units = (a % 10 + b % 10) % 10
    return tens + units

def evaluate(threshold):
    calls = errors = 0
    for a, b in TASKS:
        feature = a % 10 + b % 10
        use_tool = feature >= threshold
        answer = a + b if use_tool else limited_solver(a, b)
        calls += use_tool
        errors += answer != a + b
    return calls, errors

print('n_tasks', len(TASKS), 'latency is simulated, not measured')
for latency_ms in (0, 250, 5000):
    cost_per_call = 0.02 + 0.2 * latency_ms / 1000
    for threshold in (0, 8, 10, 12, 15):
        calls, errors = evaluate(threshold)
        objective = errors / 50 + calls / 50 * cost_per_call
        print(f'ms={latency_ms} threshold={threshold} calls={calls}',
              f'errors={errors} objective={objective:.3f}')
assert evaluate(10) == (10, 0)
assert evaluate(12) == (3, 7)
assert limited_solver(19, 24) == 33

thresholds=[]
for threshold in range(16):
    calls,errors=evaluate(threshold)
    thresholds.append(dict(threshold=threshold,calls=calls,errors=errors,call_rate=calls/50,error_rate=errors/50))
cases=[dict(a=a,b=b,units_sum=a%10+b%10,limited=limited_solver(a,b),truth=a+b,wrong=limited_solver(a,b)!=a+b) for a,b in TASKS]
crossover_ms=round((1-0.02)/0.2*1000, 9)
assert crossover_ms==4900
assert abs((10/50)*(0.02+0.2*crossover_ms/1000)-10/50)<1e-12
RESULTS=dict(tasks=cases,thresholds=thresholds,call_cost_fixed=0.02,call_cost_per_second=0.2,crossover_ms=crossover_ms)
print('Threshold 10 / never-call indifference:',int(crossover_ms),'simulated ms')

n_tasks 50 latency is simulated, not measured
ms=0 threshold=0 calls=50 errors=0 objective=0.020
ms=0 threshold=8 calls=20 errors=0 objective=0.008
ms=0 threshold=10 calls=10 errors=0 objective=0.004
ms=0 threshold=12 calls=3 errors=7 objective=0.141
ms=0 threshold=15 calls=0 errors=10 objective=0.200
ms=250 threshold=0 calls=50 errors=0 objective=0.070
ms=250 threshold=8 calls=20 errors=0 objective=0.028
ms=250 threshold=10 calls=10 errors=0 objective=0.014
ms=250 threshold=12 calls=3 errors=7 objective=0.144
ms=250 threshold=15 calls=0 errors=10 objective=0.200
ms=5000 threshold=0 calls=50 errors=0 objective=1.020
ms=5000 threshold=8 calls=20 errors=0 objective=0.408
ms=5000 threshold=10 calls=10 errors=0 objective=0.204
ms=5000 threshold=12 calls=3 errors=7 objective=0.201
ms=5000 threshold=15 calls=0 errors=10 objective=0.200
Threshold 10 / never-call indifference: 4900 simulated ms

Python runs locally in this page. Saved results and figures are visible before execution.

Download full experiment notebook