Research / Agent systems
A patch needs two test partitions
Fixing the reported issue and preserving ordinary behavior are separate obligations. Fifteen repairs make the difference visible.
A patch can fix the example in an issue while breaking the library's ordinary behavior. Replacing an absolute-value function with “always return 2” fixes the report that −2 produced −2. It is still a terrible repair. Evaluation needs to ask two questions: did the known failure become correct, and did previously correct behavior remain correct?
Inspect a case
Two partitions, one acceptance rule
SWE-bench distinguishes FAIL_TO_PASS tests from PASS_TO_PASS tests. A resolved instance requires all tests in both groups to pass; a missing required test is not counted as success. This article reproduces that acceptance logic on small functions, not the benchmark's repositories, infrastructure, or scores. SWE-bench, Appendix A.4
The fixture has three bugs. An absolute-value function incorrectly returns its input. A clamp intended to restrict values to [0, 10] handles the upper bound but not the lower bound. An average function divides by zero for an empty list, where this fixture's explicit contract specifies a result of zero. That empty-list convention is a local specification, not a universal mathematical definition.
Each bug receives five candidate implementations: the original, several plausible shortcuts, and a complete repair for the listed contract. There are fifteen candidates in total. Test functions are fixed before scoring. We call these candidates directly; nothing executes downloaded code or modifies an existing repository.
Inspect a deceptively good patch
For absolute value, the failing inputs are −2 and −3. Previously passing inputs are 0, 2, and 3. The candidate −x passes both newly failing cases, so a failure-only score is 2/2. It fails the positive regression cases, scoring only 1/3 on that partition. The correct abs function passes 2/2 and 3/3. A patch hardcoded to fix −2 fails the second failure case, exposing overfitting even before regressions are considered.
The clamp example adds another trap. max(0, x) fixes all negative inputs but loses the upper bound, so 15 is returned instead of 10. The expression max(0, min(x, 10)) retains both limits. For the average task, returning zero for every list fixes the empty input while destroying the ordinary mean of [2, 4].
The printed rows show the two counts and a resolved flag. Do not merge all tests into one average and accept a high percentage: passing many ordinary cases must not compensate for failing the reported issue, and fixing the issue must not excuse a regression.
A green matrix still has a boundary
Tests sample a specification. Even the candidate marked resolved here could fail an untested requirement involving nonnumeric values, floating-point behavior, or iterators. Passing the selected suite supports a bounded claim: no listed failure remains and no listed regression was introduced. It is not a proof of every possible program behavior.
The evaluator converts raised exceptions into failed tests so one candidate cannot prevent the rest from being inspected. In a real harness, crashes, missing results, and timeouts likewise need explicit failure treatment, with logs retained for diagnosis. Silently dropping them inflates success.
Boundary tests distinguish repair from memorization
Five additional assertions exercise a fractional absolute value, three fractional clamp boundaries, and the mean of a negative list. The complete implementations pass all five. These checks are kept separate from the original 15-candidate score so the displayed denominators remain stable.
The test matrix also shows why an all-tests average is a poor acceptance rule. A candidate can collect easy regression successes while leaving the issue unfixed, or repair two issue cases by negating every input. Requiring both partitions is a conjunction of obligations, not a weighted contest between them. Broader coverage strengthens each obligation; it does not change that logical structure.
Read the tested Python and recorded output
"""A patch needs two test partitions
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
tasks = {
'absolute': (
[(-2, 2), (-3, 3)], [(0, 0), (2, 2), (3, 3)],
[lambda x: x, lambda x: 2, lambda x: -x,
lambda x: 2 if x == -2 else x, abs]),
'clamp': (
[(-1, 0), (-5, 0)], [(0, 0), (5, 5), (10, 10), (15, 10)],
[lambda x: min(x, 10), lambda x: 0, lambda x: max(0, x),
lambda x: 0 if x == -1 else min(x, 10),
lambda x: max(0, min(x, 10))]),
'average': (
[([], 0)], [([2, 4], 3), ([1], 1), ([-2, 4], 1)],
[lambda x: sum(x) / len(x), lambda x: 0, lambda x: sum(x),
lambda x: x[0] if x else 0, lambda x: sum(x) / (len(x) or 1)]),
}
def passes(fn, case):
value, expected = case
try:
return fn(value) == expected
except Exception:
return False
resolved = 0
for task, (f2p, p2p, candidates) in tasks.items():
for index, fn in enumerate(candidates):
fixed = sum(passes(fn, case) for case in f2p)
kept = sum(passes(fn, case) for case in p2p)
ok = fixed == len(f2p) and kept == len(p2p)
resolved += ok
print(task, index, f'F2P={fixed}/{len(f2p)}',
f'P2P={kept}/{len(p2p)}', 'resolved=' + str(ok))
print('resolved_candidates', resolved, 'of', 15)
assert resolved == 3
labels={'absolute':['identity','constant 2','negate','special-case -2','absolute value'],'clamp':['upper bound only','constant 0','lower bound only','special-case -1','both bounds'],'average':['divide by length','constant 0','sum only','first element','empty-aware mean']}
rows=[]
for task,(f2p,p2p,candidates) in tasks.items():
for i,fn in enumerate(candidates):
results=[dict(partition=part,input=value,expected=expected,passed=passes(fn,(value,expected))) for part,caseset in [('F2P',f2p),('P2P',p2p)] for value,expected in caseset]
rows.append(dict(task=task,candidate=i,name=labels[task][i],tests=results,resolved=all(r['passed'] for r in results)))
held_out=[('absolute',-0.5,0.5),('clamp',-0.5,0),('clamp',10.5,10),('clamp',3.5,3.5),('average',[-4,-2],-3)]
for task,value,expected in held_out: assert passes(tasks[task][2][-1],(value,expected))
RESULTS=dict(candidates=rows,held_out=[dict(task=t,input=v,expected=e,passed=True) for t,v,e in held_out])
print('Held-out boundary checks:',len(held_out),'passed')
Recorded output
absolute 0 F2P=0/2 P2P=3/3 resolved=False absolute 1 F2P=1/2 P2P=1/3 resolved=False absolute 2 F2P=2/2 P2P=1/3 resolved=False absolute 3 F2P=1/2 P2P=3/3 resolved=False absolute 4 F2P=2/2 P2P=3/3 resolved=True clamp 0 F2P=0/2 P2P=4/4 resolved=False clamp 1 F2P=2/2 P2P=1/4 resolved=False clamp 2 F2P=2/2 P2P=3/4 resolved=False clamp 3 F2P=1/2 P2P=4/4 resolved=False clamp 4 F2P=2/2 P2P=4/4 resolved=True average 0 F2P=0/1 P2P=3/3 resolved=False average 1 F2P=1/1 P2P=0/3 resolved=False average 2 F2P=1/1 P2P=1/3 resolved=False average 3 F2P=1/1 P2P=1/3 resolved=False average 4 F2P=1/1 P2P=3/3 resolved=True resolved_candidates 3 of 15 Held-out boundary checks: 5 passed