Research / Agent systems
Retrieval relevance is not constraint satisfaction
A higher-ranked product can violate a hard requirement. Validation helps only if retrieval includes a feasible alternative.
A jacket can match every word in a search query and still be the wrong purchase. The size may be unavailable, the color may differ, or the price may exceed the user's ceiling. Retrieval answers “which items are relevant enough to inspect?” Selection must answer the stricter question “which available option satisfies every required constraint?”
Inspect a case
Preserve the distinction between item and option
WebShop models shopping as searching, opening products, selecting options, and ending with a chosen item. Its evaluation accounts for product attributes, options, and price, and distinguishes partial reward from complete success. The fixture below borrows the constraint-checking problem, not its full reward implementation or learned agents. WebShop, Section 3.1
We create twenty-five synthetic model names, each with two catalog items. One is a keyword-rich decoy; the other is a valid but tersely described alternative. Fifty items therefore support twenty-five requests. Each request specifies a model token, a color, a size, and a maximum price. All constraints are already structured so that parsing natural language is not part of this experiment.
Retrieval first keeps items containing the exact model token. It then ranks those candidates by the number of query words in the title. The decoy contains “light trail jacket” and the model name; the valid alternative contains only “jacket” and the model name. Keyword ranking consistently prefers the decoy, isolating the gap between relevance and feasibility.
Evaluate the conjunction, not a vague match
For one request, suppose the desired model is model2, the color is green, the size is XS, and the maximum price is 22. The decoy's title matches four query terms, but its price is 23. The alternative matches two terms and costs 17. If the price ceiling is a hard constraintA requirement every acceptable candidate must satisfy., choosing the first item is incorrect regardless of its superior keyword score.
The checker evaluates color equality, size equality, and price <= maximum. It selects the highest-ranked candidate for which all three conditions hold. The synthetic decoys violate exactly one condition each: nine have the wrong color, eight the wrong size, and eight exceed the budget. The baseline therefore completes none of the twenty-five requests, while the checked selector completes all twenty-five.
Those extreme totals are intentional counterexample construction, not estimates of search-engine quality or agent ability. The checker is given accurate structured attributes, and every valid item is in the retrieved candidate set. Those assumptions remove two major practical difficulties: extracting constraints and retrieving a feasible option in the first place.
Keep abstention and missing data explicit
If no candidate satisfies the request, the correct return value is “no feasible candidate found,” not the least bad item disguised as success. That does not prove no valid product exists anywhere; it describes the searched candidate set. Missing size or price fields should likewise be treated as unknown until resolved, not silently filled with favorable values.
Availability and final price can change between selection and commitment. A live workflow needs a final recheck because the stored attributes can become stale.
Test the retrieval cutoff
The additional test truncates the ranked candidate set before constraint checking. With only the top result available, validation finds no feasible option in all 25 requests. Allowing the second result restores all 25 completions. The checker has not changed; the information made available to it has.
This separates two useful failure reports: “the retrieved candidates violate the request” and “no feasible product exists.” Only the first is justified by a top-one search. Expanding retrieval, resolving an unknown attribute, and revising a hard constraint are different next steps. A selection system should retain enough evidence to tell which step is warranted.
Read the tested Python and recorded output
"""Retrieval relevance is not constraint satisfaction
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
colors = ['red', 'blue', 'green', 'black', 'white']
sizes = ['XS', 'S', 'M', 'L', 'XL']
catalog, queries = [], []
for i in range(25):
query = dict(model=f'model{i}', color=colors[i % 5], size=sizes[i // 5], budget=20+i)
queries.append(query)
valid = dict(title=f'jacket model{i}', color=query['color'], size=query['size'], price=15+i)
decoy = dict(valid, title=f'light trail jacket model{i}')
if i % 3 == 0:
decoy['color'] = colors[(i+1) % 5]
elif i % 3 == 1:
decoy['size'] = sizes[(i // 5 + 1) % 5]
else:
decoy['price'] = query['budget'] + 1
catalog.extend([decoy, valid])
def violations(item, query):
return [key for key in ('color', 'size') if item[key] != query[key]] + (
['price'] if item['price'] > query['budget'] else [])
hits = dict(keyword=0, checked=0)
failures = dict(color=0, size=0, price=0)
for query in queries:
words = {'light', 'trail', 'jacket', query['model']}
candidates = [p for p in catalog if query['model'] in p['title'].split()]
ranked = sorted(candidates, key=lambda p: -len(words & set(p['title'].split())))
baseline = ranked[0]
checked = next((p for p in ranked if not violations(p, query)), None)
hits['keyword'] += not violations(baseline, query)
hits['checked'] += checked is not None and not violations(checked, query)
for reason in violations(baseline, query):
failures[reason] += 1
print('catalog_items', len(catalog), 'queries', len(queries))
print('completed', hits)
print('keyword_constraint_violations', failures)
assert hits == dict(keyword=0, checked=25)
rows=[]
for q in queries:
words={'light','trail','jacket',q['model']}
ranked=sorted([p for p in catalog if q['model'] in p['title'].split()],key=lambda p:-len(words & set(p['title'].split())))
rows.append(dict(query=q,candidates=[dict(p,rank=i+1,score=len(words&set(p['title'].split())),violations=violations(p,q)) for i,p in enumerate(ranked)]))
# A validator cannot recover an option omitted from the candidate set.
cutoff=[dict(k=k,completed=sum(any(not violations(p,q) for p in sorted([p for p in catalog if q['model'] in p['title'].split()],key=lambda p:-len({'light','trail','jacket',q['model']}&set(p['title'].split())))[:k]) for q in queries)) for k in (1,2)]
assert [r['completed'] for r in cutoff]==[0,25]
RESULTS=dict(queries=rows,completed=hits,violation_counts=failures,retrieval_cutoff=cutoff)
print('Feasible selections at retrieval cutoffs:',cutoff)
Recorded output
catalog_items 50 queries 25
completed {'keyword': 0, 'checked': 25}
keyword_constraint_violations {'color': 9, 'size': 8, 'price': 8}
Feasible selections at retrieval cutoffs: [{'k': 1, 'completed': 0}, {'k': 2, 'completed': 25}]