Retrieval relevance is not constraint satisfaction notebook

Python starts when you choose Run.

Retrieval relevance is not constraint satisfaction

A higher-ranked product can violate a hard requirement. Validation helps only if retrieval includes a feasible alternative.

Offline retrieval-versus-feasibility counterexample with exact attributes; no WebShop run or purchase.

"""Retrieval relevance is not constraint satisfaction
Deterministic, standard-library experiment. RESULTS contains the full data.
"""
colors = ['red', 'blue', 'green', 'black', 'white']
sizes = ['XS', 'S', 'M', 'L', 'XL']
catalog, queries = [], []
for i in range(25):
    query = dict(model=f'model{i}', color=colors[i % 5], size=sizes[i // 5], budget=20+i)
    queries.append(query)
    valid = dict(title=f'jacket model{i}', color=query['color'], size=query['size'], price=15+i)
    decoy = dict(valid, title=f'light trail jacket model{i}')
    if i % 3 == 0:
        decoy['color'] = colors[(i+1) % 5]
    elif i % 3 == 1:
        decoy['size'] = sizes[(i // 5 + 1) % 5]
    else:
        decoy['price'] = query['budget'] + 1
    catalog.extend([decoy, valid])

def violations(item, query):
    return [key for key in ('color', 'size') if item[key] != query[key]] + (
        ['price'] if item['price'] > query['budget'] else [])

hits = dict(keyword=0, checked=0)
failures = dict(color=0, size=0, price=0)
for query in queries:
    words = {'light', 'trail', 'jacket', query['model']}
    candidates = [p for p in catalog if query['model'] in p['title'].split()]
    ranked = sorted(candidates, key=lambda p: -len(words & set(p['title'].split())))
    baseline = ranked[0]
    checked = next((p for p in ranked if not violations(p, query)), None)
    hits['keyword'] += not violations(baseline, query)
    hits['checked'] += checked is not None and not violations(checked, query)
    for reason in violations(baseline, query):
        failures[reason] += 1
print('catalog_items', len(catalog), 'queries', len(queries))
print('completed', hits)
print('keyword_constraint_violations', failures)
assert hits == dict(keyword=0, checked=25)

rows=[]
for q in queries:
    words={'light','trail','jacket',q['model']}
    ranked=sorted([p for p in catalog if q['model'] in p['title'].split()],key=lambda p:-len(words & set(p['title'].split())))
    rows.append(dict(query=q,candidates=[dict(p,rank=i+1,score=len(words&set(p['title'].split())),violations=violations(p,q)) for i,p in enumerate(ranked)]))
# A validator cannot recover an option omitted from the candidate set.
cutoff=[dict(k=k,completed=sum(any(not violations(p,q) for p in sorted([p for p in catalog if q['model'] in p['title'].split()],key=lambda p:-len({'light','trail','jacket',q['model']}&set(p['title'].split())))[:k]) for q in queries)) for k in (1,2)]
assert [r['completed'] for r in cutoff]==[0,25]
RESULTS=dict(queries=rows,completed=hits,violation_counts=failures,retrieval_cutoff=cutoff)
print('Feasible selections at retrieval cutoffs:',cutoff)

catalog_items 50 queries 25
completed {'keyword': 0, 'checked': 25}
keyword_constraint_violations {'color': 9, 'size': 8, 'price': 8}
Feasible selections at retrieval cutoffs: [{'k': 1, 'completed': 0}, {'k': 2, 'completed': 25}]

Python runs locally in this page. Saved results and figures are visible before execution.

Download full experiment notebook