Aryan PuttaDev JournalEvery Query Pays the Same Price, and That Is the Bug
One search platform, two tracks. Product search spends a latency budget across retrieval routes; web search spends a crawl budget across documents. Same controller. On 1,500 real Amazon ESCI queries the learned router matched full-rerank relevance at a fraction of the cost (+0.133 utility, 95% CI [0.127, 0.139]).
Where I Started
I was reading how Amazon evaluates product search and caught a mistake in my own retrieval code. The Shopping Queries dataset does not ask you to search a catalog of millions. It hands you a query and up to forty candidate products that humans already judged Exact, Substitute, Complement, or Irrelevant, and asks you to order those. My first evaluator scored each query against the whole product set and counted everything unjudged as irrelevant. That measures the wrong task. The real task is reranking a fixed candidate list. Once I fixed that, a bigger problem stood out: I was running the same retrieval pipeline on every query.
Three Queries, Three Right Answers
sony wh-1000xm5 is an exact model number. BM25 nails it in two milliseconds. comfortable shoes for nurses working long shifts is a descriptive need where lexical matching falls apart and you want semantic retrieval and a reranker. black headphones under 100 dollars without microphone has a price ceiling and a negation that no relevance score respects on its own. Three queries, three different right answers, and I was paying the most expensive pipeline on all of them. At search-engine scale that is the difference between holding your latency budget and blowing it.
The Number I Optimize
utility = nDCG@10 - latency_penalty x latency_ms - violation_penalty x constraints - timeout_penalty
The Bet
A controller picks one of four routes per query: lexical BM25, dense retrieval, hybrid fusion of the two, or a field-aware reranker. The bet is that the expensive route is only worth its cost on a minority of queries, so a model that reads the query and predicts the cheapest sufficient route can keep relevance high and latency low at the same time. Before building any serving infrastructure I wanted proof the routing had headroom, so I compared every policy against an oracle allowed to see the labels and pick the best route per query.
Try It: Route a Query
This is the routing logic from the project, running live. Type a query and watch which signals fire and which route the controller picks. The cheap lane is two milliseconds, the expensive reranker is thirty-five.
What I Measured
On 1,500 real ESCI test queries, running the reranker on everything bought 0.0045 nDCG over plain BM25 and cost seventeen times the latency. The oracle reached higher relevance than always-rerank at near-lexical cost by routing per query. A learned router trained only on query features then beat always-rerank on latency-aware utility by 0.133, with a 95% bootstrap confidence interval of 0.127 to 0.139 and a win/tie/loss of 449/0/12. The interval excludes zero, so the win is real and not noise.
The Uncomfortable Result
The router significantly beats always-rerank and the hand-written rules on utility. But on pure relevance it ties plain BM25. It learned the honest truth in my current setup: my dense route is a from-scratch TF-IDF cosine and my reranker is a field-aware boost, and on ESCI neither reliably beats BM25, so the smartest policy is to stay on the cheap route and pocket the latency. The router is correct. My routes are the weak part. That is not a failure to hide. It tells me exactly what to build next: real embeddings and a real cross-encoder, so the expensive routes finally earn their cost.
How It Serves
The router is trained in Python and written to a small file of weights. The online API is a Go service that loads that exact file, runs the same feature extraction, and applies the same decision, so the deployed router and the evaluated router are identical by construction. It also handles the request deadline: a query that routes to rerank under a tight deadline downgrades to the best route that fits, and if a downstream service is down it returns the strongest ranking it already has. The same model runs in your browser on the live demo, on the real weights, with nothing sent anywhere.
The Other Half: Google-Style Web Acquisition
Switchyard is two tracks, not one. Everything above is product search, where the scarce budget is query latency. The Google side spends a different scarce budget. You cannot crawl the whole web, so the real question is which pages are even worth fetching and indexing. The same controller runs here. A value-of-crawl model scores each candidate URL by expected relevance per unit fetch cost, an LLM acts as a judge to label page quality and train that model, and a budgeted selector picks the set that maximizes acquired relevance under a crawl budget. It is the product-search router with the axes relabeled: latency becomes crawl cost, and route choice becomes fetch or skip. This is the ML-driven web data acquisition problem real search teams live in, and it falls straight out of the same idea.
What I Took From It
The lesson generalizes past search. When a system cannot afford to do the expensive thing everywhere, do not do the expensive thing everywhere. Predict the value of each action, price it against the budget, and spend where the return is highest. Product search spends a latency budget across retrieval routes. Web data acquisition spends a crawl budget across documents, the same controller with the axes relabeled. Find the scarce resource, then spend it where it buys the most relevance.