NEEDLE: Which search API moves the needle for AI agents?

We make each query set fresh from public sources: each hour, the other verticals each day.

For we sample papers from arXiv (computer, physical, life, and social sciences) and Europe PMC (health sciences). Four age bands reach from the last week back to approximately five years. Each paper gives one query per slice, so the four slices rank engines on the same papers. The gaps between the slices are the point: title vs body isolates full-text indexing, body vs clue isolates keyword-vs-prose handling, and tot isolates semantic retrieval.

We leak-check each LLM-written query: at least two content words must not appear in the title or the abstract. We keep a paper only if each slice produced a valid query. The known-item and tip-of-the-tongue framing follows Exa's "SOTA Search Over Academic Publications" (Jul 2026).

The stream merges search queries from three public agent-trajectory releases: the anonymized DeepResearchGym logs (14M+ search requests from real agentic clients; "Agentic Search in the Wild", SIGIR '26), the OpenResearcher deep-research trajectories (TIGER-AI-Lab, 2026), and LRAT-Train ("Learning to Retrieve from Agent Trajectories", SIGIR '26).

We send each engine in a run the same query text. We pin the per-engine request shapes in git (A5). The runner sends requests in sequence, one call at a time across the whole run. Thus the latency percentiles are comparable across engines, and no engine gets concurrent load. Engines see the live web minutes apart in a run, not at the same instant.

We judge an engine on its own ranking, titles, and snippets. We do not fetch pages and we do not re-rank. We cap the evidence at the same snippet length for each engine, so engines that return more content get no extra signal to the judge or the answer matchers. We judge each distinct combination of URL, title, snippet, and date once per query. We use that rating again where the identical result occurs for other engines (A4 gives the full judging setup).

We remove failed searches and judgements from the mean and report them as search_errors and judge_errors. An outage decreases num_scored and increases the error rate; it does not decrease the quality score.

We make fresh queries each hour and new known-answer queries each day from public sources (A1), so there is no fixed benchmark set to overfit. We archive the exact queries with each run (A6).

Each vertical also reports ultimate, a synthetic engine. It pools all engines' results per query and ranks them oracle-style: best judge rating first on the judged verticals, and the gold item first on the known-answer verticals. Its score shows what one engine could get if it always returned the best results that the full field found. This separates "the engine ranked poorly" from "no engine found better results". We do not include ultimate in the overlap and uniqueness statistics.

We judge with openai/gpt-5.6-terra via OpenRouter, temperature 0, one pointwise call per document. We record the model for each run as judge_model in its archived report. The judgement browser shows it when you select a run, so judge changes stay visible in the history.

The judge does not see the engine identity. The prompt contains the query, the document title, the URL, the published date, and the snippet — not the engine name. We judge each distinct combination of URL, title, snippet, and date once per query and use that rating again where the identical result occurs for other engines. We judge the same URL with a different snippet separately, with the same blind judge and rubric.

A judge preference for a snippet style would be a shared bias across engines. We publish the per-result reasoning with each run, so you can audit this.

Rubric: 0–4 Needs Met scale (FailsM / SM / HM / FullyM) with hard gates for a wrong entity, stale coverage of dated events, and search-operator compliance ("quoted phrases", site:). The full prompt is one template in the repo: src/needle/shared/prompts/judgement.jinja.

Known limitation: we operate NEEDLE ourselves, and we did not yet do a systematic human-agreement audit of the judge. But you can check the numbers: the judge is blind and shared across engines, each rating includes its reasoning, and we publish the queries, prompts, and raw results. Thus anyone can score any rating again.