How we measure prior-art retrieval

The pinned holdout

Holdout v1 is 97,913 held-out claim queries drawn from applications filed in 2022 and 2023, scored against a 609,723-patent corpus of examiner-cited art. It was frozen in June 2026 and carries the fingerprint 6fd3de94f3b9.

The fingerprint is doing real work. Every run records the fingerprint of the data it scored, and a scorecard whose fingerprint does not match the registry is void. Without that rule the cheapest way to raise a benchmark number is to rebuild the benchmark, and nobody reading the result can tell that is what happened.

Two more rules follow from it. A sampled run is labelled SAMPLE and is never quoted outside the company. A published number is the full query set, scored once, with no reruns until one comes out well.

Keeping the answers out of training

A random document-level split leaks badly on patents, for two reasons that have nothing to do with the model.

The first is families. One invention is filed in several offices and often several times in one office, and those siblings share a specification almost word for word. Split by document and a test query’s twin sits in training, so the encoder has already been shown the answer in a slightly different wrapper. The split is therefore family-aware on DOCDB familyId: a family is assigned to training or to test as a unit, never divided.

The second is the citation graph. Examiner citations are dense and reciprocal, so a query’s cited art tends to cite back and to share citations with its neighbours. The split is citation-graph-disjoint: no query’s citation neighbourhood appears in training, not only the query itself.

Both constraints cost training data. That is the trade, and it is the reason the untrained base model in the table below scores as well as it does: the test set is not doing the training set any favours.

What each metric counts

Five numbers are reported on every rank-based run. They are quoted here for the production retriever so that each definition has a value attached to it.

The three cutoffs are not arbitrary. 10 is what a person reads. 100 is the depth the public API returns, so recall@100 is the number a customer experiences. 1000 is the window the reranker gets to reorder, which makes recall@1000 a ceiling on what any amount of reranking can recover.

On this cut most queries carry a single gold reference, which is why the citation-graph experiment in the negative results below is described as producing no rescues rather than a small gain. With one right answer per query, recall@k and success@k measure the same event.

Five systems, one pipeline

Every model runs the same three steps: encode the queries, rank the whole 609,723-document corpus, score the ranking. Same queries, same corpus, same top-K, and the metric code is imported from one module rather than reimplemented per model. Each model gets the input format it was trained for, which matters: giving a model a prompt it was not trained on produces a low number that measures the harness, not the model.

BM25 is not a strawman. It is the lexical baseline the retrieval literature reports, run on the same corpus with the same queries, and it retrieves the examiner’s reference in its top 100 on 24.3% of queries. Doubling that is the size of the problem, not a rounding difference.

The row that decides where the gain came from is Qwen3-0.6B. It is the base model our production encoder was trained from, scored untrained on the same harness. The difference between 0.3702 and 0.4528 is 8.3 points of recall@100, and it is attributable to the training program rather than to picking a good base.

The cross-domain slices

Every rank-based run reports IN, MIXED and OUT, decided by whether the gold shares a CPC section with the query. No slice is chosen after the fact, which is the rule that stops a result being reported on whichever cut it happened to win.

OUT is the column worth watching. Art cited from a different CPC section shares no classification, usually no vocabulary, and often no obvious subject matter with the claim it was cited against, which is exactly the case a keyword search cannot reach: BM25 finds it on 13.8% of queries. It is also where an invalidating reference tends to hide, because the art an applicant already knows about is the art in their own field.

What examiner citations cannot see

One caveat applies to every number above. Relevance here is examiner citations, and an examiner who has found a reference that supports the rejection stops looking. Art that is genuinely relevant but was never cited scores as a miss, so 0.4528 is a floor rather than an estimate of how often the right document is retrieved.

That deflation applies to every system in the table equally, so it does not distort the comparison. It does distort the absolute number, and the honest way to size it is to measure something citation recall cannot: whether the art we return is better disclosure than the art the examiner cited.

A separate study did that on a 105-claim sample. Each result was scored for how many elements of the claim it actually discloses, by an LLM judge validated against 251 expert-reviewed labels at Cohen’s κ = 0.695. On 82% of claims a hit inside our top 20 covered more claim elements than the examiner’s own citation did. The best result averaged 67% element coverage against the examiner citation’s 30%.

Two things follow. The gap between 45.3% and what a user would call a good result is large and measured rather than assumed. And κ = 0.695 is substantial agreement, not perfect agreement: the judge is a measurement instrument with known error, and the study is reported with the error rather than as a verdict.

Six things that did not work

Four of these are ideas a reader would reasonably expect to help, and the reason they are here is that finding out costs a week each. The reranker row is the one to read first.

Dropping an off-the-shelf cross-encoder on top of a retriever is the standard second stage, and here it took nDCG@10 from 0.268 to 0.091. It was trained on web and QA relevance, where a good match restates the query in other words. Claim-to-art relevance is close to the opposite: the reference that invalidates a claim is usually about a different application of the same mechanism, and a model tuned to reward paraphrase ranks that below a document that merely sounds similar. The purpose-trained ColBERT reranker that replaced it gains 0.038.

The title-only row is the other one worth keeping. Replacing claim text with the title halves ranking quality, from 0.101 to 0.050 nDCG@10, and pseudo-relevance feedback recovered none of it. The claim is the signal, not a longer version of the title.

Rerunning it

Everything above reruns end to end on one GPU in an afternoon. These are the pieces.

The runfile adapter is the one that matters to anyone outside the company. Score your own system offline, hand over a JSONL of ranked application numbers per query, and it goes through the identical metric code. A third-party model enters the comparison without us needing access to its internals, and without anyone taking our word for the scoring.

Two open items, stated rather than left out. No native PatenTEB scorecard has been run: their anonymized document ids defeat the leakage gate above, so the head-to-head runs their strongest model on our harness instead. And no commercial competitor appears in any of these tables, because none publishes recall or ranking figures in a form that could be quoted.

Sources for every figure

Each number on this page and the run it was quoted from. Nothing here was re-derived while writing.

Corrections to accelerate@antecedint.com. A figure that changes is corrected in place with the date on this note updated, not quietly edited.