Benchmarks

Prosecution and litigation benchmarks

Both kinds of benchmark hand the search a claim and score the art it returns. They differ in who decided the right answer. In prosecution, an examiner cites what is needed to reject a pending claim, in an office action written under a time budget. In litigation, a petitioner pays for an invalidity search on an issued patent, and the Board decides which of that art defeats the claim. Benchmarks 1 to 3 below are prosecution benchmarks. Benchmark 4 is litigation.

The litigation answer is harder to find. The Board relies on art the examiner did not cite, which is how the claim was allowed in the first place, and that art often sits outside the patent’s own field. The production first stage retrieves 45.3% of examiner citations in its top 100, but puts a Board reference in its top 25 for 19.0% of PTAB Gold claims. Benchmark 4 shows what the deep search does with the same claims.

Benchmark 1 · Internal examiner-citation holdout

Did we retrieve what the examiner cited?

97,913 held-out claim queries (applications filed 2022–2023) against a 609,723-patent corpus of examiner-cited art. The split is family-aware on DOCDB familyId and citation-graph-disjoint, so no query’s family or citation neighborhood appears in training. Every model runs the identical encode, rank-the-whole-corpus, score pipeline. Test-set fingerprint 6fd3de94f3b9.

For a typical claim, the production retriever surfaces the examiner’s cited reference in its top-10 about 20% of the time, top-100 about 45%, top-1000 about 74%: roughly 1.9× BM25 and 2.1× PatentSBERTa at every cutoff. The lift over our own untrained base is +8.3 points recall@100 (0.3702 → 0.4528), so the gain comes from the training program, not base-model selection.

One caveat applies to every row: relevance is examiner citations only. Examiners often stop after citing one reference, so relevant art they didn’t cite scores as a miss. That deflates every model’s absolute numbers uniformly; benchmark 3 measures the gap directly.

Cross-domain slices

OUT means the cited art comes from a different CPC section than the claim: analogous art from another field, the hardest case and the one a searcher is least likely to find by hand. Recall@100, with NDCG@10 for the endpoints.

We lead every slice, and the reading runs both directions. Cross-domain art is hard for the whole field: only 27% of OUT golds reach anyone’s top-100, and recall, not ranking, is the bottleneck. Our IN-to-OUT degradation is the gentlest measured (a 2.0× gap, vs 2.98× for patembed and 4.13× for PatentSBERTa). And our own promotion gate for OUT (NDCG@10 ≥ 0.0855) sits above the entire field: the non-commercial state of the art scores 0.0684, we score 0.0701, our fused system 0.0720. Nobody clears the gate, so we publish the gate rather than moving it.

Head-to-head with the research state of the art

PatenTEB anonymizes patent IDs, so our leakage gate cannot be applied to their published sets. We ran the stronger test instead: patembed-large through our leakage-controlled harness, on all 97,913 claims, using patembed’s own purpose-built prompts (encoded bare it collapses to 0.0115 OUT NDCG@10, which would be unfair to it). NDCG@10 by slice:

Ours wins every slice under both of patembed’s prompt configs. Against its best config the OUT margin is thin, +0.0012 to +0.0017 NDCG@10: the result is consistency across all four slices, not size. The license matters more than the margin: patembed cannot ship, and among shippable models the OUT comparison is 2.7× PatentSBERTa.

Benchmark 2 · DAPFAM (public, external)

A benchmark built by the patembed authors

DAPFAM (1,247 query families, 45,336 target families, NDCG@100) was built by the patembed authors, so its conventions and per-slice prompts favor their model. That caveat cuts against us, and we ran it anyway. Before scoring ourselves, our harness reproduced the paper’s published patembed numbers: OUT near-exact, ALL and IN within 0.02, and the direction of the difference scores patembed conservatively, not us.

Benchmark 3 · Element coverage vs the examiner

Is the art we find better than the citation?

Exact-citation recall scores a hit only when we retrieve the same document the examiner cited. Examiners record the few references they need, so a retriever that surfaces stronger uncited art scores zero. This study scores, per claim element, how much of the claim each reference discloses, then compares our best surfaced reference against the examiner’s own citation.

105 claims (35 IN / 50 MIXED / 20 OUT), 2,408 pairwise judgments by an LLM judge validated against 251 expert-reviewed labels (weighted Cohen’s κ = 0.695 on the exact distinction the metric consumes; acceptance gate ≥ 0.60). Every query carries two decoys, one random patent and one from the same CPC class, to set a per-query noise floor. Total judging cost: $19.07.

The effect holds in all three domains and is strongest in OUT, exactly where the examiner’s own cite is weakest (18% coverage) and cross-domain art is hardest to find. Small per-domain n (OUT = 20) means wide intervals; the cross-domain consistency is the signal.

Benchmark 4 · PTAB Gold v2 (litigation)

Did we find the art the Board relied on?

Each query is a claim the Board held unpatentable. A hit is any reference the Board relied on in a decided ground, so the headline score uses no model judgement. Every search is cut at the claim’s priority date with the challenged patent’s family removed, and every trial in the benchmark is held out of all training. 525 claims can be scored against the 9,848,869 documents in today’s index; the others rely on art the index does not hold yet.

The deep search rewrites the claim into the vocabulary of the fields it could sit in, searches six query styles, follows the patent’s own citations, its family’s citations and the citations of its best results, and fuses the seven candidate lists with equal weights. Jev, a fast model that rates how fully a document teaches a claim, then reads the top 100 together with the specification passages that share the most rare terms with the claim. Nothing in it was fitted to these claims: the fusion is unweighted, and Jev’s weights were fixed on examiner data before this benchmark was read. Model calls cost about $0.015 per claim.

Art at least as strong as the Board’s

The Board’s exact reference is often not the only art that defeats a claim. A second score asks whether the top 10 holds a document at least as strong. Gemini 2.5 Flash compares each candidate with the Board’s reference on the limitations the Board relied on, three votes per claim. That is a model’s judgement, so it runs with controls. Shown the Board’s own reference under another number, the judge picks it out in 317 of 322 claims. A top 10 borrowed from another claim in the same field fools it 11.5% of the time.

Stage 2 · Reranking

Purpose-trained against off-the-shelf rerankers

A late-interaction reranker we trained for cross-domain art (ColBERT-OUT v1) re-orders the first stage’s top-100. The usual alternative, adding a general-purpose cross-encoder, is the first thing anyone would try. We tried it. Gold-in-window NDCG@10:

The web-trained reranker rewards surface similarity and buries non-obvious cross-domain art; it was verified not to be a sort bug. The distance between those two rows, +0.042 against −0.209 on OUT, is the case for purpose-built patent models in one table. Deployment depth is top-100: reranking to 1000 buys a quarter of the extra lift at ten times the cost.

Ablations

Negative results, published

All on the same pinned harness as every success above, published so nobody has to repeat them. Two of these are the moves most teams would try first; both make retrieval worse.

Reproducibility

Rerun everything: one GPU, an afternoon

Identical queries, corpus, top-K, and metric code for every entry on this page. Encode with a candidate model, rank, score. The standard each of these numbers has to meet is written down in the benchmark methodology.

Benchmark runs June 2026; page compiled from the raw run reports without re-scoring. Technology developed 2024–2026. patembed and PatenTEB are the work of their respective authors; DAPFAM is datalyes/DAPFAM_patent.