Benchmarks

Benchmark 1 · Internal examiner-citation holdout

Did we retrieve what the examiner cited?

97,913 held-out claim queries (applications filed 2022–2023) against a 609,723-patent corpus of examiner-cited art. The split is family-aware on DOCDB familyId and citation-graph-disjoint, so no query’s family or citation neighborhood appears in training. Every model runs the identical encode, rank-the-whole-corpus, score pipeline. Test-set fingerprint 6fd3de94f3b9.

For a typical claim, the production retriever surfaces the examiner’s cited reference in its top-10 about 20% of the time, top-100 about 45%, top-1000 about 74%: roughly 1.9× BM25 and 2.1× PatentSBERTa at every cutoff. The lift over our own untrained base is +8.3 points recall@100 (0.3702 → 0.4528), so the gain comes from the training program, not base-model selection.

One caveat applies to every row: relevance is examiner citations only. Examiners often stop after citing one reference, so relevant art they didn’t cite scores as a miss. That deflates every model’s absolute numbers uniformly; benchmark 3 measures the gap directly.

Cross-domain slices

OUT means the cited art comes from a different CPC section than the claim: analogous art from another field, the hardest and commercially most valuable case. Recall@100, with NDCG@10 for the endpoints.

We lead every slice, and the honest read runs both directions. Cross-domain art is hard for the whole field: only 27% of OUT golds reach anyone’s top-100, and recall, not ranking, is the bottleneck. Our IN-to-OUT degradation is the gentlest measured (a 2.0× gap, vs 2.98× for patembed and 4.13× for PatentSBERTa). And our own promotion gate for OUT (NDCG@10 ≥ 0.0855) sits above the entire field: the non-commercial state of the art scores 0.0684, we score 0.0701, our fused system 0.0720. Nobody clears the gate, so we publish the gate rather than moving it.

Head-to-head with the research state of the art

PatenTEB anonymizes patent IDs, so our leakage gate cannot be applied to their published sets. We ran the stronger test instead: patembed-large through our leakage-controlled harness, on all 97,913 claims, using patembed’s own purpose-built prompts (encoded bare it collapses to 0.0115 OUT NDCG@10, which would be unfair to it). NDCG@10 by slice:

Ours wins every slice under both of patembed’s prompt configs. The candid read: against its best config the OUT margin is thin, +0.0012 to +0.0017 NDCG@10. The win is its consistency across all four slices, not its size. What turns a narrow research win into a wide commercial one is the license: patembed cannot ship, and among shippable models the OUT comparison is not close (2.7× PatentSBERTa).

Benchmark 2 · DAPFAM (public, external)

The other side’s benchmark

DAPFAM (1,247 query families, 45,336 target families, NDCG@100) was built by the patembed authors, so its conventions and per-slice prompts favor their model. That caveat cuts against us; we ran it anyway. Before scoring ourselves, our harness reproduced the paper’s published patembed numbers: OUT near-exact, ALL and IN within 0.02, and the direction of the difference scores patembed conservatively, not us.

Benchmark 3 · Element coverage vs the examiner

Is the art we find better than the citation?

Exact-citation recall scores a hit only when we retrieve the same document the examiner cited. Examiners record the few references they need, so a retriever that surfaces stronger uncited art scores zero. This study scores, per claim element, how much of the claim each reference discloses, then compares our best surfaced reference against the examiner’s own citation.

105 claims (35 IN / 50 MIXED / 20 OUT), 2,408 pairwise judgments by an LLM judge validated against 251 expert-reviewed labels (weighted Cohen’s κ = 0.695 on the exact distinction the metric consumes; acceptance gate ≥ 0.60). Every query carries two decoys, one random patent and one from the same CPC class, to set a per-query noise floor. Total judging cost: $19.07.

The effect holds in all three domains and is strongest in OUT, exactly where the examiner’s own cite is weakest (18% coverage) and cross-domain art is hardest to find. Small per-domain n (OUT = 20) means wide intervals; the cross-domain consistency is the signal.

Stage 2 · Reranking

Purpose-trained beats off-the-shelf, by a lot

A late-interaction reranker we trained for cross-domain art (ColBERT-OUT v1) re-orders the first stage’s top-100. The popular alternative, bolting on an open-source cross-encoder, is the first thing anyone would try. We tried it. Gold-in-window NDCG@10:

The web-trained reranker rewards surface similarity and buries non-obvious cross-domain art; it was verified not to be a sort bug. The distance between those two rows, +0.042 against −0.209 on OUT, is the case for purpose-built patent models in one table. Deployment depth is top-100: reranking to 1000 buys a quarter of the extra lift at ten times the cost.

Ablations

Negative results, published

All on the same pinned harness as every success above. Two of these are the intuitive moves a competitor would try first; both make retrieval worse.

Reproducibility

Rerun everything: one GPU, an afternoon

Identical queries, corpus, top-K, and metric code for every entry on this page. Encode with a candidate model, rank, score. The standard each of these numbers has to meet is written down in the benchmark methodology.

Benchmark runs June 2026; page compiled from the raw run reports without re-scoring. Technology developed 2024–2026. patembed and PatenTEB are the work of their respective authors; DAPFAM is datalyes/DAPFAM_patent.