# Benchmark methodology Research · Suite v1.0 How any prior-art retrieval system gets measured, whether it is ours, a new checkpoint, or anyone else's. We publish the standard so you can hold our numbers to it. The point of writing it down is comparability: a number produced this year has to mean the same thing as one produced next year, and a number we publish has to mean the same thing as one you reproduce. One artifact, two audiences. Internally the suite is the regression gate on every model change. Externally the scorecard it emits is what we publish. Same numbers, same provenance, no separate marketing math. The results themselves are on the [benchmarks page](https://antecedint.com/docs/benchmarks). Sign in and your free key is issued on the spot. ## Modes A mode is a question a user brings: a query type, a gold standard, and its slices. Modes are versioned datasets rather than opinions, and each one pins a fingerprint. Will this system find the art an examiner will cite against a pending claim? - **Query**: Title + independent claim 1 of the citing application - **Gold**: Examiner-cited applications from real office actions (X/Y citations, §102/§103) - **Notes**: holdout v1: 97,913 queries over a 609,723-doc corpus, fingerprint 6fd3de94f3b9 Are the results actually better disclosure than the examiner’s own citation? - **Query**: 105-claim sample, full result lists - **Gold**: LLM-judged per-element coverage, validated against 251 expert labels (κ = 0.695) - **Notes**: Separate machinery; the scorecard ingests its JSON summary Given a technical description with no claim language, does it find the same art? - **Query**: Title + abstract of the citing application - **Gold**: Same examiner gold as prosecution - **Notes**: Headline extra: register-shift Δ against prosecution Can a non-attorney describe an idea in plain English and find the art? - **Query**: LLM paraphrase of claim 1 with patentese removed, pinned prompt, ≥100-row human spot-check before freeze - **Gold**: Same examiner gold - **Notes**: Headline extra: paraphrase-robustness Δ Given a granted claim under invalidity challenge, does it retrieve the invalidating art? - **Query**: Challenged independent claim of a granted patent - **Gold**: References asserted in PTAB IPR petitions under §102/§103 grounds, multi-gold per query - **Notes**: PTAB bulk data; the mode people on the litigation side ask for by name Every rank-based mode reports the same three slices on every run, never a chosen subset: **domain** (IN, MIXED, or OUT of the citing CPC area), **statute** (§102 or §103), and **citation grade** (X or Y). Litigation reports ground and domain instead. ## Environments: the number changes with the haystack The same system scores differently against 609K documents and 10.5M. Every scorecard states its environment, comparing across environments is invalid, and the renderer refuses to do it. Which model is better. - **Corpus**: The pinned 609,723-doc holdout corpus - **How**: eval/bake_off.py, or a rankings file - **Depth**: R@10/100/1000, nDCG@10, MRR What a caller gets. - **Corpus**: The production index, 10.5M+ and growing weekly - **How**: The public API, exactly as anyone calls it - **Depth**: R@10/100, nDCG@10, MRR@100 (API depth caps at 100, so R@1000 is n/a) Publish both, labelled. ## The far-OUT split In-domain prior art is easy. Same CPC is the same art unit by construction, so co-citation and examiner-cited priors already cover it. What separates a real retriever from a lookup table is cross-domain recall: does the system find invalidating art living in a different CPC section from the challenged patent? The definition is frozen at a one-character CPC _section_ difference (A–H, Y), which is the strong cross-domain signal, not a mere subclass hop. Each query is tagged by the CPC overlap of its golds: | IN | Every reachable gold shares a CPC section with the challenged patent. | |---|---| | IN | Every reachable gold shares a CPC section with the challenged patent. | | OUT | No reachable gold shares any CPC section. This is the split. | | MIXED | Some golds same-section, some cross-section. | | UNKNOWN | CPC missing on the challenged patent or on all golds. Excluded from far-OUT. | Across the 239 litigation queries that lands at IN 205, MIXED 2, OUT 10, UNKNOWN 22, with CPC resolved for all 489 applications. How to read it `litigation_farout` is a stress test and never a headline: n = 10 by construction, because cross-domain invalidation is rare. Report it beside the full litigation number and its domain slice. A model that scores well on litigation overall but collapses on far-OUT is winning on in-domain co-citation, not on retrieval. Report all of IN, MIXED and OUT, never just the flattering one. ## Scoring any system The runner scores anything that implements one call: ``` search(query_text, topk) -> ordered list of application numbers ``` Three adapters ship with it: api The Antecedint live API, bearer key from env. Scores what a caller gets. runfile Bring your own rankings: a JSONL of {"qid": …, "results": ["", …]} produced by any system offline. Scored with identical metrics, which is how a third-party model enters the leaderboard without us needing its internals. mock Deterministic self-test adapter for CI, validating the metric plumbing against a known expected scorecard. Every dataset row carries `qid`, `text`, `gold_apps` and its slice fields. Where a query has several golds, the primary rank is the best-ranked one, which is what drives R@k, nDCG and MRR; litigation additionally reports gold-coverage@100, the fraction of all golds retrieved. ## What makes a number publishable Seven rules. A run that breaks any of them is not quotable, including by us. 1. 1Pinned dataA run records its dataset fingerprint. A scorecard whose fingerprint does not match the registry is void. 2. 2Identical conditionsSame queries, same depth, same corpus for every system in a comparison. The runner enforces depth; bake_off enforces the corpus. 3. 3Full runs publish, samples exploreA --sample run is labelled SAMPLE and never quoted externally. A publishable number is the full query set, one run, with no reruns-until-good. 4. 4All standard slices, every timeNo post-hoc slice selection. IN / MIXED / OUT, §102 / §103, and X / Y citation grade are reported on every rank-based run. 5. 5Immutable resultsScorecards are written to reports/scorecards/__ with the suite version and git SHA. Never overwritten, never edited. 6. 6Honest deltasA mode delta is computed only between runs sharing system, environment and dataset generation. 7. 7Baselines ride alongAny externally shown comparison includes BM25 and at least one public embedding baseline, from the same run conditions. ## The scorecard One markdown sheet per run: system and version, environment, the mode table of metrics by slice, sample sizes, dataset fingerprints, suite version, git SHA and date. Each metric carries a plain-English line beside it, because "R@100" is not self-explanatory to most readers: it means how often the examiner's citation appears in the first 100 results. The JSON twin is the machine-readable record that the benchmarks page and the full report both read. One source, so the site and the report cannot drift apart. ## Versions and registry The suite is semver: a metric or protocol change is a major bump, a new mode is a minor one. Dataset generations get their own fingerprints, and an old scorecard stays valid for its generation rather than being retroactively invalidated. | Dataset | Generation | Fingerprint | Frozen | Rows | |---|---|---|---|---| | prosecution queries | holdout v1 | 6fd3de94f3b9 | 2026-06 | 97,913 | | prosecution corpus | holdout v1 | 6fd3de94f3b9 | 2026-06 | 609,723 | | examiner-quality gold | v1 | with holdout v1 | 2026-06 | 251 expert labels | | litigation queries | litigation v1 | 1a1f6746edda | 2026-07-27 | 255 | | far-OUT split | litigation_farout | 345c0d3d13f0 | 2026-07-27 | 10 | | semantic_search queries | n/a | pending build | n/a | n/a | | natural_language queries | n/a | pending build | n/a | n/a | ## Pending modes Three modes are specified and registered but not yet built. They are listed here, unbuilt, for the same reason the negative results are published: a standard that only names the tests it passes is not a standard. 1. 1semantic_searchJoin citing application to title and abstract, freeze, fingerprint, register. Cheapest build, so it goes first. 2. 2natural_languageBedrock batch paraphrase of claim 1 with the prompt pinned and temperature off, roughly $10–20 for the full set, then a 100-row human audit before freeze. 3. 3litigationPTAB bulk: IPR petitions to challenged patent plus asserted §102/§103 references, resolved to application numbers. Multi-gold rows; expect a few thousand high-quality queries.