Benchmark methodology

Environments: the number changes with the haystack

The same system scores differently against 609K documents and 10.5M. Every scorecard states its environment, comparing across environments is invalid, and the renderer refuses to do it.

Publish both, labelled.

The far-OUT split

In-domain prior art is easy. Same CPC is the same art unit by construction, so co-citation and examiner-cited priors already cover it. What separates a real retriever from a lookup table is cross-domain recall: does the system find invalidating art living in a different CPC section from the challenged patent?

The definition is frozen at a one-character CPC section difference (A–H, Y), which is the strong cross-domain signal, not a mere subclass hop. Each query is tagged by the CPC overlap of its golds:

Across the 239 litigation queries that lands at IN 205, MIXED 2, OUT 10, UNKNOWN 22, with CPC resolved for all 489 applications.

Scoring any system

The runner scores anything that implements one call:

search(query_text, topk) -> ordered list of application numbers

Three adapters ship with it:

Every dataset row carries qid, text, gold_apps and its slice fields. Where a query has several golds, the primary rank is the best-ranked one, which is what drives R@k, nDCG and MRR; litigation additionally reports gold-coverage@100, the fraction of all golds retrieved.

What makes a number publishable

Seven rules. A run that breaks any of them is not quotable, including by us.

1. 1

   Pinned data

   A run records its dataset fingerprint. A scorecard whose fingerprint does not match the registry is void.

2. 2

   Identical conditions

   Same queries, same depth, same corpus for every system in a comparison. The runner enforces depth; bake_off enforces the corpus.

3. 3

   Full runs publish, samples explore

   A --sample run is labelled SAMPLE and never quoted externally. A publishable number is the full query set, one run, with no reruns-until-good.

4. 4

   All standard slices, every time

   No post-hoc slice selection. IN / MIXED / OUT, §102 / §103, and X / Y citation grade are reported on every rank-based run.

5. 5

   Immutable results

   Scorecards are written to reports/scorecards/<UTC-timestamp>_<system>_<env> with the suite version and git SHA. Never overwritten, never edited.

6. 6

   Honest deltas

   A mode delta is computed only between runs sharing system, environment and dataset generation.

7. 7

   Baselines ride along

   Any externally shown comparison includes BM25 and at least one public embedding baseline, from the same run conditions.

The scorecard

One markdown sheet per run: system and version, environment, the mode table of metrics by slice, sample sizes, dataset fingerprints, suite version, git SHA and date. Each metric carries a plain-English line beside it, because "R@100" is not self-explanatory to the person deciding whether to buy: it means how often the examiner's citation appears in the first 100 results.

The JSON twin is the machine-readable record that the benchmarks page and the data room both read. One source, so the site and the diligence pack cannot drift apart.

Versions and registry

The suite is semver: a metric or protocol change is a major bump, a new mode is a minor one. Dataset generations get their own fingerprints, and an old scorecard stays valid for its generation rather than being retroactively invalidated.

Pending modes

Three modes are specified and registered but not yet built. They are listed here, unbuilt, for the same reason the negative results are published: a standard that only names the tests it passes is not a standard.

1. 1

   semantic_search

   Join citing application to title and abstract, freeze, fingerprint, register. Cheapest build, so it goes first.

2. 2

   natural_language

   Bedrock batch paraphrase of claim 1 with the prompt pinned and temperature off, roughly $10–20 for the full set, then a 100-row human audit before freeze.

3. 3

   litigation

   PTAB bulk: IPR petitions to challenged patent plus asserted §102/§103 references, resolved to application numbers. Multi-gold rows; expect a few thousand high-quality queries.