Research · Doctrine

The assay doctrine

A benchmark asks who won. An assay asks what the instrument can detect, and how you know it is not lying.

A leaderboard can publish a ranking without stating whether the gap between first and second is larger than its own noise. An assay cannot: it reports a detection limit, runs controls beside every sample, and records the samples that failed. Deciding whether to put an autonomous system on client matter needs that, not a winner. Results are on the benchmarks page; how a system gets scored is in the methodology.

Benchmark or assay

One measurement, two sets of habits, two kinds of claim. The right column is the one that survives someone else rerunning it.

	Leaderboard	Assay
Headline	Rank order	The detection limit, then the reading
A close result	First place	Below the detection limit, reported as such
Ground truth	An answer key	Controls of known value, run alongside
A failed sample	Dropped	Recorded, with the reason
Losing	Hidden	Published, because the instrument is the product

Rule one: state the detection limit before the reading

Every scored set has a smallest difference it can resolve, governed by the number of independent patent families rather than the number of rows. A claim smaller than that limit is void rather than weak.

What the current gold set can resolve

977 golds · 65 families · MDE 0.2431

inside the noise

measurable

MDE 24.3 pp

40 pp

Reranker deltas
The tuple channel, measured at +2.4 points
Any single-digit ablation

Lexical against semantic retrieval
Passage against document level
Facet against whole-claim encoding

This rule costs the most, because the comparisons a vendor most wants to win sit on the left of that line. Saying so is the price of the ones on the right meaning anything. The launch target is MDE 0.10, roughly 688 decided proceedings.

Rule two: run controls on every plate

No control, no run. Four of them, none optional.

Positive control

977 / 65

Known conflict, adjudicated

Golds over families: a limitation the petitioner asserted, a passage in a reference, and a Board proceeding that recorded it.

208 golds are blocked by the query-length clause.

Negative control

132 / 8

Adjudicated non-conflict

The control most benchmarks skip, and ours is free: the Board read this reference against this claim and said no.

Ground-level labels only. Supports existence claims, never comparison.

Blank

45.8% · 0.5%

The reading on nothing

One matcher at one threshold, scored against unrelated documents and then against claim facets. A result near the floor is the floor.

CPC-matched distractors do not raise it (47.6% against 48.2% random).

Calibrator

κ = 0.695

Judge against human

Agreement on 251 expert labels. Without it, a rubric-scored arm is a model grading a model.

Bands are preregistered before labelling starts.

Rule three: reagents carry a grade

You do not run a clinical assay with reagents of unknown origin, and you do not certify a legal system against labels of unknown origin.

Tier A

Adjudicated

An act on a docket: an examiner citation, a §102 or §103 rejection, a PTAB ground outcome, an EPO X or Y grade.

Certifies

Tier B

Attorney rubric, model applied

A human-authored reference and rubric, applied by a judge panel whose agreement with attorneys is measured.

Certifies if the agreement statistic ships with it

Tier C

Model authored

Synthesised scenarios and model-invented labels. Useful for inputs, perturbations and distractors.

Never a score

A judge panel whose agreement with attorneys is unmeasured is Tier C wearing a rubric.

The other five rules

Each exists because breaking it produced a wrong number here first. The mechanism beside it is what stops it recurring, because a rule with no enforcement point is a slogan.

Record the samples that failed

Discarding what would not resolve makes the denominator a function of how well the tooling worked, which is the one thing the measurement exists to check.

Mechanism The validator returns failing clause names and never drops a record. 538 rows carry an unresolvable-passage flag and stay in the funnel.

One role per sample per run

A record the Board rejected on the merits is both a legitimate gold and the raw material for the near-miss class. It cannot be both inside one computation.

Mechanism The pool function returns disjoint sets and throws on intersection.

Every reading names its lot

The same system scores differently against 609,723 documents and 10.5M, so a number without its environment is not a number.

Mechanism Fingerprint, suite version, git SHA and timestamp, written once. A fingerprint that misses the registry voids the scorecard.

Calibrate the instrument against itself

A validator carrying its own copy of the extraction logic validates nothing. Ours reported 48% agreement on reference counts; calling the shipping function gave 95%. The defect was in the measurement.

Mechanism Every checker calls production code, or says what it therefore does not cover.

Pilot runs do not publish

Publishable means the full query set, one run, no reruns until the number improves, and every standard slice including the unflattering one.

Mechanism A sampled run is stamped SAMPLE and never quoted outside.

Could someone outside this building rebuild the inputs, rerun the measurement, and land on the same number?

If no, it is not ready. If the honest answer is "only if they trust us", it is a leaderboard. The cost of answering yes is fewer headlines and slower publication: our own cross-domain retrieval number is 0 of 10 and gets published at that value.

Where this ships

Two of these are live. The rest are listed unbuilt, for the same reason the negative results are published: a standard that names only what it has already shipped is not a standard.

Scorecards

Available

Every run we publish, with its slices, sample sizes and fingerprints.

Benchmarks →

Measurement contract

Available

Modes, environments, metrics and the rules that make a number quotable.

Methodology →

Gold datasets on Hugging Face

Not yet published

Labels and character-level spans, one repository per task family and one configuration per patent office. The reference text is not redistributed: a resolver rebuilds it from USPTO and EPO bulk data, so a consumer who rebuilds and gets a different fingerprint knows immediately.

Passage grounding first, once the set reaches MDE 0.10

The scoring harness

Not yet published

Open source, including the bring-your-own-rankings adapter that scores an offline run from any system with identical metrics.

Published alongside the first dataset

Methods paper

Not yet published

The gold record specification, the nine clauses, how the adjudicated negative class is built, and the power analysis behind the detection limit.

Preprint on release of the first dataset

Submission endpoint

Not yet published

A held-out test split with rate-limited submissions and a required system card, so the standard does not decay into a training target.

After the harness is public

If you want to be told when the first dataset lands rather than checking, say so and we will write to you once.

Vocabulary

Assay term	Here	Enforcement
Limit of detection	Minimum detectable effect	Printed beside the metric on every scorecard
Positive control	The positive gold pool	Nine clauses, all of which must pass
Negative control	The merits-rejected pool	Held disjoint from the positives at computation time
Blank	Matched nulls	Re-derived per pool, never carried over from another
Reagent grade	Label provenance tier	Recorded on the record, not asserted in prose
Lot number	Dataset fingerprint	Checked against the registry before a run publishes