Benchmarks
How we know a search found the right art
A prior-art search returns a list. Whether the list holds the right documents is the question that matters, and the public record has already answered it millions of times: examiners cite art against claims, and the Patent Trial and Appeal Board cancels claims over specific references. Our benchmarks turn those decisions into tests.
This page explains what each benchmark is and walks one test case through from start to finish. The measured numbers, with their labels, are on the results page.
IP & prior art · measured
Six terms this page uses
- Claim
- The numbered sentence at the end of a patent that defines what it protects.
- Element
- One requirement inside a claim, such as “(d) a conjugated lipid”. A claim is invalid over the prior art only if every element is found there.
- Reference
- An earlier document, usually a patent, offered as prior art against a claim.
- Priority date
- The date a claim is judged from. Only documents filed before it can count as prior art.
- Patent family
- All the filings that share one invention: continuations, foreign versions and the like.
- Held out
- Kept out of everything the search was trained on, so the test measures finding art, not remembering it.
Benchmarks by area of law
Every area is tested against decisions someone outside Antecedint already made. IP is measured today; case law is being built the same way; tax is next.
Examiner citations · judged by patent examiners
Recall at 100Given a pending claim, does the search find the art the examiner cited against it?
A test case is
One claim from a US patent application filed in 2022 or 2023.
A correct answer is
The references the examiner cited against that claim.
The search runs over
A fixed collection of 609,723 patents of examiner-cited art, all filed before 2022.
Scored by
Recall at 100: the share of the examiner’s cited references that the search returns in its first 100 results. Also reported at 10 and 1,000.
- How many
- 97,913 claims.
- Kept out of training by
- Splitting by patent family: no query’s family, and no family linked to it by citations, appears in training. The test set is pinned by its fingerprint, 6fd3de94f3b9.
What it cannot tell you
Examiners cite only what they need to reject a claim, so a strong reference the examiner did not cite counts as a miss. The score is a floor, not a ceiling.
One test case, start to finish
The same steps build every PTAB Gold test case. Here is one, from inter partes review IPR2018-00680. The challenged patent is US 9,404,127, on lipid particles that deliver nucleic acids. The Board held its claim 1 anticipated by an earlier patent.
The claim the Board cancelled, split into its elements
A composition comprising: a plurality of nucleic acid-lipid particles, wherein each particle in the plurality of particles comprises: (a) a nucleic acid; (b) a cationic lipid; (c) a non-cationic lipid; and (d) a conjugated lipid that inhibits aggregation of particles, wherein at least about 95% of the particles in the plurality of particles have a non-lamellar morphology.
The element this example follows. The Board tied it to a passage in US 8,058,069, graded strong in step 4.
Why the numbers can be trusted
The answers come from the record
Examiners and the Board decided which art answers each claim, in public documents, before we arrived. We transcribe and check those decisions; we do not choose the right answers ourselves.
Nothing in the test is trained on
The PTAB set lists every challenged family (589), application (2,903) and reference (1,272) it contains, plus a fingerprint of every claim and element text. Every training build is checked against that list before it runs, and a row that touches it is dropped.
Scored as of the day the claim was filed
A search only sees documents filed before the claim’s priority date, and the challenged patent’s own family is removed. A reference from after the fact cannot count as a hit.
Every quotation is checked by a script
Each passage is matched word for word against the exhibit and must sit inside the column and line range, paragraph or page the Board or the petition cited.
Checked by a second reader, then a person
A second AI reader re-derived 40 rows from the source documents without seeing the original grades and agreed on 95% of passages, limitations and grades. A person then reviewed 16 rows, including the 6 the second reader disputed, and signed off; two grades they and the second reader agreed on were corrected.
Frozen, and versioned when it changes
Each release pins every file by its SHA-256 hash. The current release is 2.3; any change, even one grade, becomes a new version with a changelog entry, so a published number always points at fixed files.
What it cannot measure yet
Element-to-reference pairs scorable against today’s index
2,587 of 11,792
- Of 11,792 element-to-reference pairs, 2,587 can be scored against today’s index. Most of the rest are US patents granted before 2001, foreign patents and non-patent literature, which the index does not hold yet. They stay in the files with their reason and never count as misses.
- PTAB grades are an agent’s, checked as above. When we report PTAB Gold, the figure we lead with uses no grade at all: a hit is any reference the Board relied on.