# Benchmarks Benchmarks # How we know a search found the right art A prior-art search returns a list. Whether the list holds the right documents is the question that matters, and the public record has already answered it millions of times: examiners cite art against claims, and the Patent Trial and Appeal Board cancels claims over specific references. Our benchmarks turn those decisions into tests. This page explains what each benchmark is and walks one test case through from start to finish. The measured numbers, with their labels, are on the [results page](https://antecedint.com/docs/benchmarks). IP & prior art · measured ## Six terms this page uses - **Claim**: The numbered sentence at the end of a patent that defines what it protects. - **Element**: One requirement inside a claim, such as “(d) a conjugated lipid”. A claim is invalid over the prior art only if every element is found there. - **Reference**: An earlier document, usually a patent, offered as prior art against a claim. - **Priority date**: The date a claim is judged from. Only documents filed before it can count as prior art. - **Patent family**: All the filings that share one invention: continuations, foreign versions and the like. - **Held out**: Kept out of everything the search was trained on, so the test measures finding art, not remembering it. ## Benchmarks by area of law Every area is tested against decisions someone outside Antecedint already made. IP is measured today; case law is being built the same way; tax is next. Examiner citations · judged by patent examiners ### Given a pending claim, does the search find the art the examiner cited against it? 1. A test case isOne claim from a US patent application filed in 2022 or 2023. 2. A correct answer isThe references the examiner cited against that claim. 3. The search runs overA fixed collection of 609,723 patents of examiner-cited art, all filed before 2022. 4. Scored byRecall at 100: the share of the examiner’s cited references that the search returns in its first 100 results. Also reported at 10 and 1,000. - **How many**: 97,913 claims. - **Kept out of training by**: Splitting by patent family: no query’s family, and no family linked to it by citations, appears in training. The test set is pinned by its fingerprint, 6fd3de94f3b9. What it cannot tell you Examiners cite only what they need to reject a claim, so a strong reference the examiner did not cite counts as a miss. The score is a floor, not a ceiling. PTAB Gold · judged by the Patent Trial and Appeal Board ### Given a claim the Board cancelled, does the search find the art the Board relied on? 1. A test case isA claim the Board held unpatentable in an inter partes review final written decision, or one element of that claim. 2. A correct answer isEvery reference the Board relied on for that claim or element, with the passages it cited copied word for word from the reference. 3. The search runs overThe full Antecedint index of 9,848,869 US and EP documents, limited to documents filed before the claim’s priority date, with the challenged patent’s own family removed. 4. Scored bySuccess at 25: the share of claims for which the search returns at least one reference the Board relied on in its first 25 results. Also reported at 10 and 100. - **How many**: 764 trials, 1,530 claims and 6,937 elements. 525 claims and 2,030 elements can be scored against today’s index. - **Kept out of training by**: Listing every challenged family, application and reference, and a fingerprint of every claim and element text. Every training build is checked against that list and drops any row that touches it. What it cannot tell you References the index does not hold yet, mostly US patents granted before 2001, foreign patents and articles, cannot be scored. They are reported separately, never counted as misses. DAPFAM · judged by outside researchers ### On a public benchmark we did not build, do the results hold up? 1. A test case isThe title and abstract of a patent family. 2. A correct answer isThe target families the dataset itself marks as relevant to that query. We do not choose them. 3. The search runs over45,336 target patent families. 4. Scored byNDCG at 100, a ranking score from 0 to 1 that rises as relevant families appear nearer the top of the first 100 results. - **How many**: 1,247 query families, split into same-field and cross-field cases. - **Kept out of training by**: Nothing we control: others built it, so some of its patents may appear in our training data. What it checks instead is our harness. Before scoring our own model, the harness must reproduce the scores already published on this dataset for another team’s model; if it cannot, no number is reported. What it cannot tell you It measures how alike two whole documents are, from their titles and abstracts. It does not test a claim against prior art, which is what the other two benchmarks do. Case law · being built ### Given a legal question, does the search find the passage a court relied on? Research questions · judged by courts When a later opinion quotes an earlier one, the court has already said which passage answers the point. Those pairs become test cases, and a search dated to the later opinion may only see what was decided before it. Treatment · judged by the public record of overrulings Whether a case still stands is checked against public records of which decisions overruled or limited which, so an edge saying one case follows or overrules another is tested against what courts recorded. What to expect The same rules as IP: answers taken from the record rather than chosen by us, test cases kept out of training, every search dated to the decision it is tested on, and a frozen, fingerprinted release before any number is published. Tax · next ### Tax follows case law. Its benchmark will be built the same way, from decisions already on the public record, and published with the same labels and fingerprint before any tax figure appears on this site. ## One test case, start to finish The same steps build every PTAB Gold test case. Here is one, from inter partes review IPR2018-00680. The challenged patent is US 9,404,127, on lipid particles that deliver nucleic acids. The Board held its claim 1 anticipated by an earlier patent. ### The claim the Board cancelled, split into its elements A composition comprising: a plurality of nucleic acid-lipid particles, wherein each particle in the plurality of particles comprises: (a) a nucleic acid; (b) a cationic lipid; (c) a non-cationic lipid; and (d) a conjugated lipid that inhibits aggregation of particles, wherein at least about 95% of the particles in the plurality of particles have a non-lamellar morphology. The element this example follows. The Board tied it to a passage in US 8,058,069, graded strong in step 4. ### What the Board relied on The reference is US 8,058,069 (Yaworski), _Lipid Formulations for Nucleic Acid Delivery_, filed April 15, 2009. The final written decision ties it to element (d) in these words: Final written decision, IPR2018-00680 ### The passage, copied from the exhibit and checked “In one aspect, the present invention provides lipid particles comprising: (a) one or more active agents or therapeutic agents; (b) one or more cationic lipids comprising from about 50 mol % to about 85 mol % of the total lipid present in the particle; (c) one or more non-cationic lipids comprising from about 13 mol % to about 49.5 mol % of the total lipid present in the particle; and (d) one or more conjugated lipids that inhibit aggregation of particles comprising from about 0.5 mol % to about 2 mol % of the total lipid present in the particle.” - Found word for wordA script finds the passage in the exhibit text. No paraphrase is accepted. - Inside the cited linesThe Board cites column 3, lines 16 to 19; the passage sits at lines 10 to 20, around that cite. - Cited for this elementThe Board relies on these lines for limitation (d), not for some other part of the claim. ### Graded part by part Each term of the element is matched against the passage. Both parts are disclosed in the reference’s own words, so the row is graded **strong**. - The claim requiresa conjugated lipid The reference says“one or more conjugated lipids” - The claim requiresthat inhibits aggregation of particles The reference says“that inhibit aggregation of particles” Not every row grades well. The claim’s last element, that 95% of the particles are non-lamellar, never appears in the reference’s text; the Board found it inherent in how the particles are made. That row is graded weak and stays out of the graded subsets, though the Board still relied on the reference. ### Scored as of the claim’s priority date The search runs over the full index but may only return documents filed before December 27, 2012, the challenged patent’s priority date, and never a member of its own family. The claim counts as a success if US 8,058,069, or any other reference the Board relied on, is among the first 25 results. A document filed **Apr 15, 2009** can be returned: it predates the priority date. Drag the date. Searchable: documents filed before the cutoff. Excluded: everything after it, and the challenged patent’s family whenever it was filed. ## Why the numbers can be trusted ### The answers come from the record Examiners and the Board decided which art answers each claim, in public documents, before we arrived. We transcribe and check those decisions; we do not choose the right answers ourselves. ### Nothing in the test is trained on The PTAB set lists every challenged family (589), application (2,903) and reference (1,272) it contains, plus a fingerprint of every claim and element text. Every training build is checked against that list before it runs, and a row that touches it is dropped. ### Scored as of the day the claim was filed A search only sees documents filed before the claim’s priority date, and the challenged patent’s own family is removed. A reference from after the fact cannot count as a hit. ### Every quotation is checked by a script Each passage is matched word for word against the exhibit and must sit inside the column and line range, paragraph or page the Board or the petition cited. ### Checked by a second reader, then a person A second AI reader re-derived 40 rows from the source documents without seeing the original grades and agreed on 95% of passages, limitations and grades. A person then reviewed 16 rows, including the 6 the second reader disputed, and signed off; two grades they and the second reader agreed on were corrected. ### Frozen, and versioned when it changes Each release pins every file by its SHA-256 hash. The current release is 2.3; any change, even one grade, becomes a new version with a changelog entry, so a published number always points at fixed files. ## What it cannot measure yet Element-to-reference pairs scorable against today’s index 2,587 of 11,792 - Of 11,792 element-to-reference pairs, 2,587 can be scored against today’s index. Most of the rest are US patents granted before 2001, foreign patents and non-patent literature, which the index does not hold yet. They stay in the files with their reason and never count as misses. - PTAB grades are an agent’s, checked as above. When we report PTAB Gold, the figure we lead with uses no grade at all: a hit is any reference the Board relied on. ## Where to go next