Vendor benchmarks tell you how a system performed on someone else's corpus. Here is how to run the test on yours, which measures to keep apart, and the comparison most evaluations quietly avoid.
Every retrieval vendor has a benchmark, and every benchmark was run on a corpus that is not yours. That is not necessarily dishonest, but it is not evidence about your estate either. Document types, question types and how often material changes all move the result, sometimes by a lot.
The test is not hard to run yourself. It is mostly a matter of being disciplined about four things.
Choose the questions before you see the answers
Questions chosen after someone has looked at what the system retrieves well are not a test. Take them from what your teams actually ask — support logs, internal query records, the questions that generate escalations — and freeze the list before any evaluation runs.
Include the questions you expect to be hard. An evaluation set with no hard questions in it produces a number that will not survive contact with production.
Keep three measures apart
This is where most evaluations go wrong, and it is the difference between a number you can act on and one you cannot.
- **Agreement.** Did the system's answer match a baseline answer? Useful, cheap to compute, and it says nothing about either answer being right. If both routes share the same error, that counts as agreement.
- **Correctness.** Was the answer true, judged against ground truth you supply? This is the one that matters, and it is the one that requires work, because someone has to establish what the right answer is.
- **Completeness.** Did the answer contain everything a complete answer needed? An answer can agree with the baseline, be factually correct, and still omit a material fact that both routes dropped. In a regulated context that omission is the risk.
A vendor reporting a single “accuracy” figure has collapsed these. Ask which one it is.
Pick an honest baseline
The easy comparison is against a model reading whole documents. Most retrieval approaches beat it, and the result flatters everyone.
The comparison that matters is against whatever you run today. If you already have a retrieval stack, that is the number to beat, and it is a harder number. An evaluation that skips it has answered a question you were not asking.
Where you have no existing system, say so, and note that the whole-document baseline is the weaker of the two comparisons.
Count the cost properly
Token counts are only comparable when you know what they cover. Retrieved context, total input, output and cached tokens are four different numbers, and a percentage saving on one of them says nothing about the others.
Then account for the parts that do not appear per query. What did it cost to process the corpus in the first place? What does it cost each time a document changes? How often do your documents change? A per-question saving that takes three years of query volume to repay its ingestion cost is not a saving on any timescale you are budgeting for.
Storage is the same story. Compare like for like: if the comparison excludes your source documents, your indexes, your replicas and your backups, it is measuring one component rather than the bill.
A worked example
Take fifty questions from your support logs across two document families — one stable, one that changes often. Establish the correct answer for each with someone who knows the material.
Run all fifty against your current system and against the candidate. Record agreement, correctness and completeness separately, plus the token count per category and the wall-clock latency.
Then add the two numbers that are easy to forget: what it cost to process the corpus, and what a month of normal document churn costs to reprocess.
The result will probably be mixed — better on the stable family, worse or break-even on the volatile one. That is a useful finding, and it points at where to deploy rather than whether to.
The assumptions to state
- The questions were fixed before evaluation and drawn from real usage
- Ground truth was established by someone who knows the material, not by the system under test
- The corpus sample includes your awkward formats, not only clean prose
- The token figures name which category they cover
- Ingestion and refresh costs are included
Key takeaways
- Vendor benchmarks are about someone else's corpus; run the test on yours
- Keep agreement, correctness and completeness apart, and ask vendors which one they measured
- Compare against your current system, not only against a model reading whole documents
- Name the token category, and include ingestion and refresh costs
- A mixed result is the most useful kind: it tells you where to deploy
Analect Proof runs this comparison on your own documents and reports the questions where the approach did no better. If you would rather evaluate something else with the same method, the method above works regardless of what you are testing.