Accuracy checks catch what AI adds, not what it drops. A step-by-step method and table template to check an AI summary for missing information.
To check an AI summary for missing information, decide which facts must survive before you read the summary, pull those facts from the source on their own, mark each one present, changed or missing, and report the omission rate on its own line, never blended into accuracy.
Most AI accuracy checks start from the summary and ask whether the source backs it up, which catches what the model added. It can't catch the allergy, the exclusion clause or the deadline that never made it into the summary, because there is nothing in the summary to check. This guide covers the other direction, from source to summary.
Why do AI omissions need their own measure?
Because the measures most people quote don't count them. Vectara's hallucination leaderboard, one of the most cited, scores whether a model's summaries say things the source doesn't support. It measures unsupported additions. It does not measure what was dropped.
Nor are omissions a rare edge case. In a UC Davis Health study published in April 2026, physicians evaluated 356 AI scribe notes. Accidental omissions appeared in 18% of them and hallucinations in 11.5%, so in that study missing information was the more common failure. And 5.3% of the notes carried an error rated as a serious or imminent risk.
How strictly you review changes the answer too. A preprint published in August 2026, "One note in three", audited 565 notes from three commercial ambient scribes in UK primary care and the US. Depending on the review standard applied, between 28% and 97% of the notes carried a failure. Same notes, very different number. If you don't fix the standard before you check, you don't know what your number means.
Omissions are also starting to be asked for by name. NHS England's guidance on ambient scribing says adopting organisations should get users, such as clinicians, to validate the rate of errors and omissions, and should collect monitoring data independently of the manufacturer.
Does "no contradictions found" mean nothing is missing?
No. A claim check reads each sentence in the summary and looks for support in the source. If a fact isn't in the summary, there is no sentence to check, so nothing fails. A summary that says three true things and leaves out the one thing that mattered gets full marks.
That's also why shorter summaries can look more accurate. Fewer claims, fewer chances to be wrong. Faithful and complete are different properties, and you have to test them separately.
If you've set up claim-by-claim checking using our guide on how to check an AI-written document against its sources, you've covered one direction. This is the other one.
What should count as "must be present"?
A summary is supposed to leave things out. That's its job. So the real question is whether anything was dropped that the reader needed.
You answer that before you look at any summary, for each type of document, by writing a required-facts list. The categories that matter almost everywhere are numbers, dates, names, obligations, warnings, allergies, exceptions and conditions. Watch exceptions and conditions in particular: a summary without them still reads well, so nobody notices they've gone.
Here's how that might look for common document types. Your own lists should be written and signed off by the people who do the work: a clinician for clinical notes, a solicitor for legal summaries, the policy owner for policy wording.
| Document type | Facts that must survive the summary |
|---|---|
| Clinical note or discharge summary | Allergies, current medicines and doses, warning signs to watch for, follow-up actions and dates |
| Insurance policy or contract summary | Parties, obligations, exclusions, limits, notice periods, conditions for a claim or for termination |
| Legal case or court order summary | Deadlines, what was ordered, conditions attached, who must do what |
| Board or committee minutes | Decisions, owners, deadlines, conditions on an approval, dissent |
| Financial report summary | Figures and the periods they cover, covenants, exceptions, restatements |
Keep each list short enough to check properly. A list of everything in the source is just the source.
How to check an AI summary for missing information, step by step
- Write the must-be-present rule for this document type. One page: the categories, a few examples of each, who signed it off and when. Version it. If the rule changes, results from before and after aren't comparable.
- Extract the required facts from the source without looking at the summary. Whoever does this, a person or a separate model, shouldn't see the AI's output first, or they'll anchor on it. Record where each fact sits: page, clause, paragraph or timestamp.
- Check each fact against the summary. Mark it present, present but changed, or missing. "Changed" covers a different number, date or dose, and softened wording: "must" becoming "should", or a fixed deadline becoming "promptly".
- Grade each miss or change by consequence. Keep it simple. High: could cause harm, a loss, a missed legal duty or a wrong decision. Medium: someone would have to go back to the source to put it right. Low: context only. Grade what happens if the reader relies on the summary, not how hard the fact was to find.
- Report the omission rate separately from accuracy, and never average the two. Show how many required facts were checked, how many were missing, how many were changed, and every high-consequence miss by name. Put the claim-check result next to it, not merged with it.
- Put disagreements in front of a named reviewer. Borderline calls, especially paraphrases, go to a person who decides and records the decision with their name and the date. That record is what you show when someone asks how you know.
- Pick a sample and repeat it every month. There's no published standard sample size for this, so choose a number you can check properly each month and write down why you chose it. Cover every document type and source system, and include long, messy documents, not just clean ones. Run it again straight away when the model, the prompt or the source documents change.
A table template you can copy
Use one row per required fact. The rows below are illustrative.
| Required fact | Source location | In summary? (yes / changed / no) | Consequence if wrong or missing | Reviewer decision |
|---|---|---|---|---|
| Penicillin allergy | Referral letter, allergies section | No | High | Confirmed missing. Summary returned for correction. (Name, date) |
| Notice period for termination | Contract, termination clause | Changed: summary gives the period in months, source gives it in days | Medium | Not equivalent. Corrected. (Name, date) |
| Flood exclusion | Policy schedule, exclusions section | No | High | Confirmed missing. (Name, date) |
| Deadline to file a response | Court order, final paragraph | Changed: "promptly" instead of the date | High | Rejected. The deadline must be stated. (Name, date) |
At the bottom of each batch, add four totals: required facts checked, missing, changed, and high-consequence misses. Those are the numbers you report.
Why should omissions never be averaged into an accuracy score?
An average lets one strong measure hide a weak one. A summary where every claim checks out and the only thing missing is the allergy would look excellent on any blended score. It's still the summary that hurts someone.
Keep the two numbers side by side so the reader sees both, and state the review standard with them, because the preprint audit above shows how far a number moves when the standard changes. Our piece An accuracy score is not evidence goes further into why a single score doesn't stand up on its own.
Can software check for omissions?
It can do much of the comparison, as long as it works from your required-facts list rather than deciding for itself what mattered. Two cautions.
First, don't use the model that wrote the summary to check it. In one study, on maths problems, a judge model preferred its own wrong answer over a correct one 86% of the time. Use checkers from a different vendor.
Second, automated checkers are good, not perfect. Even in the usual direction of checking, the best of them average about 77 on LLM-AggreFact, a public benchmark for checking text against its source. Paraphrase is the hard case: the fact is there in different words, and a checker may not see it.
How ETT can help
ETT Group's Accuracy Evidence File checks up to 30 of your AI outputs claim by claim against their sources, using two checking models from different vendors and never the model that wrote the output. Omissions are reported separately and never averaged into accuracy. Disagreements go to your named reviewer, and you get a certificate per output and a register.
FAQ
What is an AI omission?
A fact in the source that the reader needed and the summary left out. It's different from a hallucination, which is something in the summary that the source doesn't support. A summary can have no hallucinations at all and still leave out the one fact that mattered.
Isn't a summary meant to leave things out?
Yes, which is why you decide in advance which facts it may not leave out. Write a required-facts list for each document type, signed off by someone who does the work, before you check any output. Anything outside that list can be dropped without counting as a failure.
How is an omission rate different from an accuracy score?
An accuracy score asks whether what the summary says is backed by the source. An omission rate asks whether what had to be there is there. They fail in different ways, so report them side by side and never average them.
How often should we measure omissions?
Monthly on a sample, and again whenever the model, the prompt or the source documents change. Keep the required-facts list, the completed table and the reviewer decisions together for each month. That set of records is your evidence.
Sources
- Vectara, Hallucination Leaderboard, GitHub (last updated 11 May 2026)
- UC Davis Health, JMIR Medical Informatics, 17 Apr 2026: errors in physician-evaluated AI scribe notes
- Fox et al., "One note in three", arXiv 2608.31017, preprint, Aug 2026
- NHS England, Guidance on the use of AI-enabled ambient scribing products in health and care settings (version 2)
- Chen et al., "Do LLM Evaluators Prefer Themselves for a Reason?", arXiv 2504.03846 v3, Dec 2025
- LLM-AggreFact leaderboard (no update date shown; checked 9 Oct 2026)