AI Can Surface Safety Intelligence From Hospital Incident Reports

Hospitals already collect a great deal of patient safety data. The problem is that much of it arrives as free text: short narratives about medication delays, communication breakdowns, equipment confusion, specimen labeling problems, patient identification concerns, handoff gaps, and near misses that did not become serious harm. The practical question for AI in patient identification and hospital safety is not whether a model can sound fluent about safety. It is whether it can read across thousands of messy reports and return patterns that a safety team would recognize, question, and use.

That distinction matters because incident-reporting systems are both indispensable and incomplete. A 2025 HHS Office of Inspector General finding reported that hospitals miss capturing about half of patient harm events, which means the formal reporting stream is only one part of safety surveillance, not a complete map of harm.[1] Even within that partial stream, the most useful signals are often buried in low-harm events and near misses: reports that are individually easy to defer but collectively describe how a system is drifting.

The Stanford Medicine Children's Health study published in BMJ Quality & Safety is useful because it does not treat large language models as a generic efficiency story. It asks a narrower and more operational question: can GPT-4o extract and classify patient safety problems from real hospital incident narratives at a level that might support system-level learning? The study analyzed 9,357 safety reports and used GPT-4o to extract 15,635 unique patient safety problems from those narratives.[2]

Why the Free Text Is the Bottleneck

Structured fields in reporting systems are helpful for counting what the system already knows how to count. Free text is where reporters explain what actually happened: the workaround, the missing order detail, the misunderstood label, the device that looked like another device, the identity check that occurred too late, the escalation that depended on one person remembering a local rule.

That narrative detail is exactly what makes the reports valuable, and exactly what makes them hard to analyze at scale. Earlier machine learning and natural language processing work in patient safety showed promise, but safety narratives are difficult because they are contextual, multifactorial, and often describe several interacting problems rather than one clean event type.[3] A report might involve a communication lapse, a workflow design issue, a technology usability problem, and a patient identification risk in the same few sentences.

Traditional classification often copes with that complexity by forcing a report into a dominant category, or by relying on a miscellaneous bucket when the narrative does not fit. For a quality analyst, that is where a taxonomy starts to lose its value. The reports have not become less important; they have simply become less visible.

What GPT-4o Actually Did in the Stanford Study

The study's central task was not sentiment analysis, summarization, or chatbot-style advice. GPT-4o was used to identify discrete safety problems inside incident reports, then classify those problems into a patient safety taxonomy. That is a more demanding task than assigning one label to one report, because one report can contain multiple problems and those problems may sit at different levels of specificity.

Those numbers deserve to be read separately, not as one general claim that “AI worked.” The 94% expert agreement speaks to whether the model found the safety problems that human experts recognized as meaningful.[2] For a hospital safety program, that is the first threshold. A model that misses the problem in the narrative cannot be rescued by elegant dashboards later.

The classification results answer a different question. Parent-category accuracy of 91.5% means the model generally placed problems into the correct broad safety domain; child-category accuracy of 83.3% means it was less accurate when asked for finer-grained labels.[2] That difference is not surprising. Broad categories are usually easier to assign than narrow ones, especially when a report contains ambiguous context or overlapping causes.

For operational use, the parent-child gap should influence how much authority the output receives. Broad-category trends may be ready for routine human review sooner than highly specific subcategory counts. A patient safety committee could reasonably ask, “Are we seeing more medication process problems or identification-related risks this quarter?” before relying on the model to adjudicate every narrow subcategory without review.

The less-than-0.2% “other” rate is one of the more important findings because it speaks to a familiar failure mode in incident analysis.[2] Miscellaneous categories can become storage closets for the very narratives that need attention: unusual combinations, weak signals, and process failures that do not match the reporting system's assumptions. A very low “other” rate suggests that GPT-4o and the taxonomy together were able to preserve specificity across a large body of narrative reports.

Why Low-Harm Reports Are the Right Test

Serious harm events usually receive attention. They trigger case review, leadership awareness, documentation requirements, and sometimes external reporting. Low-harm events and near misses are different. They may be reviewed locally, but they rarely receive the same analytic bandwidth across months or thousands of reports.

The Stanford study framed this problem sharply, noting that near-miss and precursor events account for 99.9% of reported events in the studied context.[2] The point is not that every near miss carries the same importance. It is that the denominator of learning is much larger than the denominator of harm. If safety teams can see recurring patterns in the low-harm stream earlier, they may have a better chance of intervening before the same process weakness appears in a more severe case.

This is where LLM-supported analysis has its most plausible value. It does not need to replace root cause analysis, morbidity and mortality review, or human event investigation. It can make the unreviewed middle layer more legible: the cluster of reports that individually look manageable but collectively show a brittle handoff, a recurring ordering confusion, or an identification step that depends too heavily on memory.

Stakeholder Judgment Is Not a Soft Add-On

The study did not stop at model performance metrics. Stakeholder interviews found that the generated insights were viewed as clear, appropriate, and valuable.[2] That part of the work can look less technical than accuracy scoring, but it is central to whether a safety program would actually use the output.

A model can achieve attractive retrospective performance and still fail in practice if the people responsible for safety surveillance do not trust the categories, cannot understand the output, or cannot connect it to existing review routines. Implementation science language helps here because it separates different kinds of success. Appropriateness asks whether the tool fits the problem. Acceptability asks whether stakeholders find it reasonable and usable. Adoption asks whether the organization begins to use it in practice.

Those are not decorative measures. They are the difference between a study result and a functioning safety process. If the output is clear but not trusted, it will sit in a dashboard. If it is trusted but not routed to anyone with authority to act, it will become another queue. If it is adopted without a validation process, it can create a new safety risk by making weak classifications look official.

Human Review Is the Control Point

Human-in-the-loop validation is not a cautious phrase added to make AI sound responsible. In this setting, it is the control point that makes use plausible. Patient safety reports are not just text; they are institutional memory, legal risk, staff experience, and patient consequence compressed into narrative form. A false pattern can misdirect improvement work. A missed pattern can delay action. A confident but wrong subcategory can make a local hazard harder to challenge.

A reasonable pilot would keep humans in several places: sampling extracted problems for review, checking category assignments, examining the contents of any residual “other” group, and comparing model-generated themes with existing safety priorities. Reviewers should also look for what the model is not surfacing, especially in reports involving complex social context, health-equity concerns, or ambiguous accountability.

• Validate broad and narrow category performance separately, because the Stanford results were stronger for parent categories than child categories.
• Track the “other” category over time, since growth in miscellaneous classifications may signal taxonomy mismatch or model drift.
• Require review before using model output to prioritize formal improvement projects or leadership-level risk reporting.
• Reassess performance after model updates, local taxonomy changes, reporting-policy changes, or shifts in reporting culture.

The Stanford authors also acknowledged hallucination risk, model drift, and the need for human validation.[2] Those caveats should not be read as footnotes. They define the boundary between a defensible analytic aid and an unsafe automation claim.

What This Study Does Not Prove

The strongest limitation is generalizability. This was a single-site study at Stanford Medicine Children's Health using a single model, GPT-4o.[2] The results do not prove that the same performance would appear in an adult academic medical center, a community hospital, a multi-hospital system with inconsistent reporting practices, or an organization using a different taxonomy.

Reporting culture matters. One hospital's incident reports may be detailed, psychologically safe, and rich in context. Another's may be sparse, defensive, or skewed toward events that staff believe leadership wants to see. A model can only classify what the reporting system captures. If half of a hospital's relevant safety experience never enters the reporting stream, LLM analysis cannot recover it from absence.

Taxonomy fit also matters. A very low “other” rate in one institution may reflect a good match among the local reporting language, the study taxonomy, and the model instructions. Another institution may need to adapt categories before expecting similar performance. That adaptation should be measured, not assumed.

The model choice matters as well. The evidence here is evidence about GPT-4o in this task, at this site, under this study design. It is not evidence that every large language model can safely analyze incident reports, and it is not evidence that future versions of the same model will behave identically without revalidation.

Where the Evidence Points Next

The Stanford team is extending the approach beyond incident reports to complaints, grievances, health-equity data, and workforce safety data.[2] That extension is logical because many hospital safety signals live outside the formal event-reporting system. A family complaint may describe a communication failure before it appears as a reportable event. A workforce safety report may reveal a staffing or workflow condition that affects patient risk. Health-equity data may show that a process works differently across patient groups.

The important question is whether those additional narrative streams can be analyzed with the same discipline: clear extraction tasks, explicit taxonomies, expert review, separate accuracy measures for broad and narrow categories, stakeholder assessment, and monitoring after implementation. The appeal of LLMs in safety work is that they can make narrative data searchable at a scale humans rarely have time to sustain. The danger is treating that scale as judgment.

For patient safety officers and quality leaders, the study offers a serious evidence base for piloting LLM-supported incident analysis. The most compelling use case is not autonomous decision-making. It is surfacing system-level patterns from low-harm and near-miss reports so that human reviewers can spend less time finding the signal and more time deciding what it means.

That is qualified confidence, not a deployment verdict. GPT-4o performed well in one careful study: 94% agreement with expert reviewers, strong parent-category accuracy, useful child-category performance, and almost no reliance on “other.”[2] Before that becomes routine hospital infrastructure, each organization still has to test the model against its own reports, taxonomy, reporting culture, governance process, and tolerance for error.

References

1. Hospitals miss capturing half of patient harm events, HHS Office of Inspector General, 2025.
2. Using large language models to identify and classify patient safety events from incident reports, BMJ Quality & Safety, January 29, 2026.
3. Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review, JMIR Medical Informatics, 2020.