Where Medical AI Delivers Proven Value and Where It Doesn't

The uncomfortable fact about AI in medicine in 2026 is not that adoption is slow. It is that adoption is now much easier to document than clinical value. The American Medical Association reports that 81% of physicians used AI in 2026, up from 38% in 2023.[1] The FDA authorization landscape is just as crowded: 1,451 AI-enabled medical devices had been authorized, with 295 clearances in 2025 alone; Innolitics’ 2025 review found that 71.5% of those 2025 clearances were in radiology.[2]

Those numbers describe momentum. They do not, by themselves, describe benefit. A 2025 JAMA analysis found that fewer than 2% of FDA-cleared AI devices had been tested in randomized clinical trials.[3] A separate review of more than 500 medical AI studies found that only 5% tested models on real patient data, while nearly half used exam-style questions rather than clinical encounters, live workflows, or patient-linked outcomes.[4] The gap is not a minor methodological footnote. It is the difference between a tool that can answer a medical question and a tool that improves care when a nurse, physician, patient, EHR, staffing pattern, and alert queue are all involved.

Clearance matters. A technical benchmark matters. A high AUROC can be a rational reason to keep evaluating a model. But none of these should be allowed to quietly become a claim that patients live longer, clinicians burn out less, cancers are caught earlier, or the health system saves money. Those are stronger claims, and they need stronger evidence.

The First Question Is Not Whether Medical AI Works

“Does medical AI work?” is too broad to be useful. It lets weak evidence borrow credibility from stronger evidence in neighboring domains. A model that detects a radiographic pattern under controlled conditions is making a different claim from an inpatient deterioration system that pages a response team earlier, and both are different from an autonomous patient-facing chatbot giving advice under uncertainty.

The better question is: what claim is this tool making, and what kind of study would be strong enough to support that claim? The April 2026 Nature Medicine editorial called for “proportional evidence,” meaning that evidence requirements should scale with the clinical, operational, and financial claims being made.[5] That standard is not hostile to early technical work. It simply keeps early evidence in its proper lane.

This distinction is easy to lose in procurement language. “FDA-cleared,” “accurate,” “deployed,” and “clinician-adopted” each answer a different question. None is a synonym for clinically valuable. The more consequential the claim, the less acceptable it is to stop at technical performance.

Why Technical Performance Is Necessary but Incomplete

Technical performance is often where a reasonable evaluation begins. A diagnostic or predictive model that performs poorly in retrospective testing does not deserve the benefit of workflow optimism. If the model is not calibrated, fails in relevant subgroups, or degrades outside the development site, a clinical trial may be premature rather than rigorous.

The problem starts when technical success is treated as if it has already survived clinical use. An AUROC can tell a committee how well a model separated classes in a dataset. It does not tell the committee whether an alert arrives while anyone can act, whether clinicians trust it after false positives, whether it worsens workload for night coverage, or whether it changes treatment quickly enough to matter.

That is why the JAMA finding on study design is so important. If only 5% of more than 500 medical AI studies tested models on real patient data, then much of the published evidence is still upstream of the conditions that determine clinical value.[4] Exam-style evaluations may be useful for comparing systems or identifying capabilities. They are much weaker evidence for deployment decisions in care settings where the cost of a wrong answer is not evenly distributed.

This also explains why the FDA device count should be read carefully. Regulatory authorization can establish that a device has met the applicable standard for its intended use. It does not mean that the device has shown lower mortality, fewer missed diagnoses, improved clinician well-being, or lower total cost of care. Radiology’s large share of clearances partly reflects the maturity of imaging datasets and device pathways, not a field-wide proof that AI improves outcomes wherever it is placed.[2]

Where the Evidence Has Moved Closer to the Bedside

The strongest evidence in medical AI is not evenly distributed. It clusters where the task is narrow enough to measure, where the workflow has a defined action pathway, and where investigators have looked beyond model accuracy. Deterioration prediction, sepsis implementation, ambient documentation, and AI-supported mammography screening are not proof that all medical AI works. They are examples of claims that have begun to meet more appropriate forms of evidence.

Deterioration Prediction: Lead Time Only Matters If Someone Can Act

Early warning systems are a useful test case because the performance metric is not the whole clinical proposition. A deterioration model is valuable only if it identifies risk early enough, routes that risk to the right person, and triggers an intervention that would not otherwise have occurred. Scheid and colleagues reported that AI deterioration prediction models achieved 8- to 24-hour lead time over standard alerts.[6] That finding is more clinically meaningful than a discrimination statistic alone because it measures time that a care team could potentially use.

Even here, lead time is not the endpoint. A warning that arrives 12 hours earlier but produces no change in assessment, fluids, antibiotics, escalation, monitoring, or goals-of-care discussion has not yet demonstrated patient value. The stronger deterioration studies are therefore important because they begin to connect prediction with response. They put the model inside the sequence of care rather than treating it as a detached classifier.

That is also where implementation becomes part of the evidence. A health system evaluating an inpatient prediction tool should ask who receives the alert, what they are expected to do, how often the alert fires, whether staffing exists to respond, and whether delayed or ignored alerts are tracked. These are not secondary operational details. They are the route through which a prediction becomes, or fails to become, a clinical intervention.

Sepsis: Mortality Evidence Raises the Bar

Sepsis is one of the few areas where the evidence discussion can move from earlier detection to mortality. Duke Health’s Sepsis Watch was associated with a 27% reduction in sepsis mortality.[7] A separate 9-hospital study found a 39.5% reduction in in-hospital mortality after implementation.[8] These are the kinds of figures that appropriately change the conversation because they attach AI-supported workflow to patient outcomes, not only to prediction quality.

The word “associated” still matters. Sepsis programs usually include more than a model: screening protocols, response teams, escalation rules, clinician education, and monitoring infrastructure may all change together. A mortality reduction after implementation should not be casually attributed to an algorithm alone unless the study design supports that causal claim. But it is still a stronger evidentiary position than a retrospective AUROC, because it asks whether patients fared differently after the AI-enabled program entered care.

For hospital committees, sepsis evidence also illustrates why local implementation is not a formality. If the model depends on timely vitals, lab feeds, medication data, or nursing documentation, then data latency is a clinical variable. If alerts go to an overwhelmed team, the tool may create an additional queue rather than an earlier rescue. A mortality claim has to be evaluated with the surrounding care model attached.

Ambient Documentation: Operational Value Is More Plausible Than Outcome Value

Ambient AI scribes have a different evidentiary profile. They do not need to prove that they directly reduce mortality to be useful. Their central claim is operational: less documentation time, less after-hours work, and possibly less burnout. The Stanford HAI 2026 AI Index reported that AI scribe tools reduced documentation time by up to 83%.[9] Mass General Brigham data associated scribe use with a drop in burnout prevalence from 52.6% to 30.7%, a 21.2 percentage-point absolute reduction.[10]

Those outcomes are not trivial. Documentation burden is a real clinical operations problem, and a tool that returns time to clinicians can be valuable even if its benefit is measured in hours rather than survival curves. The evidence standard should match that claim: time-motion data, note quality review, clinician-reported outcomes, patient privacy safeguards, error correction workflows, and downstream billing or coding effects all belong in the evaluation.

The more aggressive financial claims need more caution. The same Stanford HAI report cited one hospital system reporting 112% return on investment for AI scribes, but the underlying methodology was not detailed in the cited source.[9] That makes the figure interesting, not portable. A health system should not assume that an ROI estimate from one institution will survive different payer mix, clinician compensation, visit lengths, documentation rules, or implementation costs.

Mammography Screening: Scale Helps, but Context Travels Poorly

AI-supported mammography has one of the more substantial screening examples. Germany’s PRAIM study included 463,094 women and found that AI-supported mammography screening achieved a 17.6% higher cancer detection rate compared with standard double reading.[11] The scale of that study matters. Screening is exactly the kind of setting where small differences in detection, recall, workload, and false positives can affect large numbers of people.

The result should still be kept inside its setting. PRAIM was conducted within the German screening program, and the available evidence does not establish generalizability to U.S. screening populations, reimbursement structures, malpractice context, radiologist staffing, or baseline double-reading practices.[11] The finding supports serious consideration of AI-assisted screening; it does not license a universal claim that the same effect size will appear in every mammography program.

Where the Evidence Still Falls Short

The weakest areas are not necessarily the least interesting. Some are weak precisely because they are clinically ambitious. Diagnostic reasoning under uncertainty, open-ended clinical decision support, and autonomous patient-facing AI make broader claims than many imaging or documentation tools. They may need to handle ambiguity, incomplete histories, changing patient goals, and problems that do not have a single correct answer. Evidence from exam-style questions is particularly easy to overread in these domains.

A system that performs well on a medical licensing-style question set may be demonstrating medical knowledge retrieval or pattern recognition. That does not show how it behaves when the patient omits a medication, the presenting complaint is vague, the safest next step is watchful waiting, or the clinician must negotiate risk with a worried family. The Bedi et al. finding that nearly half of reviewed studies used exam-style questions is therefore not just a study-design quibble. It identifies a mismatch between evaluation environment and clinical use.[4]

Human-AI teaming also needs more careful evidence than many deployments receive. A tool can be accurate on its own and still worsen decisions if clinicians over-rely on it, under-use it, misunderstand its confidence, or cannot see when it is outside its intended use. In those settings, the relevant unit of evaluation is not the model alone. It is the model plus the clinician plus the interface plus the local workflow.

A Practical Evidence Standard for Health Systems

A health system does not need the same level of evidence for every AI tool. It does need to stop accepting the wrong level of evidence for the claim being made. A documentation assistant, a triage model, a sepsis alert, and an autonomous advice tool should not be evaluated as if they raise the same risk or promise the same value.

• If the claim is technical performance, ask for external validation, calibration, subgroup performance, and evidence that the test data resemble the intended clinical population.
• If the claim is real-world usability, ask for prospective data, clinician behavior measures, alert burden, error review, and evidence that the tool works inside the EHR and staffing model where it will be used.
• If the claim is clinical impact, ask for patient-linked outcomes and a design strong enough to separate the AI-enabled intervention from secular trends or parallel quality-improvement work.
• If the claim is reduced burnout, ask for clinician-reported outcomes, documentation-time measures, after-hours work, note quality, and durability beyond the initial adoption period.
• If the claim is financial return, ask for a transparent accounting of implementation costs, maintenance, training, clinician time, coding changes, and whether gains persist after rollout.

This framework leaves room for early-stage tools. A promising model with strong retrospective performance may justify further testing. A tool with prospective workflow data may justify limited deployment with monitoring. But claims of lower mortality, improved cancer detection, reduced burnout, or positive ROI should not rest on the same evidence used to justify a pilot.

The more mature posture is neither blanket enthusiasm nor blanket suspicion. It is claim-by-claim discipline. Medical AI has already produced areas of real value, and it has also produced a large inventory of tools whose clinical benefit remains unproven. Health systems need to keep those two facts visible at the same time.

References

1. AMA survey data, American Medical Association.
2. Innolitics 2025 Year in Review, Innolitics.
3. JAMA, 2025 analysis of FDA-cleared AI devices and randomized clinical trial evidence, JAMA, 2025.
4. Bedi et al., JAMA, 2025, JAMA, 2025.
5. Nature Medicine editorial on proportional evidence, Nature Medicine, April 2026.
6. Scheid et al., Nature Communications, 2025, Nature Communications, 2025.
7. Duke Health Sepsis Watch, Duke Health.
8. Nine-hospital sepsis implementation study.
9. Stanford HAI AI Index 2026, Stanford HAI, 2026.
10. Mass General Brigham ambient AI scribe data, Mass General Brigham.
11. PRAIM study, Germany.