Why Most FDA-Cleared Radiology AI Devices Lack Prospective Clinical Validation

A radiology AI product can arrive in a procurement meeting with FDA clearance, a polished workflow diagram, and a claim that it fits neatly into medical imaging operations. The first question should still be plain: has this tool been tested prospectively in clinical conditions, with the kind of people and workflow that will actually use it?

For most cleared radiology AI devices, the answer appears to be no. A systematic review of 950 FDA-authorized AI/ML medical devices through June 2024 found that radiology accounted for 723 devices, or 76% of the total; among 717 radiology devices with available documentation, only 33 underwent prospective testing, only 56 included a human operator in testing, and only 208 incorporated any clinical testing at all. Just 15 devices combined prospective and clinical testing, and only 6 included prospective, clinical, and human-in-the-loop testing together. Those are not minor documentation gaps. They describe the evidence base that many hospitals inherit when a cleared algorithm moves from a vendor deck into a live reading room. [1]

The same review found that 97% of all FDA-authorized AI/ML devices in its sample went through the 510(k) pathway, where the central regulatory question is substantial equivalence to a predicate device rather than independent proof that the new tool improves clinical outcomes in a deployed setting. That pathway has a real function; it is not a loophole by definition. But it also explains why FDA clearance and prospective clinical validation can coexist so unevenly. [1]

For readers trying to identify a specific product on the FDA list before a committee meeting, a practical starting point is how to search FDA-authorized AI devices efficiently. The search answers whether a product is authorized. It does not answer whether the tool has been tested in conditions close enough to the hospital’s own patient mix, scanners, radiologists, reporting habits, and escalation pathways.

Clearance Answers a Different Question

FDA clearance is often treated as if it settles the clinical discussion because it is the easiest credential in the room to verify. A committee can check a database, confirm the product category, and move on to price, integration, and service terms. That is comfortable governance, but it is not careful governance.

The 510(k) mechanism asks whether a device is substantially equivalent to an already marketed predicate. In radiology AI, that can be a reasonable regulatory screen for safety and intended use, but it is not the same as asking whether the software improves detection, reduces time to diagnosis, changes radiologist behavior appropriately, or avoids creating downstream work that the department cannot absorb. The JAMA Network Open review’s testing breakdown matters because it shows how rarely those deployment-facing questions were addressed in publicly available documentation before authorization. [1]

Prospective testing should not be inflated into more than it means. It is broader than a randomized controlled trial: data are collected forward in time, but the study may still be observational, limited, or conducted in a setting unlike the purchasing hospital. The review separately reported that only 2% of devices were supported by randomized controlled trials, so the 5% prospective-testing figure should not be read as a trial-quality figure. It is a lower bar than that, and still most devices did not clear it in the available documentation. [1]

The Missing Human Operator Is Not a Detail

Radiology AI is rarely a sealed box producing a final clinical answer without people around it. It may reprioritize a worklist, flag a suspected finding, prepopulate a measurement, segment an organ, or prompt a second look. In each case, the product’s real performance depends partly on the reader’s behavior: who trusts it, who ignores it, who overcorrects after a false alarm, and who has to explain the miss when the tool is silent.

That is why the 8% human-in-the-loop figure is so uncomfortable. A bench test or retrospective dataset can establish that an algorithm recognizes patterns under defined conditions. It cannot show whether a busy radiologist changes a report appropriately at 4 p.m., whether an alert adds urgency or just another interruption, or whether the tool nudges junior and senior readers in different directions. [1]

A 2024 Nature Medicine study, discussed in a 2025 clinical practice review, adds a useful caution rather than a universal rule: AI assistance helped high-performing radiologists maintain accuracy but did not meaningfully improve low performers. The point is not that AI only helps experts. It is that reader behavior and baseline performance can shape the effect of assistance, so a single aggregate performance number may not transfer cleanly into every reading group. [2]

A procurement committee should therefore avoid asking only whether the algorithm’s standalone sensitivity and specificity look acceptable. It should ask who interacts with the output, when the output appears, whether the reader can easily inspect it, how disagreement is handled, and whether the institution will measure behavior after deployment. The absent human operator in premarket testing becomes a local implementation task.

What Local Validation Has to Carry

The evidence gap does not prove that most FDA-cleared radiology AI devices are unsafe or ineffective. That would be too broad. It proves something narrower and more useful: many devices reach the market without publicly documented prospective, clinical, human-in-the-loop evidence, so the hospital must decide how much uncertainty it is willing to own.

Local validation is not a ceremonial pilot added after the purchasing decision is effectively complete. It is where the institution checks whether the product behaves acceptably on its own scanners, protocols, patient population, acquisition quality, disease prevalence, report templates, turnaround expectations, and staffing model. A tool trained and tested in one environment may still work well elsewhere, but that transfer should be demonstrated rather than assumed.

• Ask whether validation used prospective data, retrospective data, enriched test sets, or curated challenge cases, because each design answers a different operational question.
• Ask whether the tested workflow included radiologists, technologists, worklist routing, report generation, and escalation steps, not just algorithmic image interpretation.
• Ask whether the tested population resembles the institution’s own case mix, including scanner vendors, protocols, prevalence, comorbidities, and image quality variation.
• Ask what failure modes were observed: false positives that create downstream work, false negatives that create misplaced reassurance, and cases excluded from analysis.
• Ask what will be monitored after go-live, who reviews drift or alert fatigue, and what threshold triggers rollback, retraining, configuration changes, or contract escalation.

Those questions can feel slower than the purchasing process wants to be. They are still less expensive than discovering after deployment that the tool performs differently on portable chest radiographs, overflags a low-prevalence condition, delays reads by changing worklist order, or creates a second review queue nobody owns.

Success Metrics Need to Be Chosen Before Go-Live

A vendor may reasonably report model-level performance, but a radiology department has to define success at the workflow level. For one product, the important measure may be time to notification for a suspected critical finding. For another, it may be reduction in missed measurements, fewer unread backlogs, or improved consistency of follow-up recommendations. A product that increases detection while doubling nuisance alerts has not created the same operational value as a product that improves detection with manageable review burden.

The cleanest procurement documents separate model performance, reader performance, and system performance. Model performance asks what the algorithm outputs. Reader performance asks whether radiologists act differently and accurately with assistance. System performance asks whether the department’s actual targets improve without unacceptable side effects. FDA clearance may be relevant to the first category and necessary for market entry. The purchase decision sits across all three.

Market Scale Makes the Evidence Problem More Urgent

The volume of authorized AI/ML medical devices has continued to grow beyond the June 2024 cutoff of the systematic review. FDA counts reported through the end of 2025 show 1,451 AI/ML-enabled medical devices, with 1,104 in radiology, and separate year-end analysis reported 295 clearances in 2025 alone. Those figures are a later snapshot than the JAMA Network Open review and should not be compared as if they measure the same device set. They do show the scale at which procurement committees are now being asked to evaluate medical imaging AI tools. [3][4]

Readers who want the authorization-volume backdrop can start with the 2025 AI device authorization analysis. That context is useful, as long as the later market counts are not treated as a substitute for evidence about prospective clinical validation. [1][3][4]

The software character of the market also matters. Innolitics reported that 62% of 2025 AI/ML device clearances were Software as a Medical Device, and that 10% of 2025 clearances included Predetermined Change Control Plans, or PCCPs, for planned iterative updates. PCCPs are one sign that regulatory oversight is adapting to AI systems that may change after initial authorization. They do not remove the need for hospitals to monitor whether a deployed version continues to perform acceptably in local practice. [4]

For a fuller discussion of lifecycle governance, see how the FDA is reshaping AI medical device regulation through PCCPs, TPLC, and transparency requirements. For procurement teams comparing vendors after the evidence questions are clear, the AI medical imaging company landscape is more useful once it is read through this validation lens rather than as a shopping list.

What a Cleared Device Deserves

A cleared radiology AI product deserves attention. It has passed a regulatory filter, and many departments need tools that reduce friction, surface urgent findings, standardize repetitive tasks, or support readers working under pressure. Clearance should move a product into serious evaluation; it should not move it directly into trust.

The procurement burden is therefore specific. Require the vendor to show what was tested and how. Identify whether evidence was prospective, clinical, and human-in-the-loop. Decide whether the study environment resembles the hospital’s own workflow closely enough to matter. Run local validation before full deployment. Define success metrics before go-live. Monitor performance after release, including false positives, missed cases, reader response, alert burden, version changes, and drift.

FDA clearance is a necessary filter for regulated radiology AI devices. It is not a substitute for the institution’s own judgment about clinical fit, operational risk, and ongoing accountability.

References

1. FDA Approval of Artificial Intelligence and Machine Learning Devices in Radiology: A Systematic Review. JAMA Network Open. November 2025.
2. Clin Pract review discussing Nature Medicine 2024 radiologist-AI assistance findings. Clin Pract. September 2025.
3. Numbers from the FDA Show Radiology is Maintaining Its Lead. The Imaging Wire. March 2026.
4. Year in Review: AI/ML Medical Device 510(k) Clearances. Innolitics. December 2025.