Explainable AI Medical Device Requirements Across Four Regulators

A saliency map can be useful in a design review and still be almost useless in a submission if nobody can say who it is for, what decision it supports, where it is controlled, and how it will be maintained after the model changes. That is the practical problem inside explainable AI medical device requirements: regulators are not asking for one universal XAI artifact. They are asking different questions with different legal force.

The FDA’s transparency principles are influential and operationally helpful, but non-binding. The EU AI Act creates enforceable obligations for high-risk AI systems, including AI medical devices under MDR and IVDR. The MHRA-affiliated evidence base is most useful when deciding what clinicians can actually understand at the point of use. The IMDRF draft framework points toward lifecycle quality-system integration, but it was still a draft after consultation closed on June 10, 2026.[1][2][3][4][5]

That distinction matters before any engineering team starts selecting SHAP, LIME, counterfactuals, attention maps, model cards, or feature-importance plots. A regulator-facing explanation, a clinician-facing explanation, a patient-facing explanation, and a QMS-controlled lifecycle record are not interchangeable, even when they describe the same model.

The First Split Is Legal Force

The fastest way to lose clarity is to ask whether a jurisdiction “requires explainability” without asking what kind of instrument is doing the requiring. In a submission strategy, a non-binding FDA principle, a binding EU obligation, a pre-print clinical study, and an international draft framework do not carry the same weight.

For a manufacturer preparing U.S. and European pathways, this is not an academic distinction. A model card may help answer an FDA reviewer’s transparency question. In Europe, the same information may need to sit inside technical documentation, instructions for use, risk management records, and QMS procedures with traceability to the AI Act and existing medical device files.

FDA: Useful Principles, Not a Free-Standing Mandate

The FDA’s June 2024 transparency guiding principles are best read as a disciplined way to stop vague “explainability” claims from drifting through a file. They define transparency across six dimensions: Who, Why, What, Where, When, and How.[1]

• Who: the audience receiving the information, such as clinicians, patients, regulators, health systems, or internal reviewers.
• Why: the reason the information is needed, such as safe use, performance evaluation, oversight, procurement, or complaint investigation.
• What: the type of information disclosed, including model inputs, limitations, performance, intended population, training context, or uncertainty.
• Where: the location of the information, such as labeling, user interface, technical documentation, public materials, or controlled internal records.
• When: the point in the product lifecycle when the information is provided or updated.
• How: the format and method used to communicate the information.

That framework is genuinely useful because it forces the conversation away from a generic interpretability claim. A SHAP plot in a design history file, a short warning in labeling, a dashboard confidence indicator, and a reviewer-facing model summary may all be transparency materials, but they do different jobs. The FDA framework gives a manufacturer a way to explain those differences coherently.

It does not, however, turn every preferred transparency practice into a binding requirement. The same caution applies to FDA’s 2025 lifecycle direction around structured model information. FDA lifecycle draft guidance includes an example model card for structured metadata reporting, related to Good Machine Learning Practice principles on monitoring and transparency, but not as a mandatory universal template.[2]

In practice, FDA transparency work should be treated as part of the submission argument and lifecycle control strategy. It should help show that intended users understand what the device does, where it may fail, what performance evidence supports it, and what happens when the model is updated. It should not be presented internally as if a model card alone proves compliance.

Public Transparency Is Still Thin

The gap between regulatory aspiration and public disclosure is visible in the ACTR study published in npj Digital Medicine in 2025. The study reviewed 1,012 FDA-authorized AI/ML devices and found a mean transparency score of 3.3 out of 17. It also found that 51.6% of devices reported no performance metric at all, and only 1.5% had a predetermined change control plan.[3]

That result should not be overread. Public summaries may omit information contained in confidential submissions. Still, from a manufacturer’s side of the table, it is hard to ignore what the score says about the public record: many authorized devices leave clinicians, purchasers, and outside reviewers with little structured information about model characteristics.

EU AI Act: Explainability Becomes File-Building Work

The EU AI Act changes the posture because it is not merely asking manufacturers to think carefully about transparency. It creates binding obligations for high-risk AI systems. For AI/ML medical devices regulated under MDR or IVDR, the classification trigger is broad: all AI/ML medical devices regulated under MDR/IVDR are identified as high-risk AI systems under Article 6 and Annex II, including Class I software as a medical device incorporating AI for purposes of AI Act compliance.[4]

That does not mean every EU medical device question suddenly becomes an AI Act question. It means the manufacturer’s conformity strategy has to account for a second regulatory layer. A device may already have MDR or IVDR technical documentation, clinical evaluation, risk management, usability engineering, and post-market surveillance processes. The AI Act adds high-risk AI obligations that have to be mapped into those existing files rather than left in a separate innovation folder.

Article 13 is the center of the transparency discussion. It requires disclosure of accuracy levels, known limitations, system capabilities, and foreseeable circumstances that may affect performance. For MDR/IVDR devices, these obligations are described as enforceable 36 months after entry into force.[4]

Those are not cosmetic disclosures. Accuracy levels must connect to validation evidence and the intended population. Known limitations must connect to risk controls, residual risk communication, and instructions for use. Foreseeable circumstances affecting performance must connect to deployment conditions, input quality, data drift, workflow assumptions, and user responsibilities. If those items appear only in marketing language, they will be difficult to defend under technical documentation review.

Annex IV and the Combined Technical File

The EU file-building issue becomes sharper under Annex IV and Article 11(2). Article 11(2) allows MDR technical files and AI Act Annex IV documentation to be combined into a single file.[4]

That option is administratively attractive, but it is not a shortcut. A combined file still has to make the AI Act logic visible. A notified body reviewer should not have to infer AI transparency controls from scattered MDR materials. The file needs a clear route from intended purpose to AI system description, data governance, model performance, instructions for use, risk management, human oversight, post-market monitoring, and change control.

For manufacturers already operating under ISO 13485, Article 17 is equally important. Article 17 permits AI quality management system integration into an existing ISO 13485 system.[4]

That is where explainability becomes less about a visual output and more about controlled process. If the model changes, the explanation package may change. If the intended user changes, the explanation package may change. If post-market data show a subgroup performance issue, the limitation statement, risk controls, and user communication may need review. A QMS that cannot trigger those updates will not be rescued by an elegant explanation generated during development.

The defensible EU approach is therefore not “attach an XAI appendix.” It is to show where each transparency claim is generated, verified, approved, communicated, monitored, and updated. That is heavier work than the FDA principle set, not because the FDA is irrelevant, but because the EU obligation is enforceable in a different way.

Clinicians Do Not Need the Same Explanation as Regulators

The MHRA-affiliated study is useful because it does something many regulatory discussions avoid: it separates explanation types by user need. It is a pre-print, so it should not be treated as a final regulatory position or peer-reviewed consensus. But its findings are directly relevant to the human factors and clinical safety side of AI medical device transparency.[5]

The study found that XAI explanations improved clinician diagnostic accuracy, while also introducing automation bias. In five of six cases, clinicians shifted to align with the AI.[5]

That is the kind of finding that should change design controls. If an explanation increases appropriate reliance in some cases and inappropriate alignment in others, then the manufacturer cannot simply claim that more explanation equals safer use. The user interface, training materials, warnings, local explanation format, and human oversight claims all need to be tested against foreseeable misuse and overreliance.

The same study also reports that when an interpretable logistic regression model with AUC 0.90 and a random forest model with AUC 0.96 matched black-box performance sufficiently for the task, regulators expected the simpler model.[5]

That does not create a universal rule that simple models are always required. It does support a familiar regulatory expectation: if a manufacturer chooses a less interpretable architecture, the file should explain why the added complexity is justified by clinical performance, risk reduction, usability, or other relevant benefit. “The neural network performed well” is not the same as a controlled model-selection rationale.

Local Explanations, Counterfactuals, and Visual Burden

Global model logic helps reviewers and internal teams understand how a model behaves across development and validation. Clinicians often need something narrower: why this output appeared for this patient, image, waveform, or measurement set at this moment. That is a local explanation problem, and it has different failure modes.

The MHRA-affiliated study reports that LIME stability required 20 or more runs per instance. It also states that counterfactual explanations must exclude immutable features, and that simplified aggregated visualizations, identified as ExMatrix, significantly improved clinician interpretability compared with full matrix views.[5]

Each of those findings translates into a different control. If local explanations are unstable, the manufacturer has to define how they are generated and validated. If counterfactuals propose changes to immutable features, they may be clinically confusing or ethically inappropriate. If full visualizations overload clinicians, the safer explanation may be a simplified display with clearer limits rather than a technically richer output that nobody can use correctly.

This is also where patient-facing explanation diverges from clinician-facing explanation. A patient may need a plain-language description of the device’s role, the meaning of an output, and whether a clinician remains responsible for the decision. A clinician may need uncertainty, local drivers, contraindicated use conditions, and performance caveats. A reviewer may need the evidence chain. A complaint investigator may need the exact model version, input conditions, output, user action, and post-market signal history.

IMDRF: The Harmonization Signal Is Lifecycle Control

The IMDRF draft framework should be read as a harmonization signal, not settled law. The April 2026 draft was open for consultation until June 10, 2026, and the final version may differ.[6]

Its direction is still important. The draft emphasizes transparency and explainability through quality management system integration, risk management processes, ISO 14971, AAMI TIR 34971:2023, and human oversight throughout the AI lifecycle.[6]

That is the right place for explainability to land if it is going to survive real device operations. Development teams can generate explanations during model training. Regulatory teams can describe them in submissions. But post-market teams inherit the problem when the model degrades, the user population shifts, a complaint arrives, or a planned update changes the basis for a prediction. The explanation package has to be maintained as part of the lifecycle, not preserved as a static development artifact.

What a Multi-Jurisdictional Explainability Package Should Contain

For a manufacturer seeking both U.S. and European access, the practical answer is not to build one “explainability deliverable.” It is to build a controlled package with different artifacts for different audiences, each mapped to the regulatory purpose it serves.

This package should begin with an audience matrix. The same model output may need a reviewer-facing technical rationale, a clinician-facing local explanation, a patient-facing plain-language statement, and an internal traceability record. Those materials should be consistent, but they should not be identical.

It should also include a model-selection rationale. If an interpretable model performs adequately for the intended purpose, the file should explain why a more complex model is still necessary if one is selected. If a black-box method materially improves clinically relevant performance, the manufacturer should show how the residual opacity is mitigated through validation, user information, oversight, and post-market monitoring.

The clinician-facing layer should be tested like a safety-relevant interface, not decorated like a dashboard. Local explanations should be assessed for stability, comprehension, workflow fit, and overreliance. Counterfactuals should be constrained so they do not recommend impossible or immutable changes. Visualizations should be simplified when simplification improves correct interpretation.

The EU layer needs explicit legal mapping. Article 13 transparency information should not be buried in general labeling. Annex IV documentation should not be assumed to exist because the MDR file is large. Article 17 QMS integration should be visible in procedures for design change, supplier control where relevant, post-market surveillance, complaint handling, CAPA, and periodic review of model performance.

The FDA layer can use the six transparency dimensions as a structure for the same package. Who receives the information, why they receive it, what it contains, where it appears, when it is updated, and how it is communicated are reviewable questions even when the framework itself is non-binding.

What This Comparison Does Not Cover

This comparison is limited to the FDA, EU AI Act, MHRA-affiliated evidence, and IMDRF draft framework discussed here. It does not assess China’s NMPA, Japan’s PMDA, or other national approaches. A truly global launch plan would need a separate jurisdictional analysis for those markets.

It also does not assume that every public transparency gap means the confidential submission was weak. Public authorization summaries, technical files, and internal design records are different evidence spaces. The compliance risk is not merely whether the public can see every detail; it is whether the manufacturer can retrieve, justify, and update the right explanation for the right audience when challenged.

The Defensible Posture

Explainable AI medical device requirements are now too specific to be handled with a single XAI output. The EU AI Act is more enforceable. The FDA is more principle-based. The MHRA-affiliated evidence is more granular about clinician usability and automation bias. IMDRF is pushing the discussion toward lifecycle QMS integration.

A defensible manufacturer response is a jurisdiction-aware explainability package: audience-specific explanations, technical documentation, model-selection rationale, clinical usability evidence, risk controls, human oversight, QMS traceability, and post-market update discipline. That package is less elegant than a single model card or saliency map. It is also much more likely to survive a submission review, a notified body question, a QMS audit, and a complaint investigation.

References

1. Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles, FDA, June 2024.
2. Good Machine Learning Practice for Medical Device Development: Guiding Principles, FDA.
3. Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices, npj Digital Medicine, 2025.
4. Navigating the EU AI Act: implications for regulated digital medical products, npj Digital Medicine.
5. Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations, arXiv, 2025.
6. IMDRF consults on draft guidance for AI-enabled medical devices, RAPS, April 2026.