What Makes Healthcare AI High-Risk Under the EU AI Act?

A healthcare AI system is not high-risk under the EU AI Act simply because it appears in a hospital, touches a patient record, or uses clinical language. The classification turns on a more specific question: which legal pathway does the system enter through, and what role does it play in the decision that follows?

For healthcare teams, the practical answer usually starts with Article 6. There are two routes into high-risk status. First, an AI system is high-risk when it is itself a product, or a safety component of a product, covered by listed EU product-safety legislation and the product is required to undergo third-party conformity assessment before being placed on the market or put into service. For healthcare, that means the Medical Devices Regulation and In Vitro Diagnostic Medical Devices Regulation matter immediately, but only through the notified-body trigger built into Article 6(1).[1]

Second, even if the system is not caught through the medical-device route, it may still be high-risk if its intended purpose falls within one of the Annex III use cases. In healthcare-adjacent settings, the most relevant Annex III categories include emergency healthcare patient triage, access to essential private and public services and benefits, life and health insurance risk assessment and pricing, and certain biometric identification or verification systems.[2]

That classification question is separate from the compliance calendar. Deferrals, transition rules, and phased enforcement may affect when obligations apply. They do not change the basic logic for deciding whether the system belongs in the high-risk bucket in the first place.

Start With The Product Route, Not The Hospital Setting

The cleanest healthcare case is radiology computer-aided detection or diagnosis software. If a CADe or CADx tool is medical-device software and its conformity assessment requires notified-body involvement under the MDR, the EU AI Act does not ask whether the hospital keeps a radiologist in the loop. Article 6(1) makes the AI system high-risk because of its relationship to the regulated product pathway.[1]

That point is often missed in classification memos. The trigger is not a loose statement that the product is “medical.” The trigger is that the AI system is a product, or a safety component of a product, covered by the listed Union harmonization legislation and subject to third-party conformity assessment. In a medical-device file, the important entries are therefore the intended medical purpose, the MDR or IVDR status, the applicable conformity assessment route, and whether a notified body is involved.[1]

A mammography CAD tool that highlights suspicious regions for radiologist review illustrates why “human oversight” does not answer the classification question. The radiologist still interprets the image. The AI output may still shape attention, prioritization, follow-up, and documentation. If the software sits inside the MDR route requiring notified-body assessment, it is high-risk under Article 6(1) before anyone reaches the softer arguments about workflow design.[1]

Then Ask Whether Annex III Catches The Use Case

Some healthcare AI will not be classified as medical-device software, or the product-route analysis may be uncertain. That does not end the review. Article 6(2) brings Annex III systems into high-risk status when they are referred to in that annex, subject to the limited Article 6(3) derogations.[1][2]

The Annex III categories most likely to matter around healthcare are not a general list of “medical AI.” They are specific use cases. Emergency patient triage appears under access to and enjoyment of essential private services and essential public services and benefits. Systems used to evaluate eligibility for healthcare benefits can fall under the benefits category. AI used for risk assessment and pricing in life and health insurance is separately named. Biometric systems used for remote biometric identification, or in certain identification and verification contexts, require their own analysis under Annex III.[2]

That specificity matters. A general health FAQ chatbot that gives non-urgent information is not the same classification problem as a chatbot used to sort emergency symptoms, direct patients away from urgent care, or assign acuity. Scheduling software, billing automation, and ordinary administrative tools should not be swept into high-risk status merely because a hospital uses them. The intended purpose and the consequence of the output do the work.

Sepsis Prediction Tests Both Pathways

A sepsis early-warning system is a useful classification exercise because it resists a one-line answer. In many implementations, the software analyzes patient data to identify a clinical condition or deterioration risk and prompts assessment or intervention. If its intended purpose makes it medical-device software and the applicable MDR route requires notified-body involvement, Article 6(1) is the direct path to high-risk status.[1]

If a vendor argues that the system is not caught through the MDR route, the Annex III analysis still remains. A sepsis alert may influence how quickly a patient is reviewed, whether escalation occurs, and which clinical team is pulled into the case. Depending on deployment, it may function close to emergency triage or materially affect access to urgent care resources. That is not automatically the same as every bedside score or dashboard, but it is enough to make a casual “not high-risk because a clinician decides” conclusion difficult to defend.

The compliance file should therefore avoid broad labels and record the actual workflow. Who receives the alert? Is it interruptive or passive? Does it change queue position, escalation rules, or treatment timing? Is the clinician reviewing a complete independent assessment, or is the AI output one of the first signals that starts the assessment? These facts are not decorative. They determine whether Article 6(3) is even a plausible argument.

Article 6(3) Is A Narrow Filter

Article 6(3) gives providers a way to avoid high-risk classification for certain Annex III systems, but only where the system does not pose a significant risk of harm to health, safety, or fundamental rights, including by not materially influencing the outcome of decision making. The provision identifies narrow situations: performing a narrow procedural task, improving the result of a previously completed human activity, detecting decision-making patterns or deviations without replacing or influencing a prior human assessment, or performing a preparatory task to an assessment.[1]

In healthcare, those words should be read against the workflow, not against the marketing description. A system that merely formats a clinician’s already completed note is different from a system that flags an emergency condition before a clinician has formed an assessment. A tool that checks whether required fields are complete is different from a tool that recommends whether a patient should be seen now, later, or not at all.

An AI scribe is the useful counterexample. If it listens to a clinical encounter, drafts documentation, and supports a clinician who reviews and signs the final note, it is more likely to sit in preparatory work or improvement of a completed human activity. That does not make it free of privacy, cybersecurity, accuracy, procurement, or professional-liability issues. It does mean high-risk classification should not be forced merely because the transcript lives in the medical record.

The same reasoning does not transfer cleanly to a sepsis model or emergency triage tool. If the AI output changes who is seen first, which nurse is alerted, whether a physician is paged, or whether a care pathway starts, it is hard to describe the system as merely procedural or preparatory. The human may remain formally responsible, but the AI has already shaped the clinical field in which that responsibility is exercised.

Borderline Triage Needs A Conditional Answer

A symptom-checking chatbot shows why a classification answer often has to stay conditional until the intended use is pinned down. If it offers general wellness information, explains when to seek professional care in broad terms, or routes users to a human without assigning urgency, the Annex III emergency-triage analysis may be weaker.

If the same chatbot is used by a provider, insurer, telehealth platform, or emergency service to classify symptoms, assign urgency, determine whether a patient should access emergency care, or place patients into different response queues, the analysis changes. Annex III expressly includes AI systems used to evaluate and classify emergency calls by natural persons or to dispatch, or establish priority in the dispatching of, emergency first response services, including medical aid.[2]

That does not mean every chatbot with the word “symptom” in its interface is high-risk. It means the classification file should describe the operational consequence of the answer. A patient-facing explanation tool and an emergency prioritization system may share a conversational interface while occupying different legal positions.

Health Insurance Scoring Shows Why Profiling Cannot Be Treated As A Footnote

Health insurance risk-scoring is the place where Article 6(3)’s final sentence becomes unavoidable. Annex III identifies AI systems intended to be used for risk assessment and pricing in relation to natural persons in the case of life and health insurance.[2]

If the system profiles natural persons, the Article 6(3) derogation is not available. Article 6 states that an AI system referred to in Annex III shall always be considered high-risk where the AI system performs profiling of natural persons.[1]

That rule is easy to understate because profiling can sound like a data-protection issue rather than a classification blocker. For an insurance model that scores applicants or members, stratifies people into risk bands, or influences price or coverage terms, the profiling question belongs in the first page of the classification assessment, not in an appendix.

What The Classification Record Should Contain

A defensible healthcare AI classification record should read less like a policy essay and more like a decision log. It should identify the system, intended purpose, launch market, users, affected persons, workflow location, and the decision or action influenced by the output. It should then state which Article 6 pathway was assessed and why.

• For Article 6(1), record whether the AI is a medical device or safety component, the MDR or IVDR route, and whether notified-body involvement is required.
• For Article 6(2), identify the specific Annex III category, not a generalized healthcare-risk label.
• For Article 6(3), explain which derogation is claimed and why the system does not materially influence the relevant decision outcome.
• For profiling, state explicitly whether the system profiles natural persons; if it does, do not rely on the Article 6(3) derogation.
• For non-high-risk Annex III self-assessments, preserve the documentation required by Article 6(4) and account for EU database registration under Article 49 where applicable.

The practical result is narrower than many healthcare AI summaries suggest, but still demanding. Most clinical AI that functions as regulated medical-device software and requires notified-body involvement will be high-risk through Article 6(1). Some non-device healthcare AI will be high-risk through Annex III. Article 6(3) can remove some Annex III systems from high-risk status, but it is not a general human-in-the-loop exception. Profiling of natural persons closes that route altogether.

For the separate question of when obligations phase in, use the enforcement-timeline analysis rather than changing the classification answer to fit the calendar.

References

1. Article 6: Classification Rules for High-Risk AI Systems, artificialintelligenceact.eu.
2. Annex III: High-Risk AI Systems Referred to in Article 6(2), artificialintelligenceact.eu.