A Practical Protocol for Medical Imaging AI Bias Evaluation

A hospital evaluating a medical imaging AI product usually gets a familiar packet: validation performance, intended use, sample screenshots, perhaps a fairness slide. For medical imaging AI bias evaluation, that is not enough. The practical question is narrower and harder: before a chest radiograph triage or detection model enters the worklist, can the hospital tell which patients, scanners, sites, labels, and clinical conditions were actually tested — and what will be monitored after deployment?

The useful answer is a lifecycle protocol. Dataset audit, model-level fairness assessment, and post-deployment monitoring are separate checks. A clean subgroup table cannot rescue weak labels. A strong test-set AUC cannot prove performance after a scanner mix changes. A post-market dashboard cannot compensate for a procurement decision that never asked whether race, ethnicity, insurance status, age, sex, or site were available for analysis.

Start by locating the bias, not by naming a fairness metric

Koçak and colleagues’ 2025 review is useful because it organizes radiology AI bias by where it enters the system: dataset bias, model bias, deployment bias, and human-factor bias.[1] That staging matters operationally. It changes the evidence a hospital should request, the analysis an informatics team can run, and the committee that owns the next action.

This table is not a taxonomy exercise. It is a procurement checklist with consequences. If a vendor describes demographic parity without showing whether the training and validation datasets contained usable demographic fields, the problem is upstream. If subgroup testing looks acceptable on a pooled external test set but the local hospital uses different portable radiography protocols, the unresolved question is deployment. If the model’s alert is technically correct but radiologists ignore it because it fires too often on a specific service line, the bias evaluation has moved into workflow.

The dataset audit is where many evaluations fail

The first audit should be boring, explicit, and uncomfortable: list every dataset used for training, tuning, internal validation, and external validation; describe the acquisition sites; identify the imaging equipment and protocols when available; state the label source; and report subgroup availability. If the tool is being evaluated for chest radiography, the hospital should know whether the data include inpatient, outpatient, emergency, ICU, portable, and non-portable studies in proportions that make sense for the intended use.

The subgroup fields are often the first hard stop. An RSNA article citing Yi and colleagues reported that only about 17% of 23 public chest radiograph datasets report race or ethnicity.[2] That does not prove every model trained on those datasets is biased. It proves something more practical: for many public imaging datasets, a full pre-training race or ethnicity audit is not possible from the released metadata. A fairness claim built on missing fields should be labeled as partial, not polished into certainty.

A usable dataset audit should separate three questions that are often blurred together. First, is a subgroup represented at all? Second, is the subgroup large enough to estimate performance with any stability? Third, does the subgroup carry different clinical prevalence, access patterns, scanner exposure, or label quality? Representation is not the same as statistical power, and neither is the same as clinical comparability.

Labels deserve their own audit

In imaging AI, the label is often treated as ground truth because the spreadsheet needs a column. It may be a radiology report label extracted by natural language processing, a billing code, a consensus annotation, a follow-up outcome, or a weak label inherited from a public dataset. These sources do not fail in the same way.

For a chest radiograph model, a report-derived label may reflect what was noticed, dictated, and encoded rather than everything visible on the image. If one patient group receives later imaging, lower-quality follow-up, or more ambiguous reports, the label can carry those care patterns into model development. A bias evaluation should therefore document the label source, extraction method, adjudication process, and whether label quality was checked across subgroups.

This is also where a hospital should ask for the negative class definition. “No pneumothorax,” “not mentioned in report,” “no ICD code,” and “normal chest radiograph” are not interchangeable. If the intended use is urgent triage, a false negative may delay review. If the label was weakly derived, the fairness analysis should say so plainly.

Site mix and scanner variation are not technical footnotes

External validation becomes less reassuring when it hides the acquisition environment. A model that performs well across pooled sites may still learn site-specific or protocol-specific shortcuts. Dataset documentation should identify whether images come from one institution or many, whether portable and fixed radiography are mixed, whether adult and pediatric populations are separated, and whether scanner or protocol metadata are available for stratified analysis.

Proxy variables are the quiet part of the audit. The model may never receive race, insurance status, or hospital unit as explicit inputs, but it may still infer patterns through image acquisition, positioning, embedded markers, disease prevalence, or care setting. That is why a dataset audit should not stop at the input fields shown to the model. It should ask what sensitive or site-specific information can be reconstructed from the image or metadata.

Choose the fairness metric for the clinical harm

Once the dataset is understood, the evaluation can move to model behavior. This is where many reviews get too neat. Demographic parity, equal opportunity, and equalized odds sound like interchangeable fairness options until disease prevalence differs across subgroups. Ricci Lara and colleagues emphasize that these criteria cannot all be satisfied simultaneously under differing prevalence conditions, which means the metric choice is a clinical and governance decision, not a default setting in a toolkit.[3]

For a detection task where the main harm is missed disease, subgroup false-negative rates and sensitivity may deserve priority. For a triage model that can overload a worklist, false-positive rates and positive predictive value may also matter because excess alerts can delay other patients and erode trust. For a model that assigns risk scores rather than binary labels, calibration by subgroup becomes important: a given score should mean similar risk across groups if clinicians are expected to act on it.

The protocol should require the evaluator to write down the metric rationale before seeing the most favorable table. If the disease is time-sensitive and underdiagnosis is the dominant concern, sensitivity gaps should not be buried behind an overall AUC. If the model is used to prioritize reads, the threshold should be evaluated at the operating point planned for clinical use, not only at an academic benchmark.

Subgroup error is not hypothetical

Seyyed-Kalantari and colleagues examined 707,626 chest radiograph images from MIMIC-CXR, CheXpert, and ChestX-ray14 and reported higher false-negative rates for female patients, Black patients, Hispanic patients, younger patients, and Medicaid-insured patients in AI algorithms applied to chest radiographs.[4] That finding is exactly the kind of result a bias evaluation is meant to surface: not whether the average model score looks acceptable, but whether the model underdiagnoses groups that may already face worse access to timely care.

The limitation matters too. The study used labels extracted from radiology reports with natural language processing, so the analysis inherits possible report-labeling errors and documentation patterns.[4] That does not make the result ignorable. It means a serious reader should carry both points at once: the underdiagnosis signal is clinically important, and the label source is part of the evidence quality.

A local evaluation can follow the same logic without pretending to reproduce the same study. Define the intended population. Select the clinical operating threshold. Report overall performance. Then report subgroup sensitivity, specificity, false-negative rate, false-positive rate, predictive values, and calibration where the sample size supports it. If subgroup fields are missing or too sparse, the report should say that directly and identify what proxy or prospective data collection, if any, will be used after deployment.

Shortcut learning makes local validation necessary

Bias evaluation also has to ask what the model learned. Gichoya and colleagues showed that deep learning models could predict self-reported race from medical images, including chest radiographs, even when clinical experts could not do so from the images.[5] The point is not that a deployed model necessarily uses race in a clinically harmful way. The point is that medical images can contain demographic signal through mechanisms that are not obvious to human reviewers.

Zech and colleagues provided another warning from pneumonia detection: models trained on chest radiographs could learn hospital-system features and care-setting markers, such as ICU versus outpatient differences, rather than portable disease concepts that transfer cleanly across sites.[6] In practice, this means a model can look competent in one environment because it recognizes acquisition context that correlates with disease there.

More recent work from the Ghassemi lab, reported by MIT News with the associated Nature Medicine study, connected demographic prediction capacity to fairness gaps and found that debiasing approaches did not transfer reliably across hospital systems.[7] That is a direct argument for deployment-specific evaluation. A mitigation that improves subgroup performance in one dataset should not be assumed to travel with the model into a different hospital.

Shortcut checks do not need to become speculative forensics. They can be practical: test performance by site, scanner, protocol, patient location, and acquisition type when those fields are available; compare internal and external validation behavior; inspect whether subgroup gaps widen after stratifying by acquisition context; and require the vendor to disclose known shortcut analyses or failure modes in a model card.

Use toolkits as instruments, not governance

Open-source and commercial tooling can make the evaluation less ad hoc. Banerjee and colleagues catalogued bias detection and mitigation tools relevant to radiology AI, including AIF360, Fairlearn, Aequitas, Google’s What-If Tool, and IBM OpenScale.[8] These tools can calculate subgroup metrics, compare thresholds, visualize trade-offs, and support repeatable reporting.

• AIF360 and Fairlearn can help compute and compare fairness metrics, but they require usable subgroup labels and a defined prediction task.
• Aequitas is useful for auditing error disparities across groups, especially when the evaluation is framed around false positives, false negatives, or threshold behavior.
• The What-If Tool can support interactive threshold and subgroup exploration, but it does not decide which clinical harm should dominate.
• OpenScale-style monitoring can support production oversight, but it still depends on local data feeds, outcome definitions, and governance review.

The common failure is to treat a toolkit output as the fairness evaluation itself. The software can calculate demographic parity difference. It cannot tell a stroke service whether a sensitivity gap is clinically unacceptable, whether a false-positive increase will destabilize overnight workflow, or whether the race field is missing in a way that makes the analysis incomplete.

Post-deployment monitoring is part of the bias evaluation

A model that passed pre-deployment review can still degrade after rollout. The case mix changes. A scanner is replaced. A hospital opens a new service line. Portable imaging volume rises. A reporting template changes the label feed. A triage model that was calibrated on one emergency department may behave differently across community sites in the same health system.

Monitoring should therefore repeat the same failure modes that mattered before deployment. If the pre-deployment concern was higher false negatives in a subgroup, the production dashboard should not monitor only volume and uptime. It should track subgroup performance when outcome labels become available, alert rates by site and acquisition type, missing-data rates, threshold behavior, and drift in input distributions.

Human-factor bias belongs here as well. A model can affect care through how clinicians respond to it. The monitoring plan should look for override patterns, delayed reads, alert fatigue, and whether specific units or patient groups receive systematically different downstream action after a flag. If the hospital cannot yet measure outcomes, it can still measure process: who saw the alert, how quickly it was reviewed, and whether the recommendation changed workflow.

Regulation is moving toward the same lifecycle logic

The regulatory direction is not the center of the protocol, but it is no longer background noise. The EU AI Act’s Article 10 requires data governance for high-risk AI systems, including attention to training, validation, and testing datasets that are relevant, representative, as complete as possible, and appropriately error-controlled; Article 61 addresses post-market monitoring for high-risk AI systems.[9] For high-risk systems where the provider is already subject to MDR or IVDR requirements, enforcement is phased, with obligations applying from August 2026 under the current implementation timeline.[9]

The European Society of Radiology’s 2025 recommendations also push in a practical direction, identifying gaps in how triage tools are classified and calling for model cards that document intended use, training data, performance, and limitations.[10] That is the kind of artifact a hospital AI committee can actually use: not as a substitute for local testing, but as a structured starting point for it.

In the United States, the FDA’s October 2025 request for information on real-world performance monitoring for AI-enabled medical devices signaled the same pressure point: premarket evidence alone does not settle how AI devices behave after clinical use begins.[11] The comment period ended in December 2025, so any final guidance or rulemaking should be checked at the time of a specific procurement or governance review.[11]

A reusable protocol for medical imaging AI bias evaluation

The protocol can be short enough to use in a committee meeting, but it needs enough structure to prevent a single favorable validation result from carrying the whole decision.

1. Define the clinical use case, target population, imaging modality, operating threshold, and workflow consequence of the model output.
2. Classify likely bias sources by stage: dataset, model, deployment, and human factor.
3. Audit datasets for subgroup fields, missingness, representation, site mix, scanner and protocol variation, label source, label quality, and proxy variables.
4. Document what cannot be audited because demographic, scanner, site, or outcome fields are missing.
5. Select fairness metrics with a clinical rationale before optimizing the presentation of results.
6. Report subgroup performance at the intended operating point, including uncertainty where sample sizes are small.
7. Test for shortcut behavior using site, acquisition, scanner, protocol, and patient-location stratification when available.
8. Record known limitations in a model card or equivalent governance document.
9. Monitor drift, subgroup degradation, alert burden, overrides, and downstream workflow effects after deployment.
10. Reopen the evaluation when the clinical setting changes, including new sites, new scanners, new protocols, new patient mix, or a changed threshold.

The hardest line in that protocol is often the fourth one. “Not available” is not a minor documentation issue when the missing field is needed to evaluate a plausible disparity. It should change the confidence assigned to the fairness claim and, in some cases, trigger prospective data collection before broader rollout.

A medical imaging AI system can be useful and still require a sharper bias evaluation. The goal is not to make every deployment wait for perfect evidence. It is to stop calling a model fully evaluated when the review has only measured average performance before clinical life begins.

References

1. Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects. PMC. 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC11880872/
2. Radiologists Share Tips to Prevent AI Bias. RSNA. 2025. https://www.rsna.org/news/2025/january/prevent-ai-bias
3. Fairness metrics for health AI: we have a long way to go. PMC. 2022. https://pmc.ncbi.nlm.nih.gov/articles/PMC10114188/
4. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine. 2021. https://www.nature.com/articles/s41591-021-01595-0
5. AI recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health. 2022. https://www.thelancet.com/journals/landig/article/PIIS2589-7500(22)00063-2/fulltext
6. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLoS Medicine. 2018. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002683
7. Study: Medical imaging AI models can exhibit bias across demographic groups. MIT News. 2024. https://news.mit.edu/
8. Shortcuts Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and Mitigation. Journal of the American College of Radiology. 2023. https://www.jacr.org/
9. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. EUR-Lex. 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
10. European Society of Radiology recommendations on the use of artificial intelligence in radiology. European Radiology. February 2025. https://link.springer.com/article/10.1186/s13244-025-01905-x
11. Request for Information on Real-World Performance Monitoring of AI-Enabled Medical Devices. FDA. October 2025. https://www.fda.gov/