Demographic Encoding and the Limits of Fairness in Medical Imaging AI: Why In-Distribution Mitigation Fails Under Distribution Shift

Introduction: The Promise and Perils of AI in Medical Imaging

Artificial intelligence has moved from the research lab into the radiology reading room, the dermatology clinic, and the ophthalmology suite with remarkable speed. As of early 2026, the FDA has authorized over 1,100 AI-enabled medical devices, the vast majority for imaging applications. Meta-analyses report average diagnostic accuracies above 90% for tasks like lung cancer classification on CT. These numbers suggest a technology approaching clinical maturity.

Yet a growing body of evidence reveals a more troubling picture. The same models that achieve near-human performance on benchmark datasets can fail catastrophically when deployed on populations, institutions, or imaging protocols they were not trained on. Worse, the failures are not random: they systematically disadvantage older patients, racial minorities, and other groups already underserved by healthcare. A 2024 study in Nature Medicine found that chest X-ray classifiers could predict a patient's self-reported race with accuracy far exceeding that of board-certified radiologists — not because the models were clever, but because they had learned to use demographic attributes as predictive shortcuts.

This article examines the mechanism of demographic shortcut learning in medical imaging AI, the evidence that fairness achieved during model development does not transfer to new deployment settings, and what developers and clinical adopters can do about it. It is written for radiologists, clinical researchers, AI developers, and regulatory professionals who need to understand the documented failure modes of these systems — not as abstract academic concerns, but as concrete barriers to safe, equitable deployment.

Bias Sources Across the AI Lifecycle in Medical Imaging

To understand why demographic shortcut learning occurs and why it resists standard mitigation, we first need a structured picture of where bias enters medical imaging AI systems. A comprehensive 2025 review by Koçak et al. organizes bias sources across three lifecycle phases: dataset construction, model training, and clinical deployment.

Dataset Bias

Dataset bias is the most extensively documented source. It takes several forms:

• Demographic imbalance: Training datasets drawn from single institutions or narrow geographic regions systematically underrepresent certain age groups, racial categories, and disease prevalences. A model trained primarily on chest X-rays from a tertiary hospital in one city may never have seen the image characteristics produced by a portable X-ray machine in a rural clinic.
• Institutional bias: Imaging protocols, equipment manufacturers, and post-processing pipelines vary across institutions. Models learn these institutional signatures as predictive features, which then fail when the institution changes.
• Annotation bias: Ground-truth labels reflect the expertise, training, and potential biases of the annotating clinicians. When reference standards are established by consensus panels that lack demographic diversity, the resulting labels may encode systematic diagnostic disparities.

Modeling Bias

During training, models can amplify existing dataset biases through propagation bias — where the model learns not just the intended signal (e.g., the presence of a lung nodule) but also spurious correlations (e.g., the age-related changes in bone density that happen to correlate with nodule prevalence in the training set). Data leakage, where information from outside the intended input features inadvertently enters the training process, is another well-documented mechanism.

Deployment Bias

Even a model that performed well on held-out test data can degrade in deployment due to concept drift (changes in the underlying disease prevalence or imaging technology over time), automation bias (clinicians over-relying on AI recommendations), and algorithmic aversion (clinicians dismissing correct AI outputs). These deployment-phase biases are the least studied but arguably the most consequential for patient outcomes.

Demographic Shortcut Learning: Evidence Across Radiology, Dermatology, and Ophthalmology

The most direct evidence for demographic shortcut learning in medical imaging comes from Yang et al. (2024), published in Nature Medicine. The study trained 3,456 models on MIMIC-CXR, the largest publicly available chest X-ray dataset, and systematically evaluated how well these models could predict demographic attributes — age, sex, and self-reported race — from the images alone.

The results were striking. Disease classification models predicted self-reported race from chest X-rays with an accuracy far exceeding that of three board-certified radiologists who were given the same task. This is not because race is visibly encoded in lung tissue; it is because the models learned to exploit subtle correlations between demographic attributes and image features — bone density, soft tissue distribution, lung field shape — that are present in the training data but are not causally related to the disease being diagnosed.

The study quantified the relationship between demographic encoding strength and fairness gaps. For age prediction on the 'No Finding' label in MIMIC-CXR, the correlation between how strongly a model encoded age and the size of its fairness gap was R = 0.82 (p = 4.7 × 10⁻⁸). In practical terms, models that were better at guessing a patient's age from their chest X-ray were also more likely to make systematically different error rates for older versus younger patients.

The magnitude of these fairness gaps is clinically meaningful. The study documented a 30% false negative rate (FNR) disparity between the oldest patients (ages 80–100) and the youngest (ages 18–40) for certain disease classification tasks. A model that misses 30% more cancers in elderly patients than in young adults is not merely unfair — it is unsafe.

Crucially, the phenomenon is not limited to chest radiography. Yang et al. replicated the analysis on the ISIC 2019 dataset for dermatology and the ODIR dataset for ophthalmology, finding similar patterns of demographic encoding in skin lesion classifiers and retinal image analyzers. This suggests that demographic shortcut learning is a general property of deep learning models applied to medical images, not an artifact of a single modality or dataset.

Local vs. Global Optimality: Why In-Distribution Fairness Does Not Transfer

The standard response to demographic shortcut learning is to apply debiasing algorithms during training. Methods like Domain-Adversarial Neural Networks (DANN) and Group Distributionally Robust Optimization (GroupDRO) are designed to reduce a model's reliance on demographic attributes by penalizing the model when it can predict those attributes from its internal representations.

These methods work — up to a point. Yang et al. showed that DANN and GroupDRO successfully reduced fairness gaps on held-out test data drawn from the same distribution as the training set. Models that were Pareto-optimal for fairness in-distribution (ID) achieved near-parity in error rates across demographic groups. This is the result that appears in most published evaluations of debiasing techniques, and it has led to cautious optimism that algorithmic fairness is achievable.

The problem is that this fairness does not transfer to out-of-distribution (OOD) settings. When the same models were evaluated on datasets from different institutions, different patient populations, or different imaging protocols, the fairness gains disappeared. In some cases, the correlation between ID fairness and OOD fairness was negative: models that looked fairest on the training distribution were actually the least fair when deployed elsewhere.

This is the central counterintuitive finding of the Yang et al. study, and it has profound implications for how we evaluate and select medical imaging AI models. The researchers tested this across 42 different OOD test settings — combinations of datasets, disease labels, and demographic attributes — and found a consistent pattern: selecting models based on minimum demographic attribute encoding (rather than minimum in-distribution fairness gap) produced models that were more globally optimal for OOD fairness.

Why does this happen? The explanation lies in the nature of distribution shift. When a model is trained to minimize fairness gaps on a specific dataset, it can do so by learning dataset-specific correlations that happen to equalize error rates across groups. These correlations may have nothing to do with the underlying disease pathology. When the dataset changes, those correlations break, and the fairness gains vanish. A model that encodes less demographic information in the first place, even if it has a slightly larger fairness gap on the training set, is more robust because it has not learned brittle, dataset-specific shortcuts.

Mitigation Strategies: What Works and What Doesn't Under Distribution Shift

Given that in-distribution fairness does not transfer, what mitigation strategies actually improve OOD performance? The evidence points to a combination of data-level, algorithm-level, and evaluation-level interventions.

Data-Level Interventions

The most effective data-level strategy is multi-center training. A systematic review by Suleman et al. (2025) examined six peer-reviewed studies (2022–2025) that reported both internal and external validation of AI diagnostic models on CT and MRI. The review found that models trained on multi-center cohorts consistently produced smaller performance gaps between internal and external validation compared to single-center trained models.

Data augmentation using generative adversarial networks (GANs) also shows promise. In one study reviewed by Suleman et al., GAN-based augmentation raised external AUC from 0.836 to 0.933 — a substantial improvement that brought external performance close to internal levels. This suggests that synthetic data can help bridge the gap between training distributions and deployment distributions, particularly for underrepresented subgroups.

Algorithm-Level Interventions

In-processing methods like adversarial debiasing (DANN), GroupDRO, and distributionally robust optimization remain valuable tools, but their limitations under distribution shift must be acknowledged. The evidence from Yang et al. suggests that these methods should be evaluated not on their ability to minimize ID fairness gaps, but on their ability to reduce demographic encoding — which is a more stable property across distributions.

Preprocessing strategies — re-weighting training samples, re-sampling to balance demographic groups, and applying GAN-based augmentation — address bias at the data level before it enters the model. These are complementary to in-processing methods and may be more robust because they do not depend on the model's ability to learn and unlearn dataset-specific correlations during training.

Evaluation-Level Interventions

The most actionable finding from the Yang et al. study is a change in model selection criteria. Instead of selecting the model with the smallest fairness gap on the validation set, developers should select the model with the minimum demographic attribute encoding. This requires adding demographic prediction tasks to the evaluation pipeline — measuring how well the model can predict age, sex, and race from its internal representations or outputs — and using those measurements as selection criteria.

External Validation Performance: Evidence from Systematic Review

The generalizability problem is not theoretical. The Suleman et al. systematic review provides concrete numbers: across six studies that reported both internal and external validation, internal-validation AUC ranged from 0.76 to 0.95, with sensitivities typically above 85% and specificities above 68%. On external validation, AUC declined by a median of approximately 0.03, but specificity drops were more dramatic — up to 24 percentage points in some cases.

A specificity drop of 24 percentage points means that a model that correctly ruled out disease in 90% of healthy patients during validation might correctly rule out disease in only 66% of healthy patients in a new clinical setting. The practical consequence is a flood of false positives — unnecessary follow-up tests, patient anxiety, and wasted clinical resources.

All six studies in the review showed that AI models underperformed on external data despite strong internal performance. This is not a problem that affects only poorly designed models; it affects models that have passed internal validation with flying colors. The finding reinforces the need for mandatory external validation as a precondition for clinical deployment — a standard that is not yet universally applied.

Regulatory and Clinical Implications: EU AI Act, FDA, and Deployment Requirements

The evidence on demographic encoding and OOD fairness failure has direct implications for regulatory compliance and clinical deployment. These are not abstract research findings; they map onto specific requirements in emerging regulatory frameworks.

EU AI Act: Article 10 and Bias Examination

The EU AI Act, which entered into force in 2024 and will be fully applicable by 2027, classifies medical imaging AI as a high-risk AI system. Article 10 of the Act requires that high-risk systems have bias examination and mitigation measures in place. The evidence from Yang et al. suggests that current bias examination practices — which typically evaluate fairness on held-out test data from the same distribution — are insufficient. A model that passes an in-distribution fairness audit may still fail catastrophically under distribution shift.

The Act's requirement for continuous monitoring post-deployment is therefore critical. Developers and deployers must track not just overall performance metrics, but also fairness metrics across demographic subgroups, and must have mechanisms in place to detect when distribution shift is degrading fairness.

FDA and Local Validation Requirements

The FDA's 510(k) clearance pathway for AI/ML-enabled medical devices does not currently require external validation on demographically diverse, geographically distinct datasets as a condition of clearance. The evidence reviewed here suggests that this is a gap. Many commercial AI imaging tools on the market now face these generalizability challenges, and local validation before deployment — testing the model on the specific patient population, imaging equipment, and clinical workflows of the deploying institution — is emerging as a best practice.

Continuous Monitoring and Model Updating

The finding that ID-to-OOD fairness correlation can be negative has implications for post-market surveillance. If a model's fairness on the training distribution is not predictive of its fairness in deployment, then periodic re-evaluation on local data is essential. This is consistent with the FDA's draft guidance on predetermined change control plans for AI/ML devices, which envisions a framework for continuous learning and updating while maintaining safety and effectiveness.

Recommendations for Developers and Clinical Adopters

The evidence reviewed in this article supports a set of actionable recommendations for both developers building medical imaging AI and clinical adopters evaluating tools for deployment.

For Developers

• Prioritize model selection criteria that minimize demographic attribute encoding over in-distribution fairness metrics. Add demographic prediction tasks to your evaluation pipeline and use encoding strength as a primary selection criterion.
• Mandate external validation on demographically diverse, geographically distinct datasets before claiming generalizability. Single-center validation is insufficient for clinical deployment decisions.
• Adopt multi-center training strategies whenever possible. Models trained on data from multiple institutions are more robust to distribution shift.
• Use GAN-based augmentation to generate synthetic training data for underrepresented demographic subgroups. The evidence shows this can substantially improve external performance.
• Implement continuous monitoring pipelines that track both overall performance and fairness metrics post-deployment, with automated alerts when distribution shift is detected.

For Clinical Adopters

• Require vendors to provide external validation results on datasets that match your patient population and imaging equipment. Do not accept internal validation alone as evidence of performance.
• Conduct local validation before full deployment. Test the model on a representative sample of your own patients and compare its performance against your current standard of care.
• Monitor for fairness degradation over time. Even a model that performs well at deployment may degrade as your patient population, imaging protocols, or disease prevalence change.
• Be skeptical of models that report only in-distribution fairness metrics. Ask vendors for demographic encoding measurements and OOD validation results.
• Establish a governance process for model updates. When a vendor releases a new version, the local validation and fairness assessment should be repeated before the update is deployed.