Documented examples of algorithmic bias in healthcare AI

The most useful examples of algorithmic bias in healthcare AI do not begin with a model behaving mysteriously. They begin earlier, when someone decides what the model should predict. In the Optum risk algorithm studied by Obermeyer and colleagues, the target was not illness burden itself but healthcare spending. That choice looked administratively convenient. It was also where the distortion entered.

The algorithm was used to identify patients who might benefit from extra care management. Because Black patients in the claims data generated lower healthcare costs than white patients with the same number of chronic illnesses, the model treated lower spending as lower need. In the studied population, Black patients with the same chronic illness count cost about $1,800 less per year. At one risk threshold, only 17.7% of Black patients were identified for high-risk care; after recalibration to better reflect health need, that share rose to 46.5%.[1]

That is a cleaner lesson than the usual shorthand that “the algorithm was biased.” The proxied outcome was biased. The system optimized for a record of past resource use in a healthcare system where access, trust, geography, insurance friction, and clinical attention all affect how much care is paid for. A model can be statistically competent and still be trained to reproduce the wrong construct.

The audit implication is specific. Testing demographic parity at the end would have found a disparity, but not the repair. The first question should have been whether cost was a valid stand-in for need across groups. The relevant audit compares the proxy against clinical severity, chronic illness burden, avoidable deterioration, or another outcome closer to the intervention’s purpose. The study used claims data from 2014–2015, and Optum has said the algorithm was updated after the findings; the case remains important because the mechanism is common whenever utilization is treated as a neutral measure of need.[1]

Bias Enters at Different Points in the AI Lifecycle

The Optum case is a problem-framing failure. Other documented examples sit in different places: who was sampled, how labels were generated, which biological variants were included, what hidden correlations the model learned, and how a deployed system responded to cues from a patient-facing scenario. Treating these as one generic fairness problem hides the repair work.

Sampling Bias Is Sometimes Visible Before a Model Is Built

Some bias mechanisms do not require much forensic work. In dermatology AI, the imbalance can be counted before training begins. A 2021 analysis of 21 open-access dermatology image datasets found 106,950 images in total, but only 10 images from brown skin, one from dark brown or black skin, and none from people described as African, African-Caribbean, or South Asian.[2]

That evidence does not prove every dermatology model trained on such material will fail in the same way. It does show that an apparently large dataset can be small for the patients whose skin findings look different. “Large” is not a population property. It is only a file count until the sample is checked against the clinical variation the model will face.

The audit strategy follows directly. A dermatology dataset review should not stop at diagnosis categories and train-test splits. It needs skin tone, ancestry or ethnicity where available, imaging source, acquisition setting, and lesion type by subgroup. If those variables are absent, the absence is itself an audit finding because it prevents anyone from knowing where performance claims apply.

Chest X-Ray Models Show Bias After Sampling, Labeling, and Thresholding Interact

The chest X-ray evidence is harder to dismiss as a simple representation problem. Seyyed-Kalantari and colleagues studied classifiers trained on 707,626 chest X-ray images and found consistent underdiagnosis for female, Black, Hispanic, younger, and Medicaid patients. The important detail is not just that performance differed by subgroup. The pattern showed an inverse relationship between false-positive and false-negative rates across subgroups, consistent with selective algorithmic underdiagnosis: underserved patients were more often marked healthy when disease was present.[3]

That distinction matters operationally. A noisy model makes errors in many directions. A selectively underdiagnosing model changes who is delayed, who receives follow-up, and which clinician inherits a false sense of reassurance. In radiology triage, the harm is often the missing alert, not the dramatic wrong alert.

The paper also points to a less visible source of bias: label generation. Many chest X-ray datasets use natural language processing systems to convert radiology reports into training labels. If the report language is non-standard, ambiguous, or unevenly documented across sites and patient groups, the “ground truth” can already be distorted before the image model sees a pixel. Seyyed-Kalantari and colleagues discuss this as a limitation and a plausible contributor rather than a fully isolated causal mechanism.[3]

That caveat is useful. It prevents a false precision that audits cannot support. For chest X-ray classifiers, the audit cannot be limited to the final area under the curve. It has to inspect the provenance of labels, report-labeler performance, subgroup-specific false negatives, site composition, disease prevalence, and operating thresholds. If a model undercalls disease in Medicaid patients, the next question is not whether the network architecture is modern. It is whether the pipeline taught the model that the same radiographic signal deserves a different label, threshold, or confidence pattern in one subgroup.

Removing Race From Inputs Is Not a Fairness Strategy

A common procurement comfort is that a model does not use race as an input. Imaging models have made that comfort look thin. Research has shown that deep learning models can predict self-reported race from medical images, including chest X-rays, even when the route by which they do so is not clinically obvious.[4]

The lesson is not that race should always be fed into models. It is that withholding a protected attribute does not prevent a high-capacity model from using correlated features. Scanner type, disease patterns, body habitus, site-specific acquisition settings, and other signals can carry group information. Once those correlations exist, fairness testing needs protected attributes for evaluation even when the production model does not use them as explicit predictors.

This is where subgroup auditing becomes uncomfortable but necessary. A team cannot prove that a model behaves similarly across groups while refusing to measure group-specific behavior. The protected attribute may be inappropriate as a decision input; it is still often necessary as an audit variable.

Warfarin Dosing Shows a Different Failure: Missing Biological Variation

Not every biased healthcare algorithm is biased because it learned from unequal access or documentation. Some fail because the biology represented in the model is incomplete. Warfarin pharmacogenetic dosing algorithms misestimated dosing for African American patients when they excluded African-specific genetic variants. Hill and colleagues argued that variants more common in people of African ancestry needed to be incorporated to improve dosing accuracy.[5]

The COAG trial is a useful warning here. African Americans made up about 33% of the trial population, and genetic dosing was associated with higher rates of supratherapeutic INR among African American participants.[5] The relevant mechanism is not the same as the Optum mechanism. Cost was not the wrong proxy. Skin tone was not missing from an image bank. Instead, the model’s pharmacogenetic feature set did not capture variants that mattered for a subgroup.

The audit response is therefore different. A warfarin dosing algorithm needs variant coverage analysis, ancestry-sensitive validation, and outcome checks such as time in therapeutic range or supratherapeutic INR by subgroup. A generic fairness constraint on predictions would be a poor substitute for asking whether the model contains the biological variables required to dose safely.

Deployment Bias Can Appear in the Conversation Layer

The newer deployment-stage evidence is less settled but worth watching because it sits close to patient and clinician interaction. Bouguettaya and colleagues tested four large language models on 10 psychiatric cases and compared recommendations when race was indicated explicitly or implicitly through name cues. The models proposed inferior treatments in some race-cued versions, including omitting ADHD medications, suggesting guardianship, and focusing on alcohol reduction only for African American patients.[6]

A 10-case qualitative study cannot estimate how often this happens in clinical use. The authors themselves call for larger confirmatory work.[6] Its value is narrower and still important: it demonstrates a deployment mechanism. A clinically equivalent case can move through a language model differently when the text carries social cues. The bias is not only in a training table or an image archive; it can be triggered by names, phrasing, perceived social context, and the way a recommendation is presented to a human user.

For LLMs used in clinical support, the audit has to include counterfactual prompts. The same case should be tested with varied names, dialect cues, family structure, insurance cues, and race descriptions while holding the clinical facts constant. Reviewers then need to examine not just the final recommendation, but omissions, tone, escalation behavior, and whether the model shifts from treatment planning into social control language.

The Audit Has to Match the Mechanism

These examples are often grouped under one phrase, but they do not point to one fix. Proxy bias asks whether the target measures the right construct. Sampling bias asks who is missing. Label bias asks whether the ground truth is already uneven. Missing-variant bias asks whether the feature set represents clinically relevant biology. Hidden correlation asks what the model can infer despite input restrictions. Deployment cue sensitivity asks whether recommendations shift when social signals change while the medical facts stay constant.

That is why a single fairness dashboard is usually too crude. A dashboard can show that subgroup performance differs. It cannot, by itself, say whether the team should redefine the outcome, recollect data, relabel reports, add genetic variants, change thresholds, constrain a model, redesign human review, or block a use case until counterfactual deployment tests are complete.

The practical sequence is diagnostic: locate the lifecycle stage, identify the mechanism, measure the subgroup effect that mechanism would plausibly produce, and choose a repair that acts at that point. Teams looking for governance language after that diagnostic work can connect it to assurance frameworks such as the NIST AI Risk Management Framework for healthcare, but the framework is only useful if the underlying failure mode has been named precisely enough to test.

References

1. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 2019.
2. Representation of skin of colour in dermatology image datasets: a systematic review. The Lancet Digital Health, 2021.
3. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 2021.
4. AI recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health, 2022.
5. A systematic examination of the warfarin pharmacogenetic literature: implications for African Americans. Pharmacogenomics, 2014.
6. Evaluation of large language model performance and bias in psychiatric assessments. npj Digital Medicine, 2025.