A Health System AI Procurement Checklist

The most expensive AI procurement mistake usually happens before the first vendor opens a slide deck. A committee agrees to “see what is out there,” schedules three demos, and only afterward tries to decide which patient population, clinician action, escalation pathway, and performance threshold the tool is supposed to serve. By then the conversation has already shifted from clinical need to product comparison.

A health system AI procurement checklist should reverse that sequence. Before comparing tools, the health system should name the workflow, population, data environment, minimum acceptable performance, safety owner, and implementation burden. The demo should be an examination of those requirements, not a guided tour of the vendor’s favorite use case.

That discipline is no longer optional. In 2024, 71% of U.S. hospitals reported using predictive AI, up from 66% in 2023, while governance practices remained uneven; 74% reported evaluating all or most models for accuracy, and fewer evaluated all or most models for bias. The same ONC brief defines predictive AI broadly, so it should not be read as a complete map of generative or autonomous clinical AI use, but it does show the direction of travel: hospitals are already buying and using these tools faster than many committees have matured their evaluation process.[1]

Start Before the Demo

The first gate is not whether the model is impressive. It is whether the health system can describe the job it wants done without using the vendor’s language.

For a clinical AI tool, a procurement packet should be incomplete until it answers five questions: Which patient population is in scope? Which decision or task changes if the output appears? Who receives the output, and in what part of the day? What is the minimum performance needed to justify that interruption? What happens when the model is wrong?

Those questions are not administrative niceties. They determine whether a reported model metric has any operational meaning. The external validation of Epic’s sepsis model at Michigan Medicine is the cautionary example worth keeping close: the model showed AUC values in the 0.76 to 0.83 range, but in a cohort of 27,697 patients it had a 33% sensitivity and an 88% false-positive rate. AUC did not tell the nurse manager how many alerts would arrive, how many would be wrong, or whether the right patients would be caught early enough to change care.[2]

A tool can clear a model-development bar and still fail a hospital. Procurement needs to ask whether the product works in the institution’s clinical lane, not whether it worked somewhere under conditions the health system cannot reproduce.

The Seven Domains That Should Be in the Packet

A useful checklist does not treat every AI purchase like an FDA submission. A low-risk administrative summarization tool does not need the same scrutiny as a deterioration model that pages a rapid response nurse. The point is proportionality: the higher the clinical consequence, automation level, patient-facing effect, or PHI exposure, the more evidence and accountability the committee should require.

1. Clinical Use Case and Workflow Definition

This is the domain that decides whether the rest of procurement is meaningful. A vague use case such as “reduce sepsis mortality,” “improve imaging throughput,” or “support discharge planning” is not ready for vendor comparison. It names an aspiration, not a workflow.

A procurement-ready use case should specify the clinical moment. For example, the committee should know whether a deterioration model is meant to flag patients in the emergency department, on medical-surgical floors, or across the entire inpatient census; whether it runs continuously or at defined intervals; whether the recipient is a bedside nurse, charge nurse, covering physician, rapid response team, or care manager; and which action is expected after the alert.

The action matters more than the alert. If the model output does not change an order, a bedside assessment, a phone call, a huddle, a discharge plan, or a diagnostic step, the tool is not yet tied to care delivery. Committees should ask the service line to write the downstream action in plain operational language before a vendor is invited in.

• Population: inclusion and exclusion criteria, care setting, relevant comorbidities, language needs, and known equity concerns.
• User: the person expected to see the output, the backup recipient, and the accountable clinician or manager.
• Trigger: when the model runs, where the output appears, and whether the user can suppress, defer, or override it.
• Action: the specific clinical, operational, or documentation step expected after a positive output.
• Threshold: the minimum sensitivity, specificity, positive predictive value, false-positive burden, or turnaround time needed for the workflow to be tolerable.
• Failure path: what happens after a missed case, wrong recommendation, unavailable model, or user disagreement.

This is also where committees should separate “AI would be interesting here” from “AI is the right intervention.” A service line that cannot describe what happens after the output may need a workflow redesign, staffing change, registry, dashboard, or rules-based alert before it needs a predictive model.

2. Evidence Validation Against the Intended Population

After the use case is defined, evidence review can become concrete. The committee is no longer asking whether the AI is “accurate.” It is asking whether the tool is accurate enough, in the right population, at the right point in the workflow, with an error profile the organization can absorb.

This is where procurement teams should be skeptical of evidence packets built only around retrospective performance. One analysis of FDA-authorized AI devices from 2015 to 2020 found that 97% relied on retrospectively collected data, and fewer than 13% reported age, sex, or race and ethnicity. That does not mean those devices are ineffective. It does mean a hospital cannot assume that authorization, publication, or a high aggregate metric answers whether the tool will perform safely for its own patients.[2]

External validation deserves its own line item. Performance often declines when a model moves across EHR configurations, equipment, documentation patterns, patient demographics, and local practice norms; the procurement brief should anticipate that gap rather than discover it after go-live. The Clinical AI Report summarizes the common external-validation concern as a 10% to 20% AUC drop across sites because of those differences.[2]

For clinical tools with meaningful patient-care consequences, vendor-reported AUC should be treated as a starting point. The committee should request the confusion matrix at the proposed threshold, calibration data, subgroup performance, missing-data behavior, input-variable definitions, exclusion criteria, and evidence of prospective or silent-mode testing when available.

Bias review belongs in the evidence domain, not in a separate ethics appendix that is read after selection. The Optum population health algorithm case showed how a model can appear operationally useful while reproducing inequity when healthcare cost is used as a proxy for illness severity; the algorithm underestimated Black patients’ health needs by 46.5%. A procurement team evaluating clinical AI should require subgroup performance stratification and a plan for investigating disparities before contracting, especially when the model influences triage, care management, access, or resource allocation. For a deeper treatment of this issue, see Algorithmic Bias and Health Equity in Clinical AI.

Workflow evidence is evidence too. Google Health’s diabetic retinopathy system reportedly achieved 96% lab accuracy, but a field study in Indian clinics found that 55% of images were ungradable with portable cameras, creating a practical workflow failure rather than a narrow model-performance problem.[2] For procurement, that kind of finding should change the demo: the vendor should show how the product handles low-quality inputs, delays, retries, staff handoffs, device variation, and patient throughput.

Governance Turns a Purchase Into an Operating Commitment

Once the use case and evidence clear the first review, the packet should move from “Can this work?” to “Can we safely operate this?” The answer depends less on an AI steering committee charter than on named ownership.

Every clinical AI purchase should identify a clinical owner, technical owner, operational owner, and safety escalation owner. For embedded EHR AI, this is especially important because 80% of hospitals using predictive AI reported sourcing it from their EHR developer; procurement cannot assume that native availability equals local accountability.[1] Health systems evaluating EHR-embedded tools may also need a specific governance track for vendor-native functions, as discussed in Governing Epic's AI Ecosystem.

• Clinical owner: approves the intended use, reviews performance, and decides whether clinical practice should change.
• Operational owner: manages staffing, alert routing, training, downtime process, and frontline feedback.
• Technical owner: monitors integrations, data feeds, access controls, version changes, and system availability.
• Safety owner: receives incident reports, coordinates review of harm or near miss events, and has authority to pause use.
• Equity reviewer: reviews subgroup performance, bias audit findings, and mitigation plans when disparities appear.

The monitoring plan should be written before contracting. It should define baseline metrics, live-use metrics, review cadence, drift triggers, retraining or recalibration process, communication responsibilities, and rollback criteria. Model drift is not a theoretical maintenance issue; it is what happens when patient mix, documentation behavior, clinical practice, equipment, or upstream systems change while the model keeps producing confident outputs. For more on operational monitoring, see Model Drift in Deployed Clinical AI.

If the tool is an LLM-based clinical assistant, the governance packet should also address hallucinated content, source traceability, human review, prompt or retrieval changes, and what the user is allowed to copy into the medical record. The risk review should follow the task: a drafting assistant for patient instructions, a summarizer for chart review, and a diagnostic suggestion tool do not carry the same clinical consequences. For LLM-specific failure modes, see Hallucination in Clinical LLMs.

Make the Demo Task-Based

A polished demo proves that the vendor knows how to present the product. It does not prove that a resident can find the alert at 2 a.m., that a nurse manager can staff the follow-up work, or that an analyst can reconcile the model output after an interface change.

The committee should give vendors tasks, not just time. A task-based demo uses local scenarios and asks the vendor to show how the tool behaves inside the proposed workflow. If direct use of local data is not possible during evaluation, the committee can still provide realistic synthetic or de-identified scenarios that reflect the intended care setting.

• Show where the output appears in the EHR or adjacent application.
• Show the first user action, second user action, and escalation path.
• Show how the tool handles missing inputs, conflicting data, duplicate alerts, and dismissed recommendations.
• Show what appears in the chart, what is logged, and what remains outside the legal medical record.
• Show the administrator view for monitoring use, override patterns, performance, and alert volume.
• Show the downgrade path if the integration, model service, or vendor platform is unavailable.

For tools that claim to save clinician time, the demo should count clicks, handoffs, queues, interruptions, and documentation steps. Time saved in one role often reappears as work for another role. The person inheriting that work should be at the table.

Privacy and Security Questions That Cannot Wait for Legal Review

Privacy and security review should not begin after the committee has emotionally selected a tool. The due-diligence questions are basic, but they are often where vague AI proposals become visible: What PHI leaves the health system? Where is it hosted? Who can access it? Is it used for training, tuning, monitoring, support, benchmarking, or product improvement? How long is it retained? Can the health system audit access and deletion?

Censinet’s healthcare AI vendor compliance guidance emphasizes BAA terms, PHI flow mapping, security review, and contractual controls as part of AI vendor due diligence.[3] AHIMA’s vendor questions similarly push committees to ask how systems are trained, validated, monitored, protected, and explained before adoption.[4] Vendor-adjacent checklists are not independent evidence that a product is safe, but their question sets are useful for making hidden data flows explicit.

• Require a complete PHI flow map covering ingestion, processing, storage, logging, support access, backups, and deletion.
• Confirm whether PHI, prompts, outputs, embeddings, metadata, or user feedback are used to train or improve vendor models.
• Review hosting environment, subcontractors, encryption, access controls, audit logs, vulnerability management, and incident notification obligations.
• Tie permitted data use to the contracted service, not to broad product-development rights.
• Require the vendor to disclose material model, infrastructure, or subprocesser changes before they affect production use.

Score Value After Counting Implementation Work

A return-on-investment slide is not a value analysis. The committee should score value only after identifying the implementation work: build, integration, validation, training, change management, monitoring, service-line time, reporting, and ongoing support. If a vendor claims reduced length of stay, avoided adverse events, faster coding, better capture, or less documentation time, the packet should state the baseline, measurement window, denominator, and operational owner.

This is where health systems should be careful not to demand trial-level evidence for every small tool. A low-risk scheduling or inbox-routing application may justify a lighter pilot with productivity and satisfaction metrics. A tool that changes diagnosis, triage, treatment prioritization, or escalation should carry a heavier evidence and monitoring burden.

A Weighted Rubric for Comparing Vendors

A scoring rubric should come after the committee knows what it is scoring. The Public Health AI Handbook’s vendor evaluation toolkit is useful because it treats vendor review as a weighted assessment rather than a general impression exercise.[5] For clinical AI procurement, the weights below keep evidence and workflow at the center while still giving privacy, governance, and implementation enough force to change the decision.

The exact weights can change by risk tier. A revenue-cycle automation tool may place more weight on privacy, integration, and implementation. A clinical deterioration model should keep evidence, workflow, governance, and safety near the top. What should not change is the rule that a weak score in a safety-critical domain cannot be rescued by a strong sales presentation.

Contract Terms Should Preserve the Committee’s Work

A procurement process that asks hard questions but signs a soft contract has not finished the job. The agreement should preserve the assumptions that made approval reasonable: performance, monitoring, permitted use, update controls, audit rights, safety escalation, and exit rights.

Contract language can be specific without pretending that every clinical AI tool has the same threshold. Due-diligence frameworks identify examples such as false-positive-rate guarantees, quarterly bias audits, disparity thresholds, immediate termination rights for safety risks, and BAA terms with full PHI flow mapping.[3][4] The committee should adapt those concepts to the use case rather than copy a universal number into every agreement.

• Performance commitments: define the metric, threshold, population, measurement period, data source, and remedy if performance falls below the agreed level.
• Bias and subgroup audit commitments: require periodic stratified reporting and a mitigation process when disparities exceed the agreed threshold.
• Monitoring obligations: require vendor support for drift monitoring, incident investigation, uptime reporting, and model-change documentation.
• Change controls: require advance notice and approval rights for material model, feature, data-use, hosting, or subprocesser changes.
• Safety termination rights: allow suspension or termination when the tool creates unacceptable patient-safety, privacy, security, or compliance risk.
• Data rights: prohibit secondary use of PHI or derived data unless explicitly approved and documented.

For tools that will be updated after go-live, the contract should say whether a model update requires revalidation, who pays for that work, and whether the health system can remain on the prior version. A vendor’s improvement cycle should not silently change the clinical system the hospital approved.

Red Flags That Should Pause Procurement

A red flag does not always mean the product is unusable. It means the committee should stop moving toward approval until the issue is resolved in evidence, workflow design, governance, or contract language.

• The vendor leads with aggregate AUC but cannot provide threshold-specific false-positive, false-negative, calibration, or subgroup performance data.
• The service line cannot identify the exact user action that follows the model output.
• The evidence comes only from retrospective data, with no external validation, prospective plan, or site-specific silent-mode test for a high-impact clinical use case.
• The demo cannot show how the tool handles missing data, poor-quality inputs, duplicate alerts, overrides, or downtime.
• The vendor will not disclose PHI flows, subprocessors, retention terms, model-update practices, or whether customer data improves the product.
• No internal owner has authority to pause the tool after a safety event.
• The contract contains broad performance claims but no measurable guarantees, monitoring support, audit rights, or termination path.

The best version of this process is not adversarial. It gives serious vendors a clearer target and protects the health system from mistaking enthusiasm for readiness. It also gives clinicians and operational staff a chance to reject a workflow that looks efficient only from the front of a conference room.

No checklist can prove that a clinical AI tool will succeed. It can, however, force the right questions before money, data, workflow, and institutional credibility are committed: What problem are we solving, for whom, with what evidence, under whose supervision, in which workflow, with what protections, and with what enforceable accountability?

References

1. Hospital Trends in the Use, Evaluation, and Governance of Predictive AI, 2023–2024, ONC, September 2025.
2. Clinical AI Procurement Checklist for Hospitals, Clinical AI Report, 2026.
3. AI Vendor Compliance Checklist for Healthcare, Censinet.
4. 15 Smart Questions to Ask Healthcare AI Vendors, AHIMA.
5. Vendor AI Evaluation Toolkit, Public Health AI Handbook, Appendix H.