Cleveland Clinic AI Scribe Pilot Built a Defensible Vendor Selection Method

Ambient AI scribes are starting to sound interchangeable in procurement conversations: listen to the visit, draft the note, reduce typing, give clinicians some time back. Cleveland Clinic’s AI scribe rollout is useful precisely because its pilot made that assumption harder to defend. In a head-to-head evaluation of five vendors, Beth Meese, senior director of digital innovation at Cleveland Clinic, said the outputs were “surprisingly different” and that the tools were “not necessarily a commodity.”[1]

That matters more than the winner’s name. Cleveland Clinic did not simply invite a vendor into one enthusiastic department and extrapolate from a clean demo. It ran what participants called the “Great AI Scribe Bake-Off”: five vendors, three-month rotations, more than 250 physicians, and more than 20 specialty areas.[2] The evaluation looked at cognitive burden, burnout, same-day chart closure, documentation quality, Epic time analytics, provider satisfaction, patient feedback, and revenue cycle implications.[2]

For readers who need a primer on the category itself, the broader mechanics of ambient AI scribes and where accuracy fails are a separate question. This article is about something narrower and more operational: how a health system can make a vendor choice that still looks defensible after implementation, when cardiology, primary care, documentation integrity, Epic workflows, and revenue cycle staff all start reporting what the tool actually does.

The Pilot Was Built to Expose Variation

A single-vendor pilot can answer a limited question: whether a particular group of clinicians will tolerate or like a product under local conditions. Cleveland Clinic’s design asked a harder question: when multiple ambient systems are placed into real clinical workflows, do they fail in the same way?

The answer was no. The bake-off ran across specialty areas instead of treating documentation as one generic transcription problem.[2] That decision sounds obvious only after someone has paid for the complexity. A dermatology note, a cardiology assessment, a behavioral health encounter, and a primary care visit do not put the same pressure on summarization, medication context, assessment structure, orders, or coding support.

The scale is part of why the method is impressive and also why it should not be copied casually. A five-vendor, three-month-per-vendor, 250-physician pilot is not a lightweight procurement exercise. Smaller health systems may not have enough clinical, IT, analytics, compliance, and revenue cycle capacity to reproduce it without exhausting the very people whose judgment they need. The portable lesson is the structure of scrutiny, not the exact footprint.

The Metrics Were Not a Dashboard Ornament

The most useful part of Cleveland Clinic’s approach is not that it collected many measures. Health systems can collect an impressive amount of weak evidence. The useful part is that the measures caught different failure modes.

Cognitive burden matters because a tool can reduce typing while still making the physician feel responsible for constant surveillance and repair. Cleveland Clinic used NASA-TLX to measure cognitive load and Mini-Z for burnout-related assessment in the pilot.[2] Those instruments put some discipline around a familiar post-demo problem: a clinician may say the product is pleasant, but still spend the visit tracking whether the system misunderstood the patient, omitted a key assessment, or structured the note in a way that will be painful to finalize.

Time metrics catch a different problem. Epic time analytics, pajama time, note creation time, and same-day chart closure are not interchangeable. A product can shorten note drafting but fail to improve chart closure if physicians still need to clean up the assessment later. Cleveland Clinic later reported a 49.6% decrease in pajama time, a 32% increase in face time with patients, a 25% reduction in note creation time, and a 7% increase in same-day chart closure among pilot metrics reported for Ambience.[3]

Those numbers are encouraging, but they should not be inflated into permanent productivity claims. They come from a specific deployment context, with voluntary use and early adopters in the mix. If a health system later mandates use, changes templates, expands into different specialties, or shifts documentation expectations, the measured gains may move.

Documentation quality audits are the guardrail against a subtler failure: a scribe that feels fast because it pushes work downstream. A weak note does not disappear after the visit. It lands with the physician who must attest to it, the coder who must interpret it, the CDI team that must query it, and the billing process that absorbs ambiguity. This is why Cleveland Clinic’s early inclusion of revenue cycle leaders was not an administrative courtesy; it was part of the clinical product evaluation.[2]

That is also where EHR integration stops being a technical checkbox. If the AI scribe produces a plausible paragraph but forces awkward copy-paste behavior, misaligns with Epic workflows, or creates extra reconciliation work, the burden has merely changed location. Health systems evaluating EHR-integrated AI scribes should treat integration depth as a clinical and financial issue, not just an IT preference.

Why Specialty Fit Changed the Vendor Decision

Ambience’s winning argument was not simply that it could listen and summarize. Cleveland Clinic leaders emphasized its specialty-adaptive approach, with the company describing the work less like one generic product and more like many specialty-specific products operating under one platform.[2] That is the right mental model for ambient documentation. The visit audio may be the common input, but the expected clinical output changes quickly once specialty, problem type, and workflow are allowed into the room.

The specialty differences were not theoretical. Business Insider reported that adoption varied sharply across areas, including 99% in nephrology and 55% in urology.[4] That kind of spread is not a footnote. It is the signal procurement teams should want to see before they sign an enterprise contract, because it tells leaders where workflow fit, template design, clinician trust, or specialty language may need more work.

The cardiology example is especially useful because it moves past sales language about continuous improvement. During the pilot, cardiologist adoption reportedly rose from 50% to 71% after Ambience retuned the product based on feedback.[5] That does not prove every future specialty problem will be solved. It does show a concrete loop: clinicians produced a failure signal, the vendor changed the product, and adoption moved.

For a procurement team, that responsiveness is evidence of a different kind. It is not the same as a randomized outcome study, and it should not be dressed up as one. But for vendor selection, it answers a question that traditional evidence often misses: when the tool meets local clinical variation, does the vendor have the engineering capacity and operating discipline to respond before clinician trust is gone?

Adoption Numbers Help, but They Do Not End the Argument

Cleveland Clinic reported an 80% adoption rate for Ambience in the pilot, described as two to three times higher than other vendors tested, along with 67% reduced cognitive burden.[3] Those are strong procurement signals because they came out of a comparative process rather than a standalone enthusiasm survey.

They are still adoption and experience signals, not a final verdict on long-term effectiveness. Voluntary pilots tend to attract clinicians willing to experiment. Early rollout periods can include more support, closer vendor attention, and higher novelty than steady-state operations. A mature evaluation would keep watching adoption decay, specialty-level variance, note quality, coding effects, support burden, and total cost of ownership over several years. The available sources do not yet document those longer-term costs or lock-in dynamics.

The distinction also matters for burnout evidence. Studies of AI scribes can be valuable, but vendor-specific evidence should not be casually transferred from one product to another. The 2026 evidence base on AI scribes and burnout is related, but Cleveland Clinic’s Ambience selection is primarily a procurement and implementation case, not proof that every ambient product produces the same burnout effect.

The Rollout Proved Scale, Not Finality

After the selection, Cleveland Clinic announced the rollout of Ambience Healthcare’s AI platform in February 2025.[6] The post-selection numbers moved quickly: more than 4,000 active users within 15 weeks, and Cleveland Clinic later said the platform had documented 1 million encounters by August 2025.[4][6]

Scale is meaningful here because ambient documentation often looks easier in a bounded pilot than in enterprise operations. More users mean more specialties, more edge cases, more training needs, more governance questions, and more chances for local workarounds to emerge. For organizations running Epic at scale, the scribe also becomes part of a broader AI governance problem, not an isolated documentation add-on. That is why the selection process should connect to the same operating model used for governing Epic’s AI ecosystem.

Operational details also matter before scale. Consent, recording retention, patient communication, and clinician override processes are not decorative policies. They shape whether the tool can become ordinary inside the exam room. Cleveland Clinic has discussed verbal opt-in and recording retention in its implementation context, and organizations designing their own programs should settle patient consent for ambient AI scribes before adoption metrics start driving momentum.

A Smaller System Can Borrow the Logic Without Borrowing the Whole Bake-Off

Most health systems do not need, and may not survive, a full five-vendor Cleveland Clinic-style trial. But they can avoid the weakest version of AI scribe procurement: a polished demo, a small friendly pilot, a few clinician quotes, and an enterprise decision that nobody can explain when coding problems or specialty complaints appear six months later.

• Test more than one vendor when possible, even if the comparison is narrower than Cleveland Clinic’s.
• Include specialties with different documentation patterns instead of concentrating only on the most enthusiastic department.
• Measure cognitive burden, documentation quality, time in the EHR, same-day closure, clinician satisfaction, and patient response as separate signals.
• Bring Epic analysts, coding, CDI, and revenue cycle leaders into the evaluation before the preferred vendor is chosen.
• Treat vendor responsiveness to a specific clinical failure as evidence, not as a promise.

The last point is easy to underweight. A vendor that looks slightly weaker in the first week but improves after specialty-specific feedback may be a safer long-term partner than a product that performs well in a narrow demo and then treats every implementation issue as user training. That judgment requires disciplined observation, not charisma.

For teams that need an operating sequence after vendor selection, a general ambient AI scribe implementation guide can help with rollout planning. Cleveland Clinic’s more distinctive contribution is earlier in the cycle: it showed how to make the selection itself observable.

What the Cleveland Clinic AI Scribe Rollout Actually Teaches

The lesson is not “choose Ambience.” It is not even “run a five-vendor bake-off.” The defensible lesson is that ambient AI scribes should be selected under clinical variation, with measures that can detect different kinds of failure and stakeholders who understand what happens after the note is generated.

A health system should be able to say why one vendor fit cardiology better after tuning, why another product created too much editing work, how the preferred tool affected same-day closure, whether revenue cycle staff found the documentation usable, and which specialties still need calibration. If leaders cannot answer those questions, they may still buy a capable product. They just will not have bought it in a way that survives scrutiny.

References

1. Cleveland Clinic's AI Scribe Bake-Off: How Ambience Healthcare Came Out on Top — Healthcare IT Today, February 27, 2025.
2. 3 Takeaways from Cleveland Clinic's AI Scribe Pilot Process and System Selection — AHA Center for Health Innovation, March 11, 2025.
3. Less Typing, More Talking: How Ambient AI Is Reshaping Clinical Workflow at Cleveland Clinic — ConsultQD.
4. Cleveland Clinic Doctors Are Embracing Ambient AI Scribes — Business Insider, June 2026.
5. Cleveland Clinic taps Ambience for ambient AI tech — Fierce Healthcare.
6. Cleveland Clinic Announces Rollout of Ambience Healthcare's AI Platform — Cleveland Clinic Newsroom, February 19, 2025.