The most useful evidence on conversational AI in healthcare does not ask whether a model can produce medical-sounding text. It asks whether a patient leaves an encounter with clearer next steps, whether a clinician can still catch a bad message before it lands, and whether the system behaves differently when a user is alone with an open text box.
That distinction matters because the recent evidence points in two directions at once. In a large randomized trial of a physician-supervised LLM agent in a real healthcare setting, patients reported slightly higher clarity and satisfaction, physicians rated most conversations favorably, and only 3 of 1,265 AI-written messages were hidden before reaching patients.[1] In independent stress-test and human-factors studies, however, roughly half of chatbot health answers contained problematic content, and ordinary users obtained correct answers less than 35% of the time even though the chatbots themselves retrieved correct medical answers about 95% of the time.[2]
Those are not minor variations around the same result. They are evidence that conversational AI performs as part of a workflow, not as an isolated capability. Supervision, question format, patient interpretation, and review standards change what the same broad technology becomes in practice.
The supervised RCT is the strongest positive signal
The Alan/Mo randomized controlled trial deserves attention because it studied a conversational agent inside an operating healthcare service, not as a benchmark exercise. The trial included 926 participants in the Alan health-insurance ecosystem in France and tested a physician-supervised LLM conversational agent for patient communication.[1]
Patients who responded showed a high willingness to use the tool: the opt-in rate was 81%.[1] That number should not be mistaken for universal acceptance, because it applies to patients who responded in that setting. Still, for a supervised clinical communication tool, it is a meaningful operational signal. A system that patients do not choose to use cannot improve post-encounter understanding, no matter how strong its model performance looks elsewhere.
The patient-reported differences were modest but directionally important. Clarity was higher in the conversational AI group, at 3.73 versus 3.62 on a 4-point scale, and satisfaction was higher at 4.58 versus 4.42 on a 5-point scale; both differences were reported as statistically significant at p<0.05.[1] These are not dramatic clinical outcome changes. They are the kind of small improvements that matter when the endpoint is whether a patient understands what just happened and what to do next.
The safety process is just as important as the satisfaction result. Physicians reviewed the conversations, rated 95% as good or excellent, and deemed no conversation dangerous.[1] Across 1,265 messages, physicians hid 3 before they reached patients.[1] A promotional reading would focus on how few messages were suppressed. A clinical operations reading should notice the opposite: the trial had a mechanism for suppression, and the mechanism was actually used.
That review layer changes the interpretation of the AI output. The result is not evidence that patients can safely rely on an LLM alone for medical advice. It is evidence that a physician-supervised conversational agent can improve communication measures in a defined workflow where clinicians remain responsible for review and intervention.
| Measure | Finding | Why it matters |
|---|---|---|
| Study setting | Randomized trial in the Alan health-insurance ecosystem in France; n=926 | More clinically relevant than a standalone model benchmark, but still limited to one ecosystem |
| Patient uptake | 81% opt-in among patients who responded | Suggests patients were willing to engage with supervised conversational support |
| Patient clarity | 3.73 vs 3.62 on a 4-point scale, p<0.05 | Measures whether communication felt clearer to patients |
| Patient satisfaction | 4.58 vs 4.42 on a 5-point scale, p<0.05 | Shows a modest but statistically significant improvement in experience |
| Physician review | 95% of conversations rated good or excellent; no conversation deemed dangerous | Shows clinician judgment was built into the evaluation |
| Message suppression | 3 of 1,265 messages hidden by physicians | Shows the safety pathway was not theoretical |
The trial also needs its boundaries kept visible. It is an arXiv preprint, not a final word from settled peer-reviewed literature. It was conducted in one French insurance ecosystem, so it may not generalize cleanly to U.S. care delivery, different payer incentives, different clinical liability structures, or more diverse patient populations.[1] The right response is neither dismissal nor overextension. It is to treat the study as credible early evidence for one deployment pattern: supervised conversational AI used to improve patient communication.
The same technology looks weaker when users have to steer it alone
The counterweight comes from studies that did not wrap the chatbot in the same clinical controls. A BMJ Open stress-test study, reported by The Conversation, tested five major chatbots with adversarial health prompts. About 50% of answers contained problematic content, and about 20% were highly problematic.[2]
The prompt format mattered. Open-ended questions produced a much higher share of highly problematic answers than closed questions: about 32% versus 7%.[2] That is a practical finding, not just a model critique. Patients often do not arrive with a clean multiple-choice question. They describe symptoms, fears, partial histories, medication names they may not spell correctly, and constraints they may not realize are clinically relevant.
The study also found that no chatbot produced a single fully accurate reference list across 25 attempts.[2] In a casual search setting, a weak reference list may look like a documentation flaw. In healthcare, it affects whether a clinician, patient, or reviewer can trace a recommendation back to a trustworthy source. A fluent answer without a reliable trail is difficult to audit.
The caveats are real. The BMJ Open study used adversarial red-team prompts designed to expose failure modes, so its error rates should not be treated as normal-use prevalence. It tested free model versions available in February 2025, and newer systems may behave differently.[2] But those caveats do not erase the safety signal. Healthcare systems need to know not only how a tool performs under ideal prompting, but how it behaves when questions are ambiguous, poorly constrained, emotionally loaded, or intentionally challenging.
The more unsettling evidence is the human-factors gap reported from a Nature Medicine study. The chatbots retrieved correct medical answers about 95% of the time, but real users interacting with them obtained correct answers less than 35% of the time, which was no better than people who did not use AI at all.[2]
That result should change how healthcare teams talk about accuracy. If the model contains or retrieves the right answer, but the user cannot reliably extract it, the practical system has not performed accurately. The failure may occur in the prompt, the follow-up question, the interpretation of caveats, or the user’s decision about which part of the answer to trust. The consequence still lands in the clinic, pharmacy, urgent care line, or portal inbox.
Retrieval accuracy is not the same as clinical communication
A model benchmark can make conversational AI look close to ready. A patient interaction can make the same system look unfinished. The difference is not mysterious. Clinical communication has more steps than answer generation.
- The patient has to ask a question that contains enough clinically relevant context.
- The system has to identify what is being asked and what risks cannot be ruled out.
- The answer has to separate general information from actionable advice.
- The user has to understand uncertainty, urgency, and when to seek care.
- A clinician or organization has to be able to review what happened after the fact.
The Alan/Mo workflow addressed several of these steps by keeping physicians in the loop. The stress-test and user studies exposed what happens when those supports are weaker or absent. This is why the evidence should not be collapsed into a single claim that conversational AI is either safe or unsafe in healthcare. The better question is which parts of the communication chain are controlled.
For clinical teams, the most relevant failure mode may not be a spectacular hallucination. It may be a plausible answer that leaves the patient with the wrong level of confidence. A patient may remember reassurance and miss a caveat. Another may accept a general statement as a personal recommendation. A third may stop searching after the first answer because the chatbot sounded complete. None of those failures require the model to be ignorant. They require the system to be deployed as if retrieval were enough.
The broader review literature is still fragmented
A 2025 systematic umbrella review indexed in PubMed examined 44 reviews of AI-based conversational agents across areas including clinical decision support, mental health, education, addiction support, and diet or nutrition-related domains.[3] Its conclusion was not that the field lacks promise. It was that effectiveness evidence remains fragmented and that transparency and standardization are still needed.[3]
That finding fits the pattern in the primary studies. Different studies measure different endpoints: patient satisfaction, clarity, physician ratings, answer quality, reference accuracy, user success, or adverse content. Those are not interchangeable. A system can improve satisfaction without proving clinical outcome benefit. A model can retrieve correct information without helping a user reach a correct conclusion. A supervised workflow can look safe without proving that unsupervised consumer use is safe.
This is also why vendor-reported operational claims should stay in the background. A case study reporting wait-time reduction or deflection may be useful for generating operational questions, but it is not equivalent to a randomized trial, an independent stress test, or a systematic review. Reduced waiting is attractive only if the handoff remains safe, the unresolved cases are visible, and the organization can show what happened to patients after the chatbot interaction.
What the evidence supports now
The defensible conclusion is narrow but useful. Physician-supervised conversational AI has credible early evidence for improving patient communication in at least one real-world randomized workflow. The Alan/Mo trial’s clarity, satisfaction, physician-review, and message-suppression findings are more clinically relevant than a leaderboard score because they show how the tool behaved around actual patients and clinicians.[1]
The same evidence does not justify treating unsupervised chatbot use as reliable clinical performance. Independent stress testing found problematic content at rates that would be unacceptable if translated directly into patient advice, especially for open-ended questions.[2] Human-factors evidence shows that even when chatbots can retrieve correct answers, users may fail to obtain correct answers from the interaction.[2]
For healthcare organizations evaluating conversational AI, the most important questions are therefore operational rather than futuristic. Who reviews the message before the patient acts on it? Which messages can be hidden, escalated, or corrected? What is logged? How are open-ended questions constrained? How does the organization measure whether the patient understood the answer, not merely whether the answer was generated?
Conversational AI in healthcare is not one evidence category. It is a family of tools whose performance changes when supervision, user behavior, and evaluation standards change. The strongest current evidence favors supervised communication support. It does not support handing patients a general-purpose medical chatbot and calling the underlying model’s accuracy a patient-safety result.
Comments
Join the discussion with an anonymous comment.