Healthcare AI Needs to Be More Than Just a Healthcare-Trained LLM
By Michael F. Stiefel, MD, PhD, Cerebrovascular & Endovascular Neurosurgeon | Aneurysm, AVM, Stroke, & Brain Tumor Specialist | System Medical Director – Piedmont Stroke Program
AI is entering healthcare faster than any prior digital tool, and the outputs sound more impressive by the month, but fluency and probability are not the same as trust and safety. In medicine, decisions must be correct, complete, and safe and when the data are incomplete, the right move is often to stop, verify, and escalate. The future of healthcare AI will be determined less by raw capability and more by whether we build a trust stack: provenance, constraints, verification, and governance.
You can hear the difference in a single word. Imagine being a patient and hearing your doctor say, “This is probably cancer,” or “This is probably what we should do for your stroke.” No patient feels protected by that. Not because clinicians never face uncertainty, we do every day, but because in medicine, uncertainty is never where we stop; it’s where we start. It’s where we tighten the work: confirm what can be confirmed, rule out what cannot be missed, and move from suspicion to facts. That’s why the central question for healthcare AI isn’t whether it can generate answers, it’s whether it can reliably support clinical decisions: bounded, accountable, and auditable.
Where AI has moved the needle
AI is already being used in healthcare, especially in areas like imaging, where it can support faster triage and more consistent workflows. But one of the most important lessons from real clinical use is not about model architecture; it’s about how outputs are framed. In stroke imaging, for example, we still see labels like “core infarct” used in ways that imply certainty when the output is, at best, an estimate of risk. That kind of wording can quietly steer decisions.
Healthcare AI must be designed so that ambiguity increases rigor rather than decreases it, and that the language used to communicate with clinicians improves quality rather than reducing it through false certainty.
The destination is not an AI that sounds like a clinician. It is an AI that behaves like medicine demands: structured, bounded, auditable, and accountable. In healthcare, “probably” isn’t a treatment plan.
Why probabilistic intelligence isn’t enough
Many of today’s most visible AI tools, especially conversational systems, are probabilistic by design. They infer likely outputs based on patterns in data and language. The problem is that probabilistic systems can be wrong in exactly the wrong way: confidently, smoothly, and without signaling that something essential is missing.
Clinical decision-making is often threshold-driven and constraint-based. Guidelines and pathways define what is recommended, what is contraindicated, and what must be considered before acting. Medications have hard stops, not soft preferences. The most dangerous errors are subtle omissions: an unrecognized contraindication, an assumed time window, or missing data that should have stopped the system from recommending anything at all.
Even when the end user is a physician, probabilistic output can be dangerous, especially outside one’s specialty. A plausible recommendation in an unfamiliar domain can short-circuit safeguards clinicians rely on, including awareness of edge-case contraindications and the instinct to pause when something doesn’t fit. That is why quality control cannot be a disclaimer placed on the user. It has to be engineered upstream.
The real-world adoption challenges
The barrier to safe adoption is rarely enthusiasm. It is implementation. Whether AI is deployed inside the EHR, alongside it, or as a separate layer, it must integrate into real clinical operations: how teams communicate, how orders are placed, how imaging is reviewed, and how responsibility is assigned. If it adds friction, it will be bypassed. If it overwhelms teams with low-value outputs, it will be ignored. Additionally, it becomes challenging to assess, enhance, and trust if its recommendations cannot be linked to source data and standards, particularly when results fall short of expectations.
These issues are amplified by domain shift. Models trained in one setting often behave differently in another because protocols, scanners, populations, and documentation patterns vary. A tool that performs well in a controlled pilot can drift as practice evolves, new sites come online, or data pipelines change. That is why healthcare AI requires clinical governance.
Sometimes we move too quickly. This is a marathon, not a sprint. Smooth is swift, and slow is smooth. The goal is not to be first to deploy. The goal is to deploy safely, repeatably, and at scale because the failure mode isn’t inconvenience. It’s patient harm and a loss of trust that sets adoption back for years.
That pace has to be driven by quality. Quality is what drives outcomes, drives trust, and increasingly drives reimbursement. And quality in healthcare is measurable rigor: adherence to evidence, protocol fidelity, constraint checking, and performance monitoring that holds up in the real world. If an AI system cannot demonstrate that it improves quality without introducing new failure modes, then its fluency and speed are irrelevant.
What trust-grade AI looks like
Four non-negotiables are necessary for trust-grade AI: provenance (what data and whose policy or guideline it utilized), restrictions (contraindications and thresholds it will not cross), verification of crucial inputs (and the ability to halt when they are lacking), and ongoing drift monitoring. Just as importantly, it must communicate uncertainty honestly and prompt the clinician toward the next decisive step rather than filling gaps with a confident narrative.
The destination is not an AI that sounds like a clinician. It is an AI that behaves like medicine demands: structured, bounded, auditable, and accountable. In healthcare, “probably” isn’t a treatment plan.