Beyond the AI Hype: What It Takes to Make Artificial Intelligence Work in Healthcare

For healthcare organizations, successful AI adoption starts with clearly defined problems, measurable outcomes and the operational discipline to turn promising technology into reliable results.

Artificial intelligence is rapidly expanding what healthcare organizations can automate, analyze and accomplish. But as interest in AI grows, so does a more practical question for healthcare leaders: How do you move from experimentation to meaningful, measurable impact?

The answer has less to do with choosing the latest model and more to do with understanding the work itself.

During a recent Next-Gen cohort discussion, Leading AI Transformation in Healthcare, CEO at Basalt Health, Ben Hackett, shared lessons from deploying AI within healthcare workflows, including patient record preparation and admissions review. His experience highlighted a central challenge for organizations looking to put AI into practice: The technology may be ready to solve a problem, but the organization must first define that problem, agree on how it should be solved and establish how success will be measured.

For healthcare executives navigating AI investments, several lessons stand out.

Start With the Problem, Not the Technology

When executives tell their organizations they need to adopt AI, the directive can create momentum without necessarily creating clarity. Teams are left to identify opportunities, evaluate tools and determine where to begin, often without a shared definition of what success looks like.

Hackett's recommendation is straightforward: Start with a specific, repeatable process and establish a measurable baseline.

Consider patient chart preparation. Before implementing AI, reviewing a patient record ahead of an appointment could take approximately 12 minutes. Medical assistants had to navigate extensive documentation to identify care gaps, including relevant medical history or screenings that might otherwise go unaddressed.

Basalt Health developed a workflow to review those records, identify potential care gaps and surface relevant information for providers.

The opportunity was concrete. Rather than pursuing an abstract goal of improving efficiency through AI, the organization could measure how long the process took before implementation and track how that performance changed over time.

This distinction matters. An initiative intended to improve efficiency by 10% is difficult to evaluate without a clear starting point, a defined target and an agreed-upon measurement. Without those elements, organizations may invest considerable resources in technology without being able to demonstrate whether it delivered the intended value.

The first question should not be where AI can be used. It should be which problem the organization needs to solve, why it matters and how the result will be measured.

The Biggest Obstacle May Be Organizational Alignment

In healthcare, processes are rarely as standardized as they appear on paper.

A workflow that seems straightforward at the department level may involve different interpretations across clinical teams, facilities and executive leaders. Those differences become especially consequential when an organization attempts to automate decisions or recommendations that depend on clearly defined rules.

Hackett described how implementing AI for admissions review requires health systems to establish explicit clinical criteria. The technology reviews extensive patient records and compares the information against established requirements, flagging cases that need additional clinical review.

But what happens when stakeholders disagree about the criteria themselves?

In one example, an admissions decision prompted a concern that a patient should have been flagged for additional staffing. A closer examination of the record revealed information relevant to the existing criteria that supported the tool's original output. The deeper issue was a disconnect between the criteria provided to the technology and the criteria understood by the clinical team.

The technology had exposed an underlying operational problem: The organization had not fully aligned on the process it expected the system to follow.

This is a recurring challenge in enterprise AI implementation. Automating a workflow requires organizations to make implicit assumptions explicit. Definitions that have historically varied by department, facility or individual must be documented, tested and agreed upon.

That work can be difficult, particularly in large health systems with multiple stakeholders and competing priorities. Yet it is essential to successful implementation.

Healthcare leaders should treat AI adoption as an opportunity to examine the underlying workflow, resolve inconsistencies and establish clear ownership of decisions. In some cases, that means bringing clinical expertise directly into implementation and identifying the individuals responsible for resolving disagreements.

Technology cannot compensate for a process that the organization itself has not defined.

Measure Performance Beyond Efficiency

AI investments are often evaluated through the lens of cost savings or productivity. Those measures matter, but healthcare organizations also need to understand whether the technology is improving the quality and consistency of the work.

Basalt Health's chart preparation example illustrates the value of tracking multiple outcomes.

Following implementation, the average time spent reviewing a patient record declined from approximately 12 minutes to 7 minutes at the initial measurement point, eventually reaching about 1.5 minutes. The organization tracked time through its software, measuring the interval between opening a patient record and completing the review.

The team also monitored user-reported errors and care gap closure rates. Depending on the type of care gap, reported closure rates increased by 40% to 60%, including improvements related to colorectal cancer screenings and mammograms.

These results demonstrate the importance of connecting operational efficiency to meaningful workflow outcomes.

Faster completion alone does not establish success. Organizations must also determine whether the work is being completed correctly, whether quality is maintained and whether the intended clinical or operational outcome is improving.

That requires measurement to be built into the implementation from the beginning. Organizations should define their baseline, establish targets, instrument the workflow to capture performance and monitor results as adoption evolves.

When historical data is unavailable, teams can begin collecting their own baseline measurements rather than relying exclusively on estimates or self-reported assumptions.

The metrics will vary by use case. Some initiatives may focus on time or total cost. Others may require measures of quality, consistency, patient experience or clinical outcomes.

The common denominator is a deliberate approach to evaluation.

If an organization cannot clearly articulate what it is trying to improve and how it will know whether that improvement occurred, it is not yet positioned to demonstrate the value of its AI investment.

Reliability Requires Ongoing Validation, Not a One-Time Test

In healthcare, AI performance cannot be evaluated solely by whether a tool works during a demonstration. Systems must operate reliably within real-world workflows, where records are lengthy, information may be inconsistent and decisions can carry meaningful consequences.

That makes accuracy, transparency and human oversight central to implementation.

In Basalt Healths's admissions workflow, the technology identifies relevant evidence and links it to the specific portions of a patient record where that information appears. Clinical teams can review the supporting documentation, validate the extracted information and retain responsibility for the ultimate admission decision.

This approach helps make the system's output more transparent. Rather than asking users to accept a recommendation without context, it allows them to examine the evidence behind it.

Reliability also requires systematic testing. During the discussion, Hackett described using validated examples, or “golden datasets,” to evaluate model performance. These datasets contain cases for which the expected answers have been established and checked by people.

When a model changes or a new model is introduced, the team can run known cases through the system to determine whether it continues to produce the expected results. Historical errors can also be retained as test cases to identify regressions and ensure that previously resolved problems do not return.

This becomes especially important when systems rely on multiple AI models to support continuous operations. If one model becomes unavailable or reaches a usage limit, an alternative may be needed. That transition must be tested to ensure that switching models does not compromise quality.

The lesson for healthcare leaders is that AI validation is an ongoing operational responsibility, not a box to check before launch.

Organizations need defined review processes, clear escalation paths, documented performance standards and a reliable way to investigate errors. They must also guard against the opposite of distrust: users becoming so comfortable with an automated workflow that they stop meaningfully reviewing its outputs.

The goal is not simply to build confidence in AI. It is to build justified confidence grounded in evidence.

Human Judgment Remains Essential

As AI systems become more capable, healthcare organizations will face increasingly important decisions about which tasks to automate, which to augment and where human review must remain part of the process.

The discussion surfaced a fundamental tension. Technology may become capable of completing work that previously required significant human effort, but technical capability does not automatically determine the appropriate operational or workforce decision.

In clinical workflows, the distinction between surfacing information and making the final decision is particularly important. AI can help identify relevant evidence, apply documented criteria and direct attention to cases requiring additional review. Clinical professionals remain responsible for evaluating that information and making decisions within the established workflow.

Human oversight, however, must be designed into the process. It cannot depend on an informal expectation that someone will catch a problem after the fact.

That means defining when review is required, making supporting evidence accessible, giving users a way to flag concerns and ensuring feedback reaches the teams responsible for improving the system.

It also means recognizing that automation can change the nature of work, not just the time required to complete it. As repetitive tasks become more efficient, leaders must decide how to deploy the capacity created and where human expertise delivers the greatest value.

These decisions extend beyond technology and into organizational strategy, workforce planning and clinical governance.

and clinical governance.

The Path to Scale Begins With Disciplined Execution

For healthcare organizations, scaling AI involves more than replicating a successful pilot across additional facilities or departments.

Workflows may differ by location. Clinical criteria may vary. Regulatory requirements and internal governance processes introduce additional complexity. Systems must also accommodate changing models without allowing performance to deteriorate.

Successful implementation therefore depends on establishing a repeatable approach: document the workflow, align stakeholders, define measurable outcomes, validate performance and build a process for continuous monitoring and improvement.

It also requires recognizing that the most promising AI use cases are not necessarily the most technologically ambitious. A focused application that solves a well-defined problem, integrates into daily operations and produces measurable results may offer a more credible starting point than a sweeping transformation initiative with unclear objectives.

For executives, the challenge is to create the conditions in which those practical applications can succeed. That includes assigning ownership, bringing the right clinical and operational leaders into the process, investing in measurement and ensuring that teams have the resources to sustain the work after implementation.

AI's potential in healthcare is significant. Realizing it requires a willingness to move beyond the excitement surrounding the technology and engage with the harder questions of execution.

The organizations best positioned to capture its value will be those that pair technical capability with operational clarity, rigorous measurement and human judgment. In healthcare, where reliability and quality matter as much as speed, that discipline is not separate from innovation. It is what makes innovation useful.

Creating Space for Important Conversations | About the Nashville Health Care Council

The Nashville Health Care Council serves as common ground for the vibrant healthcare community, bringing together leaders from across the healthcare ecosystem to exchange ideas, share perspectives and explore the complex issues facing the industry. Through curated convenings, educational programs and meaningful connections, the Council creates space for candid dialogue, diverse perspectives and productive conversations that might not otherwise happen. As a neutral convener, the Council does not advocate for a particular position or seek consensus. Instead, we bring the right people to the table, foster thoughtful discussion and extend important insights to the broader healthcare community, helping move conversations and opportunities forward while strengthening and elevating Nashville as The Healthcare City.