Doubling Down on Digital Quality and Evaluating the Risks of AI
October 15, 2025 · NCQA Communications
More Health Innovation Summit highlights from San Diego! We’ve got the key takeaways from two powerful keynote sessions on trending topics in health care: Digital quality measurement and the use of AI.
The Road To Digital Quality
Moderator David C. Kendrick of the OU-TU School of Community Medicine led a panel discussion of digital quality innovators—Bharat Sutariya of Oracle Health, Abdul Shaikh of Amazon Web Services and Anna Taylor of MultiCare Connected Care—who are mapping a course to deliver real value for patients.
Kendrick started by defining the problem: Data fragmentation. “A patient’s clinical data is scattered across six or more locations—the more complex a patient’s care is, the more places that data resides,” says Kendrick. “This fragmentation exists not just within communities, but across state and national boundaries. So that’s a real risk for everything we’re trying to accomplish in quality improvement.”
The move to digital quality measurement and interoperability can help to alleviate the problem of data fragmentation.
Building the Business Case for Digital Quality
Health care operates on a thin margin, so we need to optimize care delivery, patient experience and the cost of care. The return on investment for digital quality comes from investing in the data foundation. Moving to interoperable standards allows data to flow across organizations where it can be aggregated to create a more complete picture—and that applies to more than just quality measurement.
“If you can measure quality in your organization, I guarantee you are producing much higher insight operationally, financially and clinically,” says Sutariya. “When you have the infrastructure that supports quality, you can leverage that infrastructure for any business problem you’re trying to solve.”
Some organizations are already experiencing real financial impact from digital quality measures. “MultiCare took a really small use case, 30-day medication reconciliation, and started trading data with a payer. So now the payer knows how many gaps we’ve closed on a daily basis,” says Taylor. “It cost me about $15,000 to code and develop that application, and we earned about $17,000 for closing the care gaps. More importantly, we were able to contact the right patients to follow up on their care.”
Moving From Retrospective Measurement to Real-Time Insights
Digital measurement needs to go beyond the EHR—otherwise all we’ve accomplished is moving the data from a paper chart into an electronic one. The opportunity lies not in digitizing the quality metric, but integrating it upstream into the practice of medicine so practitioners know how they are doing in real time, as they are delivering care.
“This also ties in with personalized medicine,” says Shaikh. “The idea that you can have a real-time understanding of the patient with all the different types of data points and that it is accurate enough to drive a whole set of important outcomes.”
Assessing Data Quality in a Digital World
As we move toward more interoperable data formats and transmitting data through APIs, there are questions about data quality and integrity. As organizations map data to the FHIR® standard, they need quality checks on the mapping. For digital quality measures, organizations have the advantage of being able to compare results from traditional measures with digital ones.
“MultiCare reported our quality measures for Medicare shared savings this year using only FHIR data,” says Taylor. “We reported about 15,000 lives and I got to a standard deviation of 5% on my first try and I didn’t clean anything. So, the data quality and the mappings are pretty solid.”
Building Confidence in AI Through Standards
AI is transforming health care, but safe, responsible and reliable use requires more than good intentions, it demands standardized approaches that build trust. Vik Wadhwani, NCQA’s Chief Product and Transformation Officer, led a discussion with industry experts—Aaron Neinstein of Notable and Maia Hightower of Veritas Healthcare Insights—about how to build a trust layer across innovation, measurement and outcomes.
Defining Risk in AI Use
The perception of risk varies based on the stakeholder. Patients risk being hurt or having a poor outcome as a result of AI use. Clinicians may be concerned about the overall quality of care or the risk of malpractice if they follow an AI-recommended treatment. Many professionals in the health care system are also concerned about job loss. “What’s missing from the conversation, is the concern around automation bias,” says Hightower. “There is emerging evidence of overreliance on the recommendations of an AI system—and our youngest clinicians are at the greatest risk for this.”
How to Assess AI Tools
We should evaluate AI tools against existing health care system performance, rather than perfect outcomes. It’s easy to look at AI risk in a vacuum, but we have to compare it to how things are functioning today—to human performance. “Let’s be careful not to compare outcomes from AI to the health care system we wish we had, but to the health care system we actually have,” says Neinstein.
Build AI Governance Models Based on Risk Tolerance
Organizations have different levels of risk tolerance. Instead of trying to boil the ocean, be realistic and pragmatic about your current stage of AI governance. “If you’re a rock climber and you’ve got great harness, rope and helmet, you can climb high, far and fast. Think of that as your AI governance,” says Hightower. “But if you haven’t invested in that equipment, should you really be climbing way up to the cutting edge?”
Start With Low-Hanging Fruit
Rather than pursuing high-risk diagnostic AI, organizations should prioritize automating low-risk administrative tasks. “Is our biggest problem that we can’t diagnose cancer? No, the problem is that we can’t get the prior authorization or referral for a patient with cancer because the faxes are still sitting on a fax machine,” says Neinstein. “My advice is to go after the things that are high value and low risk, because there are a lot of them out there.”
Learn More
The Health Innovation Summit wraps up today. We’ll share more key takeaways on the NCQA blog.
