Patients checking their symptoms.
Payers answering questions about plan coverage.
Providers prompting patients to complete intake forms via automated reminder.
AI chatbots in healthcare offer up a lot of answers. But, they also beg a lot of questions — namely around accuracy, safety, and reliability. The excitement over chatbots’ potential to improve access to care is tempered by challenges around trust and quality.
Lirio and Vanderbilt: Partnering to Improve Chatbot Accuracy
That’s why the Lirio team is proud to partner with Vanderbilt University Medical Center to improve the quality of Large Language Model (LLM) chatbot systems so they can serve as more trustworthy, transparent and accurate sources of information and guidance for patients, their caregivers, and an increasingly engaged and curious general public.
Lirio’s scientific team, led by Dr. Christopher Symons and Dr. Amy Bucher, joins a team led by Vanderbilt principal investigators Susannah Rose, MSSW, PhD, associate professor of Biomedical Informatics, and Zhijun Yin, PhD, MS, assistant professor of Biomedical Informatics and Computer Science, for the initiative funded by the Advanced Research Projects Agency for Health (ARPA-H). The two-year $7.3 million ARPA-H grant is one of just a few prestigious awards from the new agency within the U.S. Department of Health and Human Services to support transformative research to drive biomedical and health breakthroughs. Other ARPA-H awardees include the Wyss Institute at Harvard University, the University of Pennsylvania, Stanford University, and Columbia University.
The Impact of V-CARES on Healthcare Chatbots
For this initiative — focused on mental health disorders to start – the Lirio team is helping create the Vanderbilt Chatbot Accuracy and Reliability Evaluation System (V-CARES) to detect hallucinations, omissions, and misaligned values from LLM responses in the healthcare domain. In short, the project will help ensure chatbot responses are correct, complete, and consistent with the intentions of the user’s prompts. V-CARES will produce a streamlined, open-source evaluation process and technology that can be used broadly to improve the quality of medical chatbot responses and ensure they provide a well-received user (patient) experience.
“We are excited to partner with this strong team of researchers led by Vanderbilt Medical Center to develop better tools to facilitate the trusted use of large language models in patient interactions. By contributing to the detection of omissions and misaligned values that may only be detectable when considering the expectations and needs of an individual patient, we hope to ensure that not only is the information given by medical chatbots accurate, but that such tools can provide equitable outcomes.”
Lirio’s headquarters in Tennessee lends itself to a natural partnership with Vanderbilt, among the country’s leading universities and medical centers. Technically and culturally, the initiative makes intuitive sense. Lirio’s team of researchers have leveraged their academic training to build our proprietary platform using objective-driven AI approaches that are focused on optimizing health outcomes while selectively leveraging behavioral science techniques that can address a wide array of barriers to better health behaviors. One line of research that the team specializes in relates to the personalization of conversations based on a wide variety of data that can be predictive of a patient’s values and needs (read more here, here and here). The bulk of this line of research has focused on helping to ensure that conversational AI responses align with a person’s behavioral context, potentially including that captured in Lirio’s Large Behavior Model (LBM). Our LBM is a specialized foundation model in the medical and wellness space that distills patterns in data that are most likely to aid in the targeting of behavioral interventions to achieve better health outcomes. These techniques have proven results in influencing positive longitudinal health engagement for patients who needed nudging to address potential health conditions or complications surrounding cancer, diabetes, COVID-19, and RSV. Dr. Bucher and her team have also deep experience supporting behavioral health, insights from which will inform the ARPA-H partnership priorities – namely major depression, and generalized anxiety.
As diagnoses of these disorders remain high – Generalized Anxiety Disorder (GAD), for instance, affects 6.8 million adults alone in the U.S. – it’s imperative that healthcare leaders innovate and partner to impact longitudinal change, including creating trustworthy digital tools like chatbots to support people in their journey to better health.