Evaluating AI in health care

The project’s first phase seeks to develop rigorous, scalable methods to evaluate the robustness of LLMs in clinical settings. 

“You can test a model here or there, but truly scalable evaluation is extremely challenging,” Beheshti explained. “It requires massive computational power, large datasets and robust infrastructure, and even then, it’s not always feasible. Clinicians want to know: Can I trust this tool? Will it actually improve outcomes for my patients?”

To answer these questions, Beheshti’s team will work with physicians and medical students from Nemours Children’s Health and Weill Cornell Medicine to assess scenarios such as predicting a patient’s risk of developing obesity or determining optimal treatment options. Their analyses will compare LLM outputs with established clinical guidelines, meta-analyses and other gold-standard sources to measure accuracy, balance and trustworthiness.

Using retrieval-augmented generation, a method that lets AI check its answers against trusted medical sources, the researchers hope to systematically identify potential biases and errors. Pilot studies will focus on childhood obesity prevention and treatment, but the team expects these automated evaluation approaches to be generalizable across many areas of health care.

Advancing trustworthy AI

The project’s second goal is to develop methods to improve the fairness and reliability of LLMs in medicine. The team will leverage constitutional AI, an approach that guides models to behave in line with agreed-upon rules.

“Historically, models have been refined using human feedback, in which people review and correct the model’s outputs. This process is subject to human bias,” Beheshti explained. “In constitutional AI, trusted experts — in this case, clinicians — define the golden rules, such as ‘do no harm’ or ‘base recommendations on evidence.’ The model then learns to follow those principles as it generates responses.”

The ultimate goal is for the LLM to help identify and correct its own errors based on the established constitution. In the future, Beheshti hopes this AI-guided feedback will replace time-consuming, expensive manual oversight.

Bringing responsible AI to the community

Education is another cornerstone of Beheshti’s CAREER project. Building on his 2021 human-centered AI course at UD, he plans to train engineers, computer scientists and health professionals to think critically about how AI is used in practice. He hopes to engage a wide range of learners, from K-12 students to seasoned health care professionals, to explore the opportunities and challenges posed by modern AI.

Beheshti’s research and outreach efforts promise to help define what responsible AI looks like in health care, ensuring that technology is consistently guided by evidence.

Funding is provided under NSF award number 2443639.