Chilean researchers develop a framework to evaluate the robustness of clinical AI models against data shifts

The study, published in the journal BMC Medical Informatics and Decision Making, introduces Clinical-ShiftEval, an open and reproducible methodology for measuring how clinical natural language processing models degrade when conditions in the healthcare environment change, and compares adaptation strategies using real data from the Chilean public health system.

Artificial intelligence models that process clinical text—such as those supporting patient prioritization or automatic diagnosis coding—are usually evaluated under an unrealistic assumption: that data and tasks do not change over time. In clinical practice, however, change is the norm. New diseases, updates to clinical guidelines, institutional restructuring, or modifications to prioritization policies can quickly cause a model that once performed well to lose reliability, as happened during the COVID-19 pandemic.

To address this problem, Fabián Villena, researcher at the Pontificia Universidad Católica de Chile, together with Felipe Bravo-Marquez of the Universidad de Chile and Jocelyn Dunstan, iHEALTH researcher and faculty member at the Pontificia Universidad Católica de Chile, developed Clinical-ShiftEval, an evaluation framework that makes it possible to simulate, in a controlled way, the changes that clinical language models face once deployed in the real world.

"Clinical-ShiftEval is an evaluation framework designed to study a central problem in healthcare artificial intelligence: models can perform very well when evaluated on data similar to the data they were trained on, but their performance can decline when clinical conditions or the available data change," explains Fabián Villena, lead author of the study. "Clinical-ShiftEval was created to simulate these scenarios in a controlled and reproducible way, making it possible to measure how models respond to changes that resemble those occurring in real clinical settings."

Two types of change, one Chilean case study

The framework formalizes two types of change that are common in healthcare: label set incompatibility (LSI), which occurs when new prediction categories appear or existing ones disappear—for example, the creation of a new medical specialty—and task definition evolution (TDE), which occurs when the criteria defining a label change, as happens when disease prioritization policies are updated.

As a case study, the team applied the framework to the Chilean Waiting List Corpus, a dataset of referrals from the public health system written in Spanish. Using these real-world data, they compared three adaptation strategies: continuous retraining of supervised models, in-context learning with large language models (LLMs), and a hybrid method that combines a previously trained supervised model with an LLM acting as a decision agent.

The results show that conventional supervised models are highly sensitive to these changes, with performance drops of up to 82% when prediction categories change and 43% when clinical criteria vary. In-context learning reduced those drops to 35% and 10% respectively, while the hybrid approach limited them to around 8% in both scenarios—all without the need for new labeled data. Continuous retraining, meanwhile, only outperformed the other strategies after incorporating at least 30% of data from the new period.

"The study shows that Clinical-ShiftEval makes it possible to generate controlled and interpretable scenarios of clinical change, in which performance drops in predictive models can be observed and quantified," says Villena. "This is relevant because it allows the robustness of a model to be evaluated more realistically before considering its clinical application."

Jocelyn Dunstan, iHEALTH researcher and co-author of the study, highlights the origin and future potential of this work: "I want to emphasize that Fabián Villena identified an underexplored area during his PhD, because evaluating how natural language models behave in the face of real changes in the healthcare system is key to making these tools trustworthy in clinical practice."

Beyond aggregate metrics, the error analysis revealed clinically significant failure modes, such as incorrect patient referrals when available specialties change, or missed prioritizations when clinical criteria evolve—findings that can compromise both the efficiency and the equity of prioritization decisions.

Regarding next steps, the lead author notes: "From this work, we learned that large language models can be useful for addressing problems associated with changes in clinical data and contexts. The next stage is to translate that learning into practical tools that allow healthcare providers to implement artificial intelligence models in a simpler and safer way."

Clinical-ShiftEval has been released as an open-source Python library, available on GitHub, and the datasets used are also publicly accessible, making it easier for other research teams to apply the framework to new languages, domains, and healthcare contexts.

Reference: Villena, F., Bravo-Marquez, F. & Dunstan, J. "Clinical-ShiftEval: a framework for simulating and evaluating model adaptation in dynamic clinical NLP tasks". BMC Medical Informatics and Decision Making 26, 240 (2026). https://doi.org/10.1186/s12911-026-03538-6

The study was funded by ANID Chile through the Millennium Science Initiative (ICN2021_004 – iHEALTH and ICN17_002 – IMFD), Basal Funds FB210017 – CENIA and CIA250006 – AC3E, National Doctoral Scholarship 21220200, and Fondecyt project 1241825.