This is Clinical Product Thinking 🧠, a weekly newsletter featuring practical tips, frameworks and strategies from the front line of clinical product.
Welcome, friends. This is issue No. 60 of Clinical Product Thinking. This week, we’re talking about why the same AI model can perform very differently from one hospital, patient group or even season to another.
A CE or UKCA mark tells you that a medical device has been assessed as conforming to the applicable regulatory requirements. It doesn’t guarantee that an AI model will perform equally well across every hospital, patient population or clinical setting.
Your patients might be different. Your scanners might be different. Even the time of year might affect the results. More on this later.
I was reminded of this at NHS England’s Digital Safety in Practice day last month. I’ve already written about Prof Mark Sujan’s talk on using LLMs for assurance. Another excellent session, delivered by Haris Shuaib of Newton’s Tree, has stayed with me.
Haris walked through examples of what happens when clinical AI systems encounter the complexity of real-world deployment and it’s fascinating.
Despite the growing number of approved AI medical devices, clinical adoption remains relatively limited. Haris highlighted a number of factors, including insufficient evidence, high implementation costs, institutional barriers and a lack of effective performance monitoring.
The examples below help explain why that last problem is particularly gnarly one to solve.
1. Same model, different hospital
A well studied model can perform differently (and significantly worse) in a different clinical setting and care providers should be cautious about relying on published performance figures or a vendor’s existing customer base.
Haris described that at a multi-site evaluation across London, Oxford, Northumbria and Glasgow, a model for detecting intracranial haemorrhage was tested retrospectively.
At most sites, its reported accuracy was around 90%. At Glasgow, it was just 70%. (The published literature had reported 95%).
Glasgow subsequently withdrew from the prospective arm on safety grounds. That particular vendor had more than 1,000 customers worldwide.
Another example is a German oncology centre who ran validation studies before purchasing AI for chest X-ray interpretation. They tested three leading vendors against the same 1,000 historical cases and found a 30-percentage-point difference in ability to distinguish normal from abnormal films.
The weakest performed at just around 50% accuracy!
The better approach is to perform local testing to help establish whether published results generalise to your population, equipment and clinical workflows. Prospective evaluation and ongoing monitoring are then needed to understand how the model behaves in practice and over time.
2. Same model, different patient
Even within a population that appears relatively uniform, performance can vary a lot.
One paediatric fracture detection model showed a reported 23% deterioration in performance across age groups.
It performed best on 2-year-olds and worst on 17-year-olds, where accuracy fell to around 70%.
That surprised the room, because teenage skeletons are more similar to adult skeletons. You might intuitively expect a model to perform better on older children.
But that wasn’t what the results showed.
It’s a useful reminder of why subgroup analysis is so important. Overall performance figures can conceal substantial differences between patient groups, including differences that aren’t immediately obvious from clinical reasoning alone.
For anyone evaluating clinical AI, it isn’t enough to ask how accurate a model is across the whole dataset.
You also need to understand who it performs well for, who it performs poorly for, and whether those differences create unacceptable clinical risks.
3. Same model, different image
The data pipeline feeding a model is part of the clinical system, even when it sits outside the AI product itself.
Some of the most interesting failures had surprisingly little to do with changes to the AI model itself.
At one site, chest X-rays were rotated to make them easier for radiologists to read. Unfortunately, the rotation created diamond-shaped crop marks around the edges of the images.
The AI interpreted these artefacts as urgent findings.
An investigation found that roughly half of the false positives were associated with data quality problems of this kind.
Once the cause was identified, the solution was relatively straightforward: send the original, unprocessed image to the AI and the processed image to the radiologist.
Another example involved a mammography system.
In a 2023 study published in Radiology: Artificial Intelligence, researchers evaluated a breast screening AI algorithm across more than 55,000 screening examinations.
They discovered that the AI’s recall rate increased approximately threefold following a software upgrade to the mammography equipment.
The AI model itself hadn’t changed. The scanner software had.
The researchers suggested that the upgrade may have altered characteristics of the images that the AI relied on, even though human reader performance was unaffected.
The connection was identified by examining image metadata and tracing the change back to the equipment software.
For anyone building or deploying clinical AI, it’s a useful reminder that changes to scanners, firmware, image processing, file formats or integration pathways can affect what the model receives and, consequently, what it produces.
4. Same model, different season…!
A safety assessment carried out at a single point in the year may miss important changes in how a clinical AI system performs over time.
This is the example that made people sit up in their chairs. Haris shared monitoring data from a deployment at Leeds. During winter pressures, a 30-day rolling measure of agreement between the AI and clinical labels rose to around 90%. After New Year, it fell to around 25%, before recovering.
The model itself hadn’t changed.
Haris offered an interesting explanation for the swing.
During winter pressures, patients who might otherwise have been seen in primary care were being directed to A&E, sometimes receiving chest X-rays that showed no abnormalities. This increased the proportion of normal cases, which Haris suggested were easier for the AI to classify correctly.
After the winter-pressure period, the case mix changed. Patients attending hospital tended to have more significant disease, while lighter clinical workloads potentially gave clinicians more time to review and challenge AI recommendations.
The result was a substantial change in observed agreement, despite the AI model remaining unchanged.
Importantly, concordance isn’t the same as diagnostic accuracy. The data doesn’t establish exactly how much of the variation was attributable to changes in case mix, model performance or clinician behaviour.
But it illustrates how the conditions surrounding a clinical AI system can change over time, sometimes considerably.
For buyers and builders, this raises a practical question: how will you know whether a model continues to perform appropriately as patient populations, disease prevalence and clinical workflows change?
What buyers and builders can do
The recommendations from Haris’s session were pretty clear.
Test on your own data before signing a contract.
Haris described pre-contract evaluation using local data as one of the most valuable steps a buyer can take.
It allows hospitals to compare vendors against the patients and conditions in which the technology will actually be used.
Where possible, that evaluation should include clinically relevant subgroups, equipment configurations and representative cases.
Awarding a contract first and discovering performance problems afterwards leaves the care providers with fewer options.
A practical checklist for clinical AI
If you’re buying, building or deploying a clinical AI system, here are six questions worth asking:
Has the model been evaluated on data representative of our own patients and clinical setting?
Do we understand how performance varies across relevant patient subgroups?
Have acceptable performance thresholds and responsibilities been agreed with the vendor?
Can we identify changes to equipment, data processing or integrations that might affect the model?
Are we monitoring performance over time, including changes in patient populations and clinical workflows?
Is there a clear process for investigating deterioration and suspending use when necessary?
None of these questions is particularly complicated. But answering them requires cooperation between clinical teams, product teams, technical teams, procurement and the manufacturer.
The takeaway
Clinical AI performance depends on far more than the model itself.
The hospital, patient population, equipment, data processing and clinical workflow can all influence how a system behaves. Those conditions can also change after deployment.
Regulatory conformity assessment and manufacturers’ post-market surveillance obligations provide important safeguards. But hospitals still need to establish whether a product is appropriate for their own setting and monitor how it performs in use.
If you’re buying clinical AI, ask what evidence exists that it performs safely and effectively on your patients, using your equipment and within your workflows.
If you’re building it, ask how you’ll know when those conditions change.
And if you’re responsible for clinical safety, consider whether your assurance activities address the system as it is actually deployed, including the people, processes and technology surrounding the model.
Thanks to Haris Shuaib of Newton’s Tree for the fascinating talk!
That’s all for this week. See you next time! 👋
🤝 Work with me | 📅 Attend an event | ✍️ Send a message
Written by Dr Louise Rix, Clinical Product, AI, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn & Instagram.
Made with 💜 for better, safer HealthTech.