Where Clinical AI Fails When the Model Looks Fine
Highlights from our Clinical AI panel, on right answers, wrong actions, and the behavioural risks no one's monitoring.
This is Clinical Product Thinking š§ , a weekly newsletter featuring practical tips, frameworks and strategies from the front line of clinical product.
Welcome, friends, this is issue No. 046 of Clinical Product Thinking. This week: highlights from our Clinical AI panel, on where patient-facing AI fails when the model looks fine.
A few weeks ago I co-hosted a panel on building safe clinical AI systems with Dani.
If you only have a few minutes, hereās your tl;dr. If you want to dive deeper, the full recording is below.
On the panel:
Dani Brightman ā Clinical Director, Numan
Dr Lucinda Scharff ā Product Manager, Counsel Health (formerly Google Health / DeepMind)
Dr Paul Sacher ā Founder & CEO, Sacher AI
Letās dive in.
A bot responded correctly but still failed clinically.
Dani gave an example from Numanās customer care side. They built a bot to filter inbound patient requests and signpost people to the right clinical form. On paper, it worked well.
In practice, it missed what a human would have caught:
āEven though the bot was handling the inbound request correctly and providing the right signposting, the clinical team noticed that if the patient was also providing context, like they want to stop medication because āIām having X side effectā, what should have happened is they escalate that ticket to a clinician to review it. The bot missed this important stepā
ā Dani Brightman, Numan
The bot completed the task but missed the clinical significance of the patientās context.
A clinically correct model is not the same as better care
Dr Lucinda talked through a recent paper on using AI to detect cancer on chest X-rays. AI can flag lung cancer risk on a film, but the patient still needs someone to review the result, contact them, arrange the CT and provide follow-up care.
Faster diagnosis on an X-ray doesnāt automatically result in better patient outcomes if downstream dependencies havenāt been accounted for.
A useful signal with no reliable route to action means the product isnāt finished.
Behavioural harms are the hidden risk
Behavioural risk can create or amplify clinical risk, and theyāre much harder to spot: sycophancy, erosion of patient agency and failure to escalate when a patient says theyāre anxious, low or questioning whether to continue treatment.
Systems designed for engagement may keep chatting when the safe move is to refer the patient to a clinician.
āIf you think about the scale of behavioural interventions that are happening through patient-facing AI, you can think of the whole thing as one massive behavioural interventionā¦ that no oneās really monitoring.ā
ā Dr Paul Sacher, Sacher AI
Accuracy isnāt enough if the interaction nudges the patient towards an unsafe next step.
Safety is a team sport, not a clinician sign-off
A key point of the evening was about culture. Clinicians are essential to clinical AI teams, but safety canāt belong to them alone.
If the clinician is the person who turns up at the end to say yes or no, you get slow teams, frustrated clinicians, and products that treat clinical risk as a final check rather than a design concern.
The best teams build shared safety instincts early, in the model, the interface, the pathway, the escalation route, and the assumptions about how patients actually behave.
The takeaways
The eveningās themes:
Correct routing isnāt enough. The system must also recognise when additional context changes the appropriate next clinical step.
Models need pathways. A right answer still needs a route to action.
Measure behaviour, not just accuracy. The harms that scale are often patterns, not single unsafe statements.
Safety is designed in. Not bolted on at the end by a clinician.
š„ Watch the full panel below. Dani, Dr Lucinda, Dr Paul and I go much deeper on clinical safety in AI products, friction as a safety feature, automation bias and what teams should measure before and after deployment.
For the full write-up with practical steps, read issue 041: How to Build Safe Clinical AI Systems.
Hiring spotlight š
š¬š§ Doccla are hiring a Head of Clinical Innovation. The role will be crucial in translating Doccla's clinical model into safe, evidence-based, and commercially impactful products and pathways. š Apply here.
Join the next clinical product panel š¤
š 14th July, 7pm, online
The next clinical product panel is on 14th July and weāll be covering the āclinical product gapā or why healthtech needs a new kind of product leader. š Sign up here.
Thatās all for this week. See you next time! š
š¤ Work with me | š Attend an event | āļø Send a message
Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (ā¦obviously). Based in London. You can find me on LinkedIn.
Made with š for better, safer HealthTech.
A model that "looks fine" on validation data is a model nobody's stress-tested against the patient who doesn't fit the pattern. That's exactly where clinical judgment can't be automated away.