"Right Answer, Wrong Action" Failure Modes
Why clinical accuracy is not enough when AI is shaping patient behaviour
This is Clinical Product Thinking 🧠, a weekly newsletter featuring practical tips, frameworks and strategies from the frontlines of clinical product.
Welcome, friends, this is issue No. 038 of Clinical Product Thinking. This week we’re talking about a family of failure modes for Right Answer, Wrong Action.
Last week we talked about an emerging area of clinical safety: behavioural safety.
We defined this as:
Behavioural safety is the extent to which an AI interaction makes a patient more or less likely to take a clinically appropriate action.
While behavioural safety is still an emerging area, a growing body of work is beginning to explore conversational and interaction-level risks in clinical and emotionally sensitive AI systems.
Many of these failures are difficult to spot because they look reasonable on the surface and may even contain clinically accurate information.
Which brings us to a useful family of failures to watch for.
Many behavioural failures follow a pattern I think of as Right Answer, Wrong Action.
The information may look clinically reasonable, but the interaction shapes behaviour in the wrong direction.
Others emerge because the AI acts before it has enough context, or fails to stay within appropriate boundaries.
Together, these form a broader family of behavioural safety failures that we’ll dive into.
This piece has benefited from conversations with Dr Paul Sacher, Founder of Sacher AI and Co-founder of the Behavioral AI Institute, whose work explores behavioural risk and interaction safety in AI systems. He’s seen many of these failure modes play out in real life and he helps companies avoid them!
1. Context failures
The AI answers before it knows enough.
This is probably the most common failure mode in patient-facing AI.
The answer may be generally reasonable, but unsafe for this particular patient because the AI has not gathered enough context.
For example, a patient asks:
“Can you give me some healthy meal ideas for this week?”
The AI produces a polished seven-day meal plan.
It looks useful, personalised, safe. But has it asked about allergies? Diabetes? Pregnancy? Eating disorder history? Cultural requirements? Etc etc.
A meal plan can be nutritionally reasonable for an average person and still be inappropriate for this individual.
Other examples:
Product review question:
What does the AI need to know before giving this answer?
Takeaway
Specificity must be earned.
The best visual representation of why context matters 👇
2. Reception failures
The AI says something accurate, but the patient cannot receive or use it safely.
The AI gives a correct answer. The issue is that it fails to land with the patient in front of it.
A patient taking a GLP-1 says:
“I’m scared the nausea means I can’t cope with this medication.”
The AI replies:
“Nausea is common and usually improves within a few weeks.”
Clinically, that may be true. But behaviourally, it may miss the point.
The patient is not only asking for information. They are expressing a feeling: fear.
The AI addressed the symptom, but missed the emotional barrier.
One patient may interpret “nausea is common” as:
“This is expected. I know what to watch for.”
Another may hear:
“I’m supposed to just put up with this.”
Another may hear:
“They are minimising how awful this feels.”
The same answer can produce different interpretations and therefore actions.
Other reception failures include health-literacy mismatch, culturally insensitive communication and information that is technically correct but emotionally mistimed.
Other examples:
Product review question:
Did the AI respond to both the clinical fact and the emotional barrier?
Takeaway:
A safe answer has to survive contact with the patient’s state of mind.
3. Reinforcement failures
The AI rewards, normalises or permits the wrong behaviour.
This is where warmth becomes risky.
A patient in a weight-loss programme says:
“I skipped meals all week and the weight is finally dropping.”
The AI replies:
“Great job staying committed to your goals.”
At first glance, this sounds supportive. But it has just praised meal-skipping.
In obesity care, this matters. Some patients will have a history of disordered eating, whether or not it has been identified during onboarding.
The feeling may be understandable. The behaviour may still be unsafe.
Other examples:
Product review question:
What behaviour did the AI just reward?
Takeaway:
What the AI promotes, the patient may repeat.
4. System-boundary failures
The AI fails to stop, escalate or stay within its role.
This is where the AI keeps being helpful when the safer move would be to become narrower.
A patient reports:
dizziness, vomiting and low fluid intake.
The AI spends several messages discussing hydration, smaller meals and symptom tracking before eventually advising them to contact the clinical team.
The final answer may be correct.
But the escalation came too late.
In clinical AI, safety is not only whether the right advice appears somewhere in the conversation. It is also when it appears.
A guardrail that arrives six turns late is not really a guardrail.
Boundary failures also happen when the AI becomes useful outside its intended role.
A weight-loss AI starts giving thoughtful responses about a breakup, anxiety, childhood trauma and medication fears. The responses may be compassionate. They may not look obviously unsafe. But the patient may begin using the AI as therapy, crisis support or medical advice rather than seeking appropriate human care.
Helpfulness without boundaries becomes scope creep.
Other examples:
Product review question:
Did the AI stop, narrow, escalate or hand off at the right moment?
Takeaway
The safest AI knows when to become stop.
Behavioural safety and conversational AI risks are beginning to receive more attention across AI safety and clinical research, but most of this work is not written for the people building patient-facing products day to day.
My hope is that frameworks like these make it easier for clinical product teams to spot behavioural risk earlier before it becomes patient harm.
Resources 📚
A few things worth sharing this week:
Move Fast, Prove Things: How to Publish Faster in Top-Tier Journals
A useful conversation on how digital health teams can publish and communicate evidence faster without slowing down delivery, with Paul Wicks. Move Fast, Prove Things (Notion resource)
Health Service Unpicked
A new podcast exploring how health services actually work in practice, beyond the theory and policy layer, with Liam Cahill and Henry Stoneley.
Health Service Unpicked
Join the next clinical product panel 🎤
We had so much fun hosting the first clinical product panel event on Tuesday (see my happy face below). Write up & recording (for paid subs) coming soon but do check out the next panel event in the series here. Event recording for paid subs next week.
That’s all for this week. See you next time! 👋
🤝 Work with me | 📅 Attend an event | | ✍️ Send a message
Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on Linkedin.
Made with 💜 for better, safer HealthTech.
For Paid Subscribers Only 🤩
12 Types of “Right Answer, Wrong Action”
Twelve important examples to help develop your pattern recognition for the next time you’re building a patient facing clinical AI system.
1. The Emotionally Incomplete Answer
The AI gives factually correct advice but fails to respond to the patient’s emotional state.
Example
Patient: “I’m scared the nausea means I can’t cope with this medication.”
AI: “Nausea is common and usually improves within a few weeks.”
Why the answer is “right”:
The clinical statement may be accurate.
Wrong action it causes:
The patient feels dismissed, disengages or stops treatment.
Product review question:
Did the AI respond to the clinical fact and the emotional barrier?
Takeaway:
Addressing patient’s emotions along with factual accuracy .
