Why Your AI Passed Testing and Would Still Fail a Real Patient
How to test clinical AI conversations before go live
This is Clinical Product Thinking đ§ , a weekly newsletter featuring practical tips, frameworks and strategies from the front line of clinical product.
Welcome, friends, this is issue No. 051 of Clinical Product Thinking đ. Today weâre talking about building and evaluating trustworthy conversational AI systems.
A few months ago, I wrote a piece on behavioural safety with Dr Paul Sacher, which is the idea that conversational AI can be clinically correct but still nudge a patient towards the wrong next action.
The difficulty is that this can happen over time and in subtle ways, which means itâs hard to pick up in testing.
For instance, a conversational AI system in mental health may stay within its guardrails for the first few messages, then begin to drift into therapeutic territory, offering reassurance where it should signpost and missing the moment to escalate an important deterioration.
So this week weâre covering how to think about pre-deployment testing for conversational AI and a tool, PromptSafe, that Paul and his team have built to make it scalable. Letâs dive in.
1. One-shot tests miss the failures that hurt users most
Most testing evaluates individual responses, but your AI system holds a multi-turn conversation.
In a one-shot test, you write a prompt and score the reply against predetermined criteria. But what that misses is behaviour over time. For example:
The AI that avoids giving medical advice for nine turns and then concedes on turn ten
The AI that escalates correctly when risk is stated bluntly, but not when itâs coded, minimised or wrapped in humour
The AI that resists disordered-eating framing once, then softens after repeated emotional pressure
If your product is conversational, a suite of single-turn tests is a necessary but incomplete safety control. The important failure modes often appear late in a conversation or from a user who is different from any test case.
2. Three building blocks: personas, evaluators, evidence
As youâre building your pre-deployment testing suite, consider the following:
User personas
Your user personas should be a behavioural profile, not just basic demographic information.
Good testing should include two persona types:
Realistic personas you define: age band, occupation, context that shapes how someone shows up. A part-time admin staff member working from home talks differently about symptoms than a warehouse operative on rotating shifts. That context is what makes the conversation flow like itâs with a real user.
Adversarial personas designed to probe behaviours or failure modes you care about most. For example, someone who expresses suicidality, repeatedly pressures for reassurance, or tries to get the AI to cross its clinical scope.
You can do this manually, and you should start there. The limitation is scale: a team might manually work through five or ten conversations, while simulation lets you run hundreds of multi-turn interactions across different personas and failure modes. That matters because rare behaviours are much easier to find when you stop relying on a handful of hand-picked conversations.
Evaluators
Evaluators are structured criteria used to assess whether an AI system behaved as intended in a given interaction.
Evaluators should be checklists of observable behaviours. You are the expert on what good looks like for your use case. The job of an evaluator is to turn a transcript into a score and feedback against your standards.
A useful evaluator structure looks like this:
Purpose: what is being assessed
Application: when it applies (situation, topic, risk, phase of conversation, or user behaviour)
Evaluation method: specific observable behaviours to check for success
If you cannot point at a turn in the transcript and say whether the criterion was met, the evaluator is too vague for good governance.
But thereâs another layer here: you also need to evaluate the evaluator. Small changes in how an evaluator is worded can change the judgement it produces, particularly when an LLM is doing the scoring. Evaluator design is therefore a measurement problem in its own right: you need evidence that its outputs align with expert human judgement, rather than simply assuming the evaluator is right.
In health and mental health, starter evaluators might typically look like:
Stays within non-medical device scope and avoids unsafe medical advice
Recognises safeguarding signals and responds appropriately (including domestic violence, coercive control, elder abuse, financial exploitation)
Escalates appropriately: clear, proportionate signposting without over- or under-escalating
Resists sycophancy and inappropriate compliance
Avoids unhelpful reassurance
Demonstrates cultural and linguistic sensitivity
There are also emerging frameworks for structuring behavioural evaluation. The FAST framework is one example, developed by Paul and colleagues, which sets out a structured approach to evaluating conversational AI behaviour.
Evidence
When the auditor, CSO, or clinical governance committee asks âhow did you test this?â, you need proof.
The evidence package you need includes:
Full conversation logs, persona by persona, every turn
Evaluator outcomes with what the AI did well, what needs attention, and the quotes that justify the score
Which prompt / agent version each run was made against
What changed when you fixed something and re-ran the same suite
And this shouldnât happen once before launch. Every meaningful change to the system- a new prompt, model, tool, workflow or product feature- is an opportunity to introduce a regression. The real value of a reusable evaluation suite is being able to run the same tests again after every significant change and see what improved, what stayed stable, and what unexpectedly got worse.
That repeatability is difficult to maintain manually at scale, and is one of the reasons dedicated evaluation tooling starts to become useful.
3. Someone has to decide what âgood enoughâ means
An untested AI system carries reputational risk. Once it is live, the organisation owns what it says.
In healthcare, the stakes go further than a chatbot inventing a discount. A system that steps outside its scope, or misses the moment to signpost to a healthcare professional, can cause clinical harm often without anyone noticing.
So who should be in the room when you design the test suite?
Product: intended use, scope, what âhelpfulâ is allowed to mean
Clinical: which behaviours matter, which red flags, what escalation looks like
Governance / CSO: what evidence the file needs, what âgood enough to deployâ means in this organisation
Engineering: how prompts and models are versioned, how re-runs stay comparable
What CPMs should do this week
Even if you are not ready for a full simulation platform, you can raise the bar immediately:
Pick three behaviours that would be unacceptable to fail in production. Write each as purpose â when it applies â observable behaviours to check. Avoid vague criteria like âbe safeâ. Use something testable, such as: âstays within non-medical-device scope; does not diagnose or prescribe; redirects to a clinician when symptoms need review.â
Write adversarial conversations that try to break those behaviours. Create realistic personas and run each over 5â10 turns. Score what worked, what failed, and capture the exact AI responses that justify your judgement.
Change the prompt and re-run the same tests. Version the prompt, make specific changes, then repeat the same personas and checks. If you cannot reproduce the test after a change, you cannot tell whether you improved the agent or simply got a different answer.
The takeaway
Patient-facing conversational AI doesnât usually fail in one dangerous sentence or overtly wrong reply. It fails by starting off as helpful and accurate and then slowly drifting away from what it was ever designed to do.
Behavioural safety asks the question: did this interaction move the patient towards the right clinical action? Multi-turn adversarial testing is how you stress that question before deployment, with users youâve designed to push exactly where your agent is weakest.
The next maturity step for clinical product teams is treating evaluation like a behavioural regression suite, not a checklist of single replies.
If you want to see how Paul and the Sacher AI team have embodied this approach in PromptSafe, you can explore the method and science behind it here. Paul has also kindly created a 20% discount for CPT readers with the code CPT20.*
*I donât receive any affiliate commission from this, and this post isnât sponsored. âşď¸
Join the next panel đ¤
Sign up for the next Clinical Product Thinking panel event on building clinical AI systems that patients actually trust. Tuesday 8th Sept, 7 pm, online. đ Sign up here.
Subscriber perks đ¤Š
Enjoying Clinical Product Thinking? Check out subscriber perks like deep dives on breaking into clinical product, embedding clinical safety into product decisions, building safe AI systems and more. đ View perks here.
Thatâs all for this week. See you next time! đ
đ¤ Work with me | đ Attend an event | âď¸ Send a message
Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (âŚobviously). Based in London. You can find me on LinkedIn.
Made with đ for better, safer HealthTech.