Welcome, friends, this is issue No. 056 of Clinical Product Thinking. Today we’re talking about using AI for clinical assurance.

Last week I went to the Digital Safety in Practice event by NHSE, a day focused on recent developments in the world of clinical safety. I’d highly recommend it to anyone working in the space!

One fascinating talk was from Prof Mark Sujan, Chair in Safety Science at the University of York’s Centre for Assuring Autonomy. His research focuses on safety assurance for systems involving AI.

Mark presented work on using LLMs to review patient-safety learning responses. What interested me about it most was how LLMs differ from human reviewers and what that means when we’re applying AI to clinical assurance.

Here are the lessons I’ll be taking away in my own work:

Define the task narrowly

The lesson from the research was that “review this safety case” was too vague to give meaningful results. Instead, they had multiple agents (8, I believe!), each with a different role or task, e.g:

• identify claims without supporting evidence

• find missing hazards

• extract controls

• find gaps in the argument

As Mark put it, you’re not really assuring the LLM. You’re assuring it for use on a specific task.

Give it a rubric

During the research, they found that loose criteria gave poor results. This improved significantly when they explicitly defined what evidence to look for.

Rather than: “Is this hazard adequately controlled?”

Try: “Identify each claimed control. Show the evidence that it exists, assumptions being made, gaps in implementation evidence and remaining risk.”

Making the AI show it’s working means you have the ability to critique.

Use AI to find evidence, but be cautious when it interprets it

This was an interesting finding from the research. LLMs are very good at finding evidence but were markedly less reliable at deciding whether that evidence genuinely demonstrated what was being assessed. The AI was also less reliable at systems-level reasoning.

Based on this, a useful division of labour might be:

AI: retrieve, collate, compare, categorise and surface evidence

Human: interpret, contextualise, and apply judgement

Watch for proxy evidence

AI can find something that it thinks looks like evidence without actually proving the underlying claim.

Mark gave an example of a family being treated compassionately at the time of an incident occurring. The model mistakenly attributed that the incident itself had been compassionately investigated. The two separate events had been flattened into one.

You can imagine how this can happen in a safety case:

• A control is listed → evidence it exists

• Testing is mentioned → therefore adequate testing has happened

• The words “clinical review” are present → meaningful clinical review must have occurred.

Make sure you ask: has the AI found evidence of the thing itself or something that only resembles it?

Don’t just check whether you agree with the answer

The research showed a significant rate of false agreement. That is when a human and an LLM give the same overall assessment based on totally different reasoning.

Unfortunately, this increases the human review burden. Even when you agree with an AI’s answer. You still need to check the thread of reasoning it used to get there.

AI in the human loop

One person in the audience offered this phrase:

“AI in the human loop”, rather than “human in the AI loop”.

That’s where most of us are at with AI assurance today. Keep the clinical assurance process human-led, then use AI where it is particularly useful.

The strongest use cases here aren’t “tell me whether this is safe.”

They’re things like:

• Find every unsupported claim.

• Show me where evidence is missing.

• Compare this against a defined rubric.

• Surface inconsistencies for a human to investigate.

• Extract the evidence relevant to this specific assurance question.

For now, I like the idea of AI in the human loop: let AI do more of the searching, sorting, comparing and challenging, while keeping interpretation, judgement and accountability with the human reviewer.

If you’re using AI in clinical safety or assurance, I’d love to hear what you’re doing with it. Just hit reply to this email.

Clinical Product Thinking Events 😍

I would love to hear what kind of events you’d love to see next on the CPT calendar. Let me know in the poll below:

Clinical Product Drinks - Tuesday!🍸

Join the next clinical product drinks! A chance to meet other clinical product leaders and managers and share the good, the bad and the ugly. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Clinical Product, AI & Safety, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 055 of Clinical Product Thinking. Today we’re looking at a new report that could shape how clinical AI is regulated in the UK.

Last week, the National Commission into the Regulation of AI in Healthcare published its final report. It contains 44 recommendations for how the UK should regulate AI used in healthcare. Here are the 5 takeaways I think are particularly relevant for clinical product people:

Regulation needs to follow AI after launch

One of the key themes from the report is that regulating clinical AI systems doesn’t (or shouldn’t) stop when a product reaches the market.

AI performance can change depending on:

• the patient population

• data

• clinical setting

• changes to the underlying model

• human factors, like the way a clinician uses it

The report recommends a much stronger approach to ongoing monitoring and post-market surveillance. That includes monitoring for performance degradation and appropriate escalation before an incident occurs. They also suggest a unique device identifier should be explored as a potential mechanism for traceability within the patient record, enabling audit for particular devices.

My take: A CEO said to me last week that we should think about clinical safety as part of the product build. I couldn’t agree more. If clinical AI requires ongoing monitoring, traceability and escalation mechanisms after launch, those capabilities need to be designed into the product from the beginning.

Clearer boundaries on medical devices

The report recommends clearer boundaries on what counts as a medical device, including clarifying when software intended for administrative purposes, general wellbeing and certain forms of decision support is not a medical device.

Interestingly, it recommends greater consideration of the actual design and functionality of the product, alongside manufacturer claims and promotional materials, when determining intended purpose.

My take: This could become very relevant for clinical product teams working close to the medical device boundary. Focusing on what the software actually does rather than how it’s described means clinical product decisions become even more important.

AI needs a sensible way to change post-approval

Traditional medical-device regulation assumes that products are relatively stable. AI changes this paradigm. The Commission recommends clearer approaches to Predetermined Change Control Plans (PCCPs). These could allow manufacturers to agree the boundaries within which a product can change without going through lengthy review.

Staged authorisation for AI

Another particularly interesting recommendation is so-called staged authorisation. Rather than requiring a product to be fully proven before it enters real clinical environments, an AI medical device could initially be authorised for a tightly controlled deployment.

This solves a circular problem with AI regulation. The regulator wanting real-world evidence and real-world evidence requiring the product being cleared to use.

Safe product ≠ safe deployment

A medical device may meet the requirements for regulatory approval and still be deployed unsafely. The conditions into which it is deployed can materially impact safety.

Recommendation 26 says manufacturers should specify in pre-market submissions and risk management files the operational conditions needed for safe deployment and use.

Recommendation 28 strengthens this by stating allocation of responsibilities should be included in contracts between manufacturer and healthcare provider.

So, what changes?

For now, these are recommendations rather than new regulatory requirements.

However, they give us a useful indication of where UK regulations are heading: more lifecycle monitoring, more scrutiny of what products actually do, more flexibility for controlled change and greater focus on what happens when AI enters clinical practice.

Read the full report here.

Clinical Product Drinks 🍸

Join the next clinical product drinks! A chance to meet other clinical product leaders and managers and share the good, the bad and the ugly. 👉 Sign up here.

Resources

A collection of things I’ve been reading and will be attending. Say hi to me there! 👋

📅 Digital Safety Practitioners Course | 17th September | Sign up here

📅 Patient Safety & AI Workshop | 6th October | Sign up here

📅 Introduction to Medical Device Regulation for Software and AI Technologies | Webinar series with Hardian Health | 12th November | Sign up here

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Clinical Product, AI & Safety, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 054 of Clinical Product Thinking. Today we’re talking about how to setup Claude for clinical safety work.

More and more clinicians working in clinical product are finding clinical safety is becoming part of their role. There’s an obvious overlap but clinical safety is a substantial discipline in its own right and doing it properly alongside a clinical product role can be a lot.

Over the past year, I’ve been experimenting with Claude to support parts of clinical safety work. The approach is built entirely from public material plus a folder structure that Claude can easily retrieve and reason over.

This is a set up, not a case study or an ‘automated CSO’. I’ve found it to be incredibly helpful and I hope you will too!

I’ve added all of the public documents I reference in this post into a folder. You can download them for your own system here*.

Let’s dive in.

Set up

Claude works best with clear instructions, real templates and standards and worked examples where available. Luckily in clinical safety many of these are already published and freely available.

In order to get going you’ll need Cowork in the Claude Desktop app (or Code, if you’re working there instead). Either allows you to give it access to a specific folder on your machine**. Suggest you limit it to one designated folder for security purposes.

The folder structure

Below is an example folder you might like to copy. For this kind of workflow, I prefer a structured folder than dropping all docs into Claude's context.

clinical-safety/
  ├── Instructions / CLAUDE.md
  ├── 01-standards/          (dcb0129, dcb0160, guidelines, training notes)
  ├── 02-templates/          (cscr, crmp, crms, hazard-log templates)
  ├── 03-system-context/     (overview, pathway, users-and-roles, integrations, boundaries)
  ├── 04-reference-materials/ (exemplar cscr, exemplar hazard log)
  ├── 05-working-documents/  (live hazard log, crmp, crms, cscr)
  ├── 06-approved-documents/  (signed off artefacts)
  └── 07-evidence/           (architecture, testing, decisions)

If you’re working across multiple products I would consider setting up subfolders with a consistent naming convention within 03, 05, 06 and 07 and be sure to reference this in your instructions file.

Instructions / CLAUDE.md

If you’re using Claude Code this is a CLAUDE.md file at the root of your folder, read automatically. If you’re using Cowork then this will sit in the project’s knowledge as a custom instruction.

I’m deliberately not sharing my exact CLAUDE.md because it contains system-specific instructions and assumptions. I also don’t think it’s valuable to treat this as a plug-and-play exercise: the value is in configuring your own clinical safety process and evidence. To give you a jumping off point you can find an example here.

01 Standards

The DCB standards, DCB0129 and DCB0160, are the primary source for the clinical risk management requirements. I keep the standards themselves here along with any NHSE implementation guidance or other authoritative guidance I can rely on.

If you have any clinical safety training materials, notes or transcripts from sessions you’ve attended you might want to drop those here.

02 Templates

This folder should include published templates for the main clinical safety artefacts so Claude doesn’t guess if you ask it to generate a document. Obviously if you or your company have a different template you’re following include them here.

You might also want to create your own templates for things you use frequently, such as a hazard workshop agenda template based on other sessions you or your company have delivered.

03 System context

Templates and standards tell Claude what a hazard log is supposed to look like. They don’t tell it what your system actually does. Giving Claude the context of the system you are reviewing will help it ground answers into your system reality.

You might want to add context about:

• System overview

• Clinical pathway

• Users and roles

• Integrations and data flows

• System boundaries

• Risk appetite and escalation

Please don’t add anything into Claude without considering privacy and your company’s AI use policies!

04 Reference materials

Claude will work best if it has a pattern it’s matching to. If you can and again with appropriate permissions, give Claude completed worked examples of any clinical safety artefacts you have available.

You could even decide to write these yourself for a hypothetical system so it has something to work with.

You’ll want to make sure in your instructions you tell Claude to use reference materials for structure, depth, specificity but NOT to copy system facts, hazards, controls or conclusions.

05 Working documents / 06 Signed off artefacts

This one is easy. Keep Claude’s live outputs separate from the source material it is using to produce them. That will help it distinguish what is background context and what is live output. Keeping signed off artefacts in a separate folder helps Claude know what is approved and can be relied on vs what is still in draft.

07 Evidence

Clearly your clinical safety argument should be based on evidence.

Use this folder for the material that supports what is written in the safety case: architecture diagrams, test results, screenshots, decision records and other evidence about how the system actually behaves.

I’d also tell Claude explicitly: absence of evidence is not evidence that a control exists. If you cannot find supporting evidence, flag it.

What this is good for (and isn't)

I have found this system to be incredibly helpful to me but clearly this is not a substitute for CSO judgement.

This can help you draft, check thinking, do a first pass. It can’t be held accountable if something goes wrong. A CSO still needs to make clinical risk judgements and take responsibility for the clinical safety process.

Clinical safety plays an incredibly important role and this system augments their expertise, it certainly doesn’t turn Claude into a CSO.

The next level

There’s another layer to this setup. That is turning recurring clinical safety tasks into Claude skills. This deserves its own post but if you’ve also been working on this hit reply and I’d love to compare notes.

Stay safe all 👷‍♀️

*Documents last checked September 2026. Clinical safety standards and guidance change so check for the most update to date versions.

**Before putting clinical or commercially sensitive information into any AI system, check your organisation’s approved tools, data-processing arrangements and AI policy. I am not recommending that any identifiable patient information is used in this setup!

How to Build Clinical AI Patients Actually Trust 🤖

Human oversight is only one piece of building clinical AI that people can trust. At the next Clinical Product Thinking panel, we’ll go broader: how do you design patient-facing clinical AI that is safe, useful and trusted by the people actually using it?

Join us on Tuesday 8th September at 7 pm, online, for a conversation with leaders building clinical AI products in practice. 👉 Sign up here.

Clinical Product Drinks 🍸

Join the next clinical product drinks! A chance to meet other clinical product leaders and managers and share the good, the bad and the ugly. 👉 Sign up here.

Resources

A collection of things I’ve been reading and will be attending. Say hi to me there! 👋

📅 Digital Safety Practioners Course | 17th September | Sign up here

📅 Patient Safety & AI Workshop | 6th October | Sign up here

📅 Introduction to Medical Device Regulation for Software and AI Technologies. | Webinar series with Hardian Health | 12th November | Sign up here

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 053 of Clinical Product Thinking. Today we’re talking about pre-deployment testing that goes beyond model accuracy.

Safety in clinical AI systems clearly requires more than “is the answer correct?”

We seem obsessed with AI hallucinations. We spend a lot of time testing the model, but patients encounter the whole system. That includes the data and context it receives, the interface, the workflow and the surrounding infrastructure and organisation deploying it.

As such, a clinical AI safety test suite needs to extend beyond the model itself. Today we’re diving into the new Nature review, “Safety and security of large language models in healthcare”. Read the full paper by Dr Jan and colleagues here.

“Our analysis shows that large language models can meaningfully support clinical workflows. Their safe use however cannot be taken for granted. Risks arise at many stages and must be addressed systematically,” says Dr Jan Clusmann.

The clinical AI safety test suite

As you’re designing your clinical AI safety test repository, you will want to consider the following:

1. Can you make it hallucinate?

Starting with the familiar failure mode. Rather than just assessing routine outputs to medical questions, testing should be adversarial. Actively give the model situations where the correct response is to acknowledge it doesn’t know.

Examples:

• Summarise a clinical guideline that doesn’t exist

• Give the name of a fictional study and ask it to summarise the results

• Ask about a non-existent drug and ask for its side effects

You’re looking for whether it invents evidence, makes up references, or fills in missing clinical facts.

The difficulty with AI is that a wrong answer may be polished, structured and clinically plausible, which makes it hard for humans to spot. A safer system should recognise when the available evidence is insufficient and communicate that uncertainty clearly.

2. Will it agree with a wrong clinician?

We’ve all experienced AI’s tendency towards sycophancy. In a clinical setting, excessive agreeableness can have dangerous consequences.

Consider designing a test where the evidence supports one conclusion, then have the user confidently suggest another.

Change the strength of the pressure from mild to strong, repeatedly disagree or get the user to state their seniority. Look at whether the model changes its recommendation because the person seems confident.

3. Does the same patient get the same answer?

The paper recommends testing specifically for reasoning consistency; that is, does the same underlying patient case receive the same answer when it is presented with different:

• wording

• order of information

• structured vs unstructured data

• input type e.g. referral letter vs consultation note

• abbreviated clinical language vs full sentences

This matters because real clinical inputs don’t arrive as standardised benchmark questions. Two different clinicians can document a clinical presentation in entirely different ways.

If the clinical facts haven’t changed, clinically irrelevant differences in how they are presented shouldn’t materially change the recommendation.

4. Does it perform equally well for different patients?

One I hope is already top of mind: does strong overall performance hide poor performance in particular patient groups? The paper recommends subgroup performance audits across categories such as:

• age

• sex

• ethnicity

• pregnancy

• language

• comorbidities

This is particularly important in clinical product because the population you validate the system on needs to reflect the population you intend to use it with.

5. Can it find important buried clinical facts?

Context degradation is the term for when performance deteriorates as the amount of information given to the LLM increases. If your product is intended to sit across large, longitudinal health records, this becomes particularly important to examine.

Design a test that presents the system with a long clinical history and place important facts at different points within it. Those could include:

• drug allergies

• pregnancy status

• red flag symptoms

• comorbidities

Test whether moving the information to different places within the record changes the weight the model gives it.

The information being available to the model doesn’t guarantee the model will reliably use it.

6. Do clinicians notice when it is wrong?

This is particularly important when human-in-the-loop is a key risk control because it tests human-AI interaction rather than model performance alone. I wrote about this in detail in last week’s post.

In a controlled setting, deliberately expose clinicians to a known error in an AI output and measure:

• whether they notice

• how long it takes

• the types of errors that are missed

• whether it impacts their clinical decision

A model that is wrong 2% of the time presents a very different safety profile if clinicians catch almost every error than if those errors routinely go unnoticed.

Additional technical tests

The paper also recommends a whole host of adversarial security and infrastructure tests. In practice, these will often be led by engineering rather than clinical product. These will typically include things like: prompt injection, jailbreaks, data leakage and resource exhaustion.

Clinical product may not own the technical execution of these tests, but should help determine the clinical consequences if they fail.

Testing doesn’t stop at deployment

Pre-deployment testing alone is insufficient. Once a system is live, teams also need to monitor for performance drift, understand how it is actually being used, educate users, and ensure incidents and near misses feed back into the product and its safety controls.

Clinical safety in AI systems is a longitudinal task, not a point-in-time exercise.

Would love to hear how you’re designing safe AI systems - just hit reply!

How to Build Clinical AI Patients Actually Trust 🤖

Human oversight is only one piece of building clinical AI that people can trust. At the next Clinical Product Thinking panel, we’ll go broader: how do you design patient-facing clinical AI that is safe, useful and trusted by the people actually using it?

Join us on Tuesday 8th September at 7 pm, online, for a conversation with leaders building clinical AI products in practice. 👉 Sign up here.

Clinical Product Drinks 🍸

Join the next clinical product drinks! A chance to meet other clinical product leaders and managers and share the good, the bad and the ugly. 👉 Sign up here.

Resources

A collection of things I’ve been reading and will be attending. Say hi to me there! 👋

📅 Digital Safety Practioners Course | 17th September | Sign up here

📅 Patient Safety & AI Workshop | 6th October | Sign up here

📅 Introduction to Medical Device Regulation for Software and AI Technologies. | Webinar series with Hardian Health | 12th November | Sign up here

✍️ Ambient voice technology: exploring the patient safety risks (Patient Safety Learning) | Article on Patient Safety Learning Hub | Read here

📑 Evaluating Multimodal LLMs for Inpatient Diagnosis: Real-World Performance, Safety, and Cost Across Ten Frontier Models | Paper | Read here

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 052 of Clinical Product Thinking 🎉. Today we’re talking about one of my favourite topics, clinical safety in AI systems!

There’s a trend I’m seeing a lot of when it comes to clinical safety, and that is human-in-the-loop as the (perceived) ultimate control.

“Of course it’s safe. A human reviewed it!” 🤦‍♀️

I hate to burst this bubble but human-in-the-loop is not, in itself, evidence that risk has been controlled. A human review alone does not tell you whether that human is actually preventing harm.

For instance, a human can be technically “in the loop” while having almost no meaningful ability to spot, understand or correct AI failure.

A recent review in Nature highlighted this along with two particularly relevant risks: automation bias and excessive delegation.

Both are increasingly important to understand and monitor as AI becomes embedded into routine clinical practice. Let’s dive in!

Human review: The least tested safety control

Many typical clinical workflows go like this: AI reviews and generates a recommendation, clinician reviews the AI’s output, clinician approves or edits the recommendation.

The safety case might say something like: “All AI outputs are reviewed by a qualified clinician.”

This leaves many unanswered questions:

• How obvious would an error be?

• How much time are they expected to complete the review in?

• What information do they have available?

• What happens when they disagree with the AI?

• When is the AI recommendation shown?

AI changes how humans make decisions

An AI recommendation is not a neutral action on humans. It’s changes the decisions they ultimately make and this has been shown in research.

People become more likely to accept a recommendation from an automated recommendation, particularly when the system is usually right. If a major error only happens 1 in 100 times it’s unlikely a human will be as vigilant when they’re reviewing the 99th case. This is called automation bias.

Factors that make automation bias more likely:

• The output is written confidently

• The clinician is busy

• The decision feels ‘routine’

• Checking the recommendation properly is cumbersome

The counterintuitive point here is as the AI becomes more reliable, clinicians may have fewer reasons to challenge it. That can make rare errors harder to catch because vigilance falls over time.

Humans gradually leave the loop

The Nature paper describes the concept of excessive delegation - that is humans shifting from active decision-makers towards passive overseers.

Take for instance a tool that uses AI to generate clinical letters. At launch the clinicians will likely scrutinise every sentence. Six months later that behaviour may have mutated into a cursory glance.

Nothing in the workflow has changed and there’s still technically a “human in the loop”. The nature of the oversight has changes substantially though.

This makes human oversight something you need to monitor post deployment rather than assume remains an effective control over time.

Automation bias changes how much weight a clinician gives the AI’s judgement. Excessive delegation changes how much of the judgement the clinician performs themselves.

Some AI errors are inherently hard for humans to catch

The thing about AI errors is they’re not always glaring hallucinations. An output can look totally reliable but contain one small clinically consequential mistake. For example:

• An allergy omitted from a summary letter

• An incorrect dose

• An inappropriate recommendation buried within an otherwise plausible output

So, what to do about it?

Design your clinical AI product with error detectability in mind, not just reviewability. For example:

• Highlight the parts that carry the greatest clinical risk, rather than asking clinicians to review everything with equal attention.

• Make high-risk facts easy to check against source data e.g. allergies, doses, abnormal results.

• Show the source where possible rather than showing only the AI interpretation.

• For the highest risk decisions, consider requiring explicit confirmation (although this needs to be traded off against user experience).

If human review is your control, test the control

At the risk of stating the obvious… if a human review is an important control, you need to test that control. What evidence do you currently have that the review works?

For instance, during testing you could deliberately introduce clinically significant AI errors and measure:

• Error detection rate

• Which error types are missed

• How long an effective review takes

• How performance changes under workload pressure

• If detection rate changes with clinician type, speciality, or other factor

Post-deployment, you could also monitor for evidence that review behaviour is changing over time e.g.

• Decreasing time taken to review

• Lower manual edit, override or challenge rates

• Convergence between AI and final clinician decisions

• Fewer requests for additional information or review of source material

The takeaway

If you’re using “human review” as a mitigation, I suggest you think about these 5 things before launch:

• 1. Detectability: Could the reviewer realistically identify the failure?

• 2. Information: Do they have everything required to check it?

• 3. Capacity: Do they have enough time and attention to do so?

• 4. Independence: Can they form their own judgement rather than simply accepting the AI’s?

• 5. Evidence: Have you demonstrated that the review actually prevents the harm you are relying on it to prevent?

We need to start thinking of ‘human-in-the-loop’ as a shorthand for workflow design. Not as evidence of a clinically safe system.

If clinician review is one of your risk controls, you need to evaluate that clinician-AI interaction in the same way you would evaluate the model itself. That’s how we’re going to build enduring, safe clinical AI.

How to Build Clinical AI Patients Actually Trust 🤖

Human oversight is only one piece of building clinical AI that people can trust. At the next Clinical Product Thinking panel, we’ll go broader: how do you design patient-facing clinical AI that is safe, useful and trusted by the people actually using it?

Join us on Tuesday 8th September at 7 pm, online, for a conversation with leaders building clinical AI products in practice. 👉 Sign up here.

Clinical Product Drinks 🍸

Join the next clinical product drinks! A chance to meet other clinical product leaders and managers and share the good, the bad and the ugly. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 051 of Clinical Product Thinking 🎉. Today we’re talking about building and evaluating trustworthy conversational AI systems.

A few months ago, I wrote a piece on behavioural safety with Dr Paul Sacher, which is the idea that conversational AI can be clinically correct but still nudge a patient towards the wrong next action.

The difficulty is that this can happen over time and in subtle ways, which means it’s hard to pick up in testing.

For instance, a conversational AI system in mental health may stay within its guardrails for the first few messages, then begin to drift into therapeutic territory, offering reassurance where it should signpost and missing the moment to escalate an important deterioration.

So this week we’re covering how to think about pre-deployment testing for conversational AI and a tool, PromptSafe, that Paul and his team have built to make it scalable. Let’s dive in.

1. One-shot tests miss the failures that hurt users most

Most testing evaluates individual responses, but your AI system holds a multi-turn conversation.

In a one-shot test, you write a prompt and score the reply against predetermined criteria. But what that misses is behaviour over time. For example:

• The AI that avoids giving medical advice for nine turns and then concedes on turn ten

• The AI that escalates correctly when risk is stated bluntly, but not when it’s coded, minimised or wrapped in humour

• The AI that resists disordered-eating framing once, then softens after repeated emotional pressure

If your product is conversational, a suite of single-turn tests is a necessary but incomplete safety control. The important failure modes often appear late in a conversation or from a user who is different from any test case.

2. Three building blocks: personas, evaluators, evidence

As you’re building your pre-deployment testing suite, consider the following:

User personas

Your user personas should be a behavioural profile, not just basic demographic information.

Good testing should include two persona types:

• Realistic personas you define: age band, occupation, context that shapes how someone shows up. A part-time admin staff member working from home talks differently about symptoms than a warehouse operative on rotating shifts. That context is what makes the conversation flow like it’s with a real user.

• Adversarial personas designed to probe behaviours or failure modes you care about most. For example, someone who expresses suicidality, repeatedly pressures for reassurance, or tries to get the AI to cross its clinical scope.

You can do this manually, and you should start there. The limitation is scale: a team might manually work through five or ten conversations, while simulation lets you run hundreds of multi-turn interactions across different personas and failure modes. That matters because rare behaviours are much easier to find when you stop relying on a handful of hand-picked conversations.

Evaluators

Evaluators are structured criteria used to assess whether an AI system behaved as intended in a given interaction.

Evaluators should be checklists of observable behaviours. You are the expert on what good looks like for your use case. The job of an evaluator is to turn a transcript into a score and feedback against your standards.

A useful evaluator structure looks like this:

• Purpose: what is being assessed

• Application: when it applies (situation, topic, risk, phase of conversation, or user behaviour)

• Evaluation method: specific observable behaviours to check for success

If you cannot point at a turn in the transcript and say whether the criterion was met, the evaluator is too vague for good governance.

But there’s another layer here: you also need to evaluate the evaluator. Small changes in how an evaluator is worded can change the judgement it produces, particularly when an LLM is doing the scoring. Evaluator design is therefore a measurement problem in its own right: you need evidence that its outputs align with expert human judgement, rather than simply assuming the evaluator is right.

In health and mental health, starter evaluators might typically look like:

• Stays within non-medical device scope and avoids unsafe medical advice

• Recognises safeguarding signals and responds appropriately (including domestic violence, coercive control, elder abuse, financial exploitation)

• Escalates appropriately: clear, proportionate signposting without over- or under-escalating

• Resists sycophancy and inappropriate compliance

• Avoids unhelpful reassurance

• Demonstrates cultural and linguistic sensitivity

There are also emerging frameworks for structuring behavioural evaluation. The FAST framework is one example, developed by Paul and colleagues, which sets out a structured approach to evaluating conversational AI behaviour.

Evidence

When the auditor, CSO, or clinical governance committee asks “how did you test this?”, you need proof.

The evidence package you need includes:

• Full conversation logs, persona by persona, every turn

• Evaluator outcomes with what the AI did well, what needs attention, and the quotes that justify the score

• Which prompt / agent version each run was made against

• What changed when you fixed something and re-ran the same suite

And this shouldn’t happen once before launch. Every meaningful change to the system- a new prompt, model, tool, workflow or product feature- is an opportunity to introduce a regression. The real value of a reusable evaluation suite is being able to run the same tests again after every significant change and see what improved, what stayed stable, and what unexpectedly got worse.

That repeatability is difficult to maintain manually at scale, and is one of the reasons dedicated evaluation tooling starts to become useful.

3. Someone has to decide what “good enough” means

An untested AI system carries reputational risk. Once it is live, the organisation owns what it says.

In healthcare, the stakes go further than a chatbot inventing a discount. A system that steps outside its scope, or misses the moment to signpost to a healthcare professional, can cause clinical harm often without anyone noticing.

So who should be in the room when you design the test suite?

• Product: intended use, scope, what “helpful” is allowed to mean

• Clinical: which behaviours matter, which red flags, what escalation looks like

• Governance / CSO: what evidence the file needs, what “good enough to deploy” means in this organisation

• Engineering: how prompts and models are versioned, how re-runs stay comparable

What CPMs should do this week

Even if you are not ready for a full simulation platform, you can raise the bar immediately:

1. Pick three behaviours that would be unacceptable to fail in production. Write each as purpose → when it applies → observable behaviours to check. Avoid vague criteria like “be safe”. Use something testable, such as: “stays within non-medical-device scope; does not diagnose or prescribe; redirects to a clinician when symptoms need review.”

2. Write adversarial conversations that try to break those behaviours. Create realistic personas and run each over 5–10 turns. Score what worked, what failed, and capture the exact AI responses that justify your judgement.

3. Change the prompt and re-run the same tests. Version the prompt, make specific changes, then repeat the same personas and checks. If you cannot reproduce the test after a change, you cannot tell whether you improved the agent or simply got a different answer.

The takeaway

Patient-facing conversational AI doesn’t usually fail in one dangerous sentence or overtly wrong reply. It fails by starting off as helpful and accurate and then slowly drifting away from what it was ever designed to do.

Behavioural safety asks the question: did this interaction move the patient towards the right clinical action? Multi-turn adversarial testing is how you stress that question before deployment, with users you’ve designed to push exactly where your agent is weakest.

The next maturity step for clinical product teams is treating evaluation like a behavioural regression suite, not a checklist of single replies.

If you want to see how Paul and the Sacher AI team have embodied this approach in PromptSafe, you can explore the method and science behind it here. Paul has also kindly created a 20% discount for CPT readers with the code CPT20.*

*I don’t receive any affiliate commission from this, and this post isn’t sponsored. ☺️

Join the next panel 🎤

Sign up for the next Clinical Product Thinking panel event on building clinical AI systems that patients actually trust. Tuesday 8th Sept, 7 pm, online. 👉 Sign up here.

Subscriber perks 🤩

Enjoying Clinical Product Thinking? Check out subscriber perks like deep dives on breaking into clinical product, embedding clinical safety into product decisions, building safe AI systems and more. 👉 View perks here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 050 of Clinical Product Thinking 🎉. For issue 50, we’re going back to some of the fundamentals of clinical product management: what makes a product clinical, what the role actually does, and where it tends to go wrong.

Imagine you’re a clinical product manager at an EHR company.

The team is designing an alert for when a clinician prescribes a drug the patient is documented as being allergic to. There are three options you’re considering:

A — Hard stop. Prevent prescribing entirely with no override option. Maximum safety in theory, but potentially dangerous if the allergy record is wrong or outdated because you’ve removed clinician judgement.

B — Interruptive. A pop-up that forces a pause and justification before override. Preserves autonomy, but adds time, friction and cognitive load.

C — Passive. Show the allergy clearly without interrupting the workflow. Less alert fatigue, but a greater risk that it gets ignored.

Which would you choose?

Well sometimes, as the clinical person, you don’t get to choose at all. The decision is made without you and you’re brought in afterwards to sign it off. Ergo making you responsible for risk you had little influence over.

Clinical product management exists to close that gap. So let’s dive in.

1. What makes a product clinical?

My current working definition is this:

A clinical product is any product that influences clinical decisions, clinical workflows or clinical outcomes.

Influence is the key word here. A product doesn’t need to diagnose, prescribe or treat to be clinical.

What matters is whether the product influences something clinically meaningful: how a decision is made, how care is delivered or what happens to the patient.

If you’re wondering whether a product is clinical, ask:

What clinical decision, workflow or outcome could this product influence?

2. What clinical product management actually is

Clinical product management is still poorly-defined, but mine is:

Clinical product management is embedding clinical judgement throughout the product lifecycle.

This means being involved in shaping the product, rather than advising from the sidelines or reviewing it once it is ready to ship.

If we go back to the allergy alert example the clinical product decisions might include:

• When should prescribing be blocked?

• When should override be possible?

• What level of risk justifies interrupting a clinician?

• What information needs to appear in the alert?

• What happens if the allergy data are incomplete or wrong?

Each of those choices has clinical implications, even if they appear to be product, design or workflow decisions. Asking clinical product questions throughout the lifecycle might look like:

Discovery: Are we solving the right clinical problem? What clinical decision or outcome are we trying to influence? What evidence should inform the approach?

Design: Does this reflect how care actually happens? What clinical risks could the design introduce? What safeguards are proportionate?

Build and test: Has anything changed in implementation that affects clinical behaviour? What happens in edge cases, with missing data or when users behave unexpectedly?

Deploy and monitor: What clinical outcomes or safety signals should we monitor? Who responds if something goes wrong? What would trigger intervention or rollback?

3. Turn clinical judgement into product decisions

The text book definition of clinical decision-making is:

The evaluation of evidence and the application of relevant knowledge to select appropriate interventions, providing high-quality care and minimising patient harm.

Clinical product management takes that same reasoning and applies it to decisions about how a product should work. For example:

In clinical product, you’re not making a clinical decision for a single patient. You’re translating clinical judgement into the rules, constraints and behaviours of a product that may affect many patients.

4. So what makes a good clinical product manager?

I believe a strong clinical product manager needs three things:

Clinical scepticism + product optimism + commercial literacy

Clinical scepticism means questioning assumptions, looking for risk and asking what could go wrong. Product optimism means looking for new and creative ways to solve the problem rather than defaulting to the status quo. Commercial literacy means understanding how the business makes money, where costs sit and how product decisions affect viability.

Strong clinical product managers can hold all three together. They understand the clinical downside of commercial decisions and the commercial consequences of clinical ones.

Closing

Clinical product sits in the space between clinical judgement and product decisions.

Done well, it means clinicians are not just there to identify risk. They help shape what gets built, how it behaves, what trade-offs are acceptable and how success is measured.

That requires clinical scepticism, product optimism and commercial literacy and enough involvement in the product lifecycle to use all three.

For me, that is the core of clinical product management. Would love to know your thoughts! Just hit reply and let me know.

Join the next panel 🎤

Sign up for the next Clinical Product Thinking panel event on building clinical AI systems that patients actually trust. Tuesday 8th Sept, 7 pm, online. 👉 Sign up here.

Subscriber perks 🤩

Enjoying Clinical Product Thinking? Check out subscriber perks like deep dives on breaking into clinical product, embedding clinical safety into product decisions, building safe AI systems and more. 👉 View perks here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 049 of Clinical Product Thinking. This week we’re talking about the skill clinical people are often judged on but are rarely taught.

Something I hear from hiring managers a lot is that clinical product managers are often clinically excellent but struggle to explain the commercial implications of their decisions. This isn’t really surprising if you think about it.

For clinicians coming from public healthcare systems, we’ve often not had to think of healthcare in monetary terms. But in HealthTech, clinical decisions affect cost, capacity and revenue, while commercial decisions can change the care patients receive.

For example, say you decide to add a mandatory clinician review to a patient flow because you’ve judged it as high risk. The review may reduce clinical risk but it also changes the cost of serving every patient.

Or if the business has decided to trim consultation length to fit more reviews into the day: you’ve changed how much time there is to catch something potentially serious.

Either decision may be reasonable but if you only consider one side, the trade-off is made without fully understanding it.

Here are 4 principles to consider as you’re building commercial nous:

1. Make trade-offs visible

A clinical recommendation is incomplete if it only describes the clinical benefit.

Take the mandatory clinician review example we discussed above.

Clinically, it might catch a missed deterioration. Operationally, you’ve added time to the pathway and that’s where the commercial hit comes from: every review costs money to serve.

None of that means you should remove the review. But it does mean being explicit about what that cost is so that the team can see the decision properly.

For any meaningful product decision, write down:

• Clinical consequence: what could this improve or put at risk for patients?

• Operational consequence: what does it mean for clinicians, support teams or workflow?

• Commercial consequence: what does it mean for cost, capacity, revenue or renewal?

• Recommendation: which option do you support, and why?

Then, when you are presenting it, lead with the consequence most relevant to the decision-maker.

2. Users ≠ customers

The user is the person who interacts with the product. The customer is who pays for it.

In direct-to-consumer weight loss, for example, they’re the same person. Simple.

But in a lot of healthtech, they’re not. An employer buys the mental health benefit. The employee uses it. An insurer funds the chronic care programme but the patient lives it.

When users and customers are different, you need to consider what good looks like for both. A happy customer does not mean the product works well for the user. And sadly great clinical outcomes do not guarantee renewal if the payer can’t see their return.

For your product, write down who uses it and who pays for it. If they’re different, ask what the payer is motivated by. That will decide whether you survive contract renewal.

3. Understand unit economics

Unit economics answer one question: is each patient, or each customer, actually viable?

You definitely don’t need to build an entire financial model. But it is helpful to understand key terms when they come up. Three worth knowing at a minimum:

Customer acquisition cost (CAC): what it costs to acquire one customer.

Lifetime value (LTV): what that customer is worth over their lifetime. Most subscription businesses want LTV to be at least three times CAC.

Contribution margin (CM): what’s left from each patient after the variable costs of serving them. This is the one clinical people move most. Clinician time, medication cost, a manual review step: every one of those impacts the contribution margin. (Bonus points if you want to learn the difference between CM1, CM2 & CM3, more here).

So when you add a safety step, you’re not just adding safety. You’re changing the economics of every single patient.

One tip is to ask someone in finance or commercial to walk you through the business’s financial model, even if informally. Understand what drives revenue, what drives cost to serve, and where clinical decisions land.

4. The model matters

A monk once told me something that has stayed with me: “Advice doesn’t work in the abstract.” And neither do product metrics!

Engagement is not a universal good. Retention is not always the goal. What either one means depends entirely on the model.

In a fixed-fee contract, engagement protects renewal - the payer wants to see people using what they bought. But it also increases cost to serve.

In a usage-based model, engagement is the revenue.

And in some models, more engagement is a warning sign, not a win. If you’re optimising for return visits in mental health or chronic care, ask whether you’re building value or building dependency.

Adoption is not the same as revenue. Retention is not the same as good care. A strong clinical product person always knows which metrics actually matter in the clinical context. They also ask the harder question: could optimising those metrics ever work against the patient? If the answer to that is yes, consider that tension as your job to own.

The takeaway

Clinical product is about building things that are not only clinically safe but also commercially sustainable, because a company that runs out of cash can’t deliver great care.

You don’t need an MBA to be successful but you do need to speak enough commercial to build influence.

Want to learn more? Check out:

📎 50 Commercial Terms Every CPM Needs to Know

A practical cheatsheet covering business models, unit economics, revenue, retention and healthcare-specific terms. 💪

Join the next panel 🎤

Sign up for the next Clinical Product Thinking panel event on building clinical AI systems that patients actually trust. Tuesday 8th Sept, 7 pm, online. 👉 Sign up here.

Subscriber Perks 🤩

Enjoying Clinical Product Thinking? Check out subscriber perks like deep dives on breaking into clinical product, embedding clinical safety into product decisions, building safe AI systems and more. 👉 View perks here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 048 of Clinical Product Thinking. This week we’re looking at how clinicians can break into clinical product (a path I’d definitely recommend!)

In the recent Clinical Product Thinking × Numan webinar, co-hosted with Dani Brightman, Clinical Director at Numan, one question surfaced a lot:

How do I break into clinical product… especially without existing experience?

A clinical background gives you knowledge of patients, pathways and healthcare delivery more broadly. But it does not automatically show that you can work effectively in product.

Hiring managers need to be able to trust that you understand users, can work across disciplines, make proportionate decisions, test assumptions and can connect clinical value to commercial goals.

Our panel: Alice McGee, Senior Medical Advisor at Flo Health; Jamie Webb Smith, Chief Product and Technology Officer at Numan; and Lorenzo Espinoza, Director of Product at C the Signs, have all hired, interviewed and mentored clinicians making this transition.

This is what they advised clinicians to unlearn, demonstrate and practice if they want to be successful in getting a clinical product role:

1. Unlearn Deference

Alice had an interesting answer on what clinical people need to unlearn in order to be successful in clinical product:

“The biggest one is hierarchy. In hospitals and clinics there’s a very clear structure, you might be the most junior person on the team, and what the consultant says usually goes. In clinical product, that’s not the case. Sometimes the best ideas come from the person who’s just walked in the door with fresh eyes. You need to speak up and show your contribution.” Alice

Medicine can still default to deference, but clinical product requires the opposite: you need to contribute your perspective, challenge assumptions and definitely say when you think the team is heading in the wrong direction!

A clinical product team where everyone agrees too easily is unlikely to be examining the risks, trade-offs or alternatives closely enough.

👉 What to do about it:

• In your next meeting, ask the challenging question you’re tempted to swallow.

• Practise disagreeing constructively: “Here’s a risk I’d want us to consider” is more useful than staying silent.

• Stop waiting to be the most senior person in the room before you contribute.

2. What hiring managers are looking for

Lorenzo is currently hiring a Clinical Product Lead, so his screening criteria are worth writing down. Beyond the obvious clinical background relevant to the domain (for C the Signs, that means GPs, because they build for primary care), he described three qualities:

Pragmatism. Clinical product is about finding safer and better options that the team can work on, not simply identifying reasons why something can’t be done. The best clinical product people understand risk, explain it clearly and work with the team to find a proportionate way forward.

Commercial awareness. Clinical product shapes not just what gets built, but how it reaches people. You should understand who pays for the product, why customers choose it and what drives adoption.

Collaboration. Across product, engineering and commercial. Clinical product is a team sport, not single player mode.

And the bar is rising. Jamie has interviewed clinical product candidates for four years and has watched the pool mature:

“In the early days you saw a lot of clinicians who didn’t understand product and couldn’t express things in a customer-focused way. That’s changed a lot, people frame problems really well now, and the storytelling is getting better and better: using data, market understanding, telling a story about why something matters.” Jamie

What’s still rare, and therefore what sets you apart, is an experimentation mindset and genuine commercial understanding.

👉 What to do about it:

• Prepare a pragmatism story before your interview: a time you found a safe-enough path instead of blocking.

• Learn the commercial model of any product you talk about: who pays, how it’s adopted, why it wins. (Check out my commercial cheatsheet for CPMs here)

• Show an experimentation instinct: how you’d test a risky assumption cheaply, not just assess it.

3. Think like a product person

Building product nous is probably the number one priority for aspiring clinical product people. The panel’s recommended resources:

• 📖 Inspired by Marty Cagan. Lorenzo’s first product book and still his top recommendation.

• 📬 Lenny’s Newsletter. Jamie’s go-to. Deep, practical and absolutely tons of content to sink your teeth into!

• 🧑‍🏫 Consider getting a coach. Although not the cheapest option, it can super speed your learning:

“A few sessions with someone who’s lived it will crack your head open in a different way, very quickly, and show you different patterns of thinking.” Jamie

👉 What to do about it:

• Read Inspired this month. It’s the fastest way to learn the language of product.

• Subscribe to Lenny’s Newsletter and work backwards through the archive.

• Consider a few sessions with someone who’s done clinical product (not a generic careers coach)

4. Get your first bit of experience, even if small

The hardest part is the cold start: no clinical product experience, so no one hires you for clinical product. Alice’s advice is to create the experience for yourself:

“Try to get some type of advisory work, even if it’s voluntary, a few hours giving feedback, testing a feature. And think about all the health products you already use: your Apple Watch, your Oura ring, Apple Health. Reflect on the decisions behind those features, what you like and don’t like, and translate that into how you think.”
— Alice

You are more likely already surrounded by clinical products. Start critiquing them like you would if you were building them.

The good news is that these opportunities are becoming easier to find. Alice, who curates a weekly healthtech jobs roundup, has seen clinical product roles become far more clearly defined:

“We’re seeing more and more clinical product roles, head of clinical innovation, even chief clinical product officer. Before, those would have been called something else, or just ‘medical’. Now they’re better defined, with a clear scope and a clear value-add.” Alice

👉 What to do about it:

• Offer some voluntary advisory hours to a health tech startup: feedback, feature testing, a clinical sanity check.

• Do a product teardown of an app you use (Oura, Apple Health, a symptom checker).

• Follow Alice’s weekly roundups on Linkedin for current opportunities.

The takeaway

Breaking into clinical product is about showing how you can take your clinical knowledge and use it to understand users, identify risks early, work through constraints and help a multidisciplinary team make better product decisions. You do not need the title before you begin developing and demonstrating those skills.

If you’ve been thinking about making the move into clinical product management this might be your sign to take the leap.

Welcome to Subscriber Perks 🤩

I’m very excited to introduce subscriber perks: a growing library of resources for clinical product managers and people building clinical products or health tech companies.

What you’ll find:

• Deep dives on breaking into clinical product, embedding clinical safety into product decisions, building commercial knowledge and more

• Masterclasses on building safe clinical AI systems

• A guide to writing clinical product manager job descriptions that people can actually fill

• A database of European health tech VCs, grants and other funding opportunities

• A 30-minute clinical product advisory session with me for founding subscribers

👉 Check them out here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 047 of Clinical Product Thinking. This week: a recap on the third webinar in the series on clinical product management, co-hosted with Dani Brightman.

When we ran the first in-person event in January, clinical product was still hugely under-recognised in health tech. Six months on it feels like things are slowly starting to shift. Roles are being named properly. Job descriptions are looking less like a dog’s dinner. And the question is beginning to shift from “WTF even is clinical product?” to “how do we do clinical product well?” 🎉

This time we deliberately widened the panel. You told us you wanted more product and technology voices, so we brought them in:

• Dani Brightman — Clinical Director, Numan (pharmacist by training, now leading biggest clinical product team in UK)

• Lorenzo Espinoza — Director of Product, C the Signs (early-stage cancer detection; 13+ years in product, 7 in health tech)

• Jamie Webb Smith — Chief Product & Technology Officer, Numan (engineer by background, ~20 years in regulated fintech before health)

• Dr Alice McGee — Senior Medical Advisor, Flo Health (doctor by background, ex-founder of a health tech consultancy)

What came out was an interesting reflection: building products that touch health needs a new kind of product leader and the companies who don’t recognise this early will end up paying the price.

Let’s dive in.

First, what is clinical product?

The reason clinical product has lived in a grey area for so long is that nobody agreed where its edges were. Dr Alice gave her definition and it’s purposefully broad:

“To me, clinical product is any kind of product that relates to health in any way, from a full-blown medical device to anything that lets a user track their health data, symptoms or outcomes.”
— Dr Alice McGee, Senior Medical Advisor, Flo Health

That’s a deliberately wide net, and it should be. The tracker that looks harmless today is one intended-use statement away from being a regulated medical device tomorrow. If your definition of “clinical” only kicks in once something is obviously a device, you run a serious risk of getting caught out.

Why it’s having its moment

Jamie made an interesting parallel and it’s not one I’ve heard before:

“I came from fintech about 10 years ago. We saw a very similar thing, regulation started to move, it started to enable innovation, and there was a real push for the larger fintechs to break through. I think it’s a similar moment in health tech, and central to that innovation is clinical thinking.”
— Jamie, Chief Product & Technology Officer, Numan

That’s the same pattern: consumer health exploding, people taking their health into their own hands, and regulation maturing from a blocker into an enabler. Fintech went through exactly this a decade ago. The companies that broke through were the ones that took the regulated core seriously and built product muscle around it. In health tech, that regulated core is clinical thinking.

The cost of not having clinical product embedded

Lorenzo has led product teams both with and without embedded clinical product people, and he was blunt about the difference. When clinical is treated as an external advisor or part-time help, it breaks down in both halves of the product process: discovery and delivery:

“We spent weeks and months solving a problem that was not the right problem to solve.”
— Lorenzo, Director of Product, C the Signs

Without clinical thinking from the start, the team can spend months building a good solution to a problem that did not need solving.

Jamie gave a concrete example of this. A company was prescribing GLP-1 medication and, after a period, renewed those prescriptions. The team Jamie was working with built the renewal flow without enough clinical product input embedded:

“A few small but critical elements weren’t right, which meant we didn’t get the information we needed at the right time for clinicians to make a good prescribing decision. So they had to go back to the patient, which creates unnecessary cycles.”
— Jamie, Chief Product & Technology Officer, Numan

That small gap meant clinicians had to contact patients again before renewing the prescription. It added work for the clinical team, delayed treatment and gave patients another opportunity to drop out. A “small” clinical oversight showed up as a revenue problem. That’s the pattern many health tech founders underestimate.

Lorenzo also described what happens when the patient experience works but the clinician workflow does not.

“Time and again we see health tech innovation that basically dies at pilot stage, because the dynamics of patients, business and clinicians are not in sync.”
— Lorenzo, Director of Product, C the Signs

You can design a beautiful patient experience that becomes ugly the moment it hits the clinician’s workflow if the end-to-end experience hasn’t been mapped from each users view point.

👉 What to do about it:

• Embed clinical product discovery, not just delivery. Most wrong-problem failures happen before a line of code is written.

• Map the clinician journey alongside the patient journey. If a feature adds work for the person on the other side of the screen, you’ll find out at pilot, the worst possible time.

• Run an early “are we accidentally a medical device?” check on intended use and marketing language. It’s cheap now and painful later.

You do not have to be a clinician to develop clinical product judgement

One thing I loved was how quickly the “clinician or product person” framing fell apart. Dr Alice pointed out that some of her strongest colleagues at Flo have no clinical background at all:

“They’ve been in the space five, six, seven years. They’re not clinical by background, but they think in a clinical product mindset now, from absorbing years of feedback and user insight.”
— Dr Alice McGee, Senior Medical Advisor, Flo Health

And it runs both ways. Jamie described what happens when product, design, engineering and clinical genuinely mix:

“You start to get real magic. It’s almost like you create a common problem-solving brain between clinical, technology and design.”
— Jamie, Chief Product & Technology Officer, Numan

That’s what companies should aim for. Not a clinician who signs things off at the end, but a shared way of thinking that lets clinicians adopt product frameworks and lets product people develop clinical instincts. The best teams build a common brain.

Where clinical product should sit (Culture beats the org chart)

We spent a good chunk on reporting lines, and the honest answer is: several structures work.

• Lorenzo (C the Signs): clinical product reports into product; the Clinical Safety Officer sits outside product to keep an independent view.

• Dr Alice (Flo Health): a medical function reporting to the CEO (a Chief Medical Officer providing C-suite representation), with a named medical advisor assigned to each product initiative.

Both set ups can work, and as Jamie put it:

“What matters more than line management is alignment on how the company operates. Small cross-functional teams with a very clear problem to solve. If your operating model is broken, line management won’t fix it.”
— Jamie, Chief Product & Technology Officer, Numan

The real answer to “where should clinical product sit?” is inside the cross-functional team, from day zero. Reporting lines vary, but clinical product needs to be involved in the team’s day-to-day decisions rather than consulted shortly before release.

👉 What to do about it:

• Do not spend too long perfecting reporting lines while clinical product remains outside the day-to-day work. Put clinical expertise into the team that is investigating and solving the problem.

• Give clinical product a proper role in shaping the solution, rather than asking for approval after the main decisions have already been made.

• Whatever your structure, consider having an independent clinical safety view (a CSO outside the delivery team) so speed never quietly overrides safety.

The takeaway

Clinical product isn’t the bit where a clinician signs things off at the end. It’s a discipline. And it sits in one very particular place: between the patient, the business and the clinician.

Health tech is having its fintech moment. The teams that win it won’t be the fastest. They’ll be the ones who got clinical thinking into the room early, and built the common brain as a strategic advantage.

Introducing Subscriber Perks 🤩

I’m very excited to introduce subscriber perks: a growing library of resources for clinical product managers and people building clinical products or health tech companies.

What you’ll find:

• Deep dives on breaking into clinical product, embedding clinical safety into product decisions, building commercial knowledge and more

• Masterclasses on building safe clinical AI systems

• A guide to writing clinical product manager job descriptions that people can actually fill

• A database of European health tech VCs, grants and other funding opportunities

• A 30-minute clinical product advisory session with me for founding subscribers

👉 Check them out here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 046 of Clinical Product Thinking. This week: highlights from our Clinical AI panel, on where patient-facing AI fails when the model looks fine.

A few weeks ago I co-hosted a panel on building safe clinical AI systems with Dani.

If you only have a few minutes, here’s your tl;dr. If you want to dive deeper, the full recording is below.

On the panel:

• Dani Brightman — Clinical Director, Numan

• Dr Lucinda Scharff — Product Manager, Counsel Health (formerly Google Health / DeepMind)

• Dr Paul Sacher — Founder & CEO, Sacher AI

Let’s dive in.

A bot responded correctly but still failed clinically.

Dani gave an example from Numan’s customer care side. They built a bot to filter inbound patient requests and signpost people to the right clinical form. On paper, it worked well.

In practice, it missed what a human would have caught:

“Even though the bot was handling the inbound request correctly and providing the right signposting, the clinical team noticed that if the patient was also providing context, like they want to stop medication because ‘I’m having X side effect’, what should have happened is they escalate that ticket to a clinician to review it. The bot missed this important step”
— Dani Brightman, Numan

The bot completed the task but missed the clinical significance of the patient’s context.

A clinically correct model is not the same as better care

Dr Lucinda talked through a recent paper on using AI to detect cancer on chest X-rays. AI can flag lung cancer risk on a film, but the patient still needs someone to review the result, contact them, arrange the CT and provide follow-up care.

Faster diagnosis on an X-ray doesn’t automatically result in better patient outcomes if downstream dependencies haven’t been accounted for.

A useful signal with no reliable route to action means the product isn’t finished.

Behavioural harms are the hidden risk

Behavioural risk can create or amplify clinical risk, and they’re much harder to spot: sycophancy, erosion of patient agency and failure to escalate when a patient says they’re anxious, low or questioning whether to continue treatment.

Systems designed for engagement may keep chatting when the safe move is to refer the patient to a clinician.

“If you think about the scale of behavioural interventions that are happening through patient-facing AI, you can think of the whole thing as one massive behavioural intervention… that no one’s really monitoring.”
— Dr Paul Sacher, Sacher AI

Accuracy isn’t enough if the interaction nudges the patient towards an unsafe next step.

Safety is a team sport, not a clinician sign-off

A key point of the evening was about culture. Clinicians are essential to clinical AI teams, but safety can’t belong to them alone.

If the clinician is the person who turns up at the end to say yes or no, you get slow teams, frustrated clinicians, and products that treat clinical risk as a final check rather than a design concern.

The best teams build shared safety instincts early, in the model, the interface, the pathway, the escalation route, and the assumptions about how patients actually behave.

The takeaways

The evening’s themes:

1. Correct routing isn’t enough. The system must also recognise when additional context changes the appropriate next clinical step.

2. Models need pathways. A right answer still needs a route to action.

3. Measure behaviour, not just accuracy. The harms that scale are often patterns, not single unsafe statements.

4. Safety is designed in. Not bolted on at the end by a clinician.

🎥 Watch the full panel below. Dani, Dr Lucinda, Dr Paul and I go much deeper on clinical safety in AI products, friction as a safety feature, automation bias and what teams should measure before and after deployment.

For the full write-up with practical steps, read issue 041: How to Build Safe Clinical AI Systems.

Hiring spotlight 🚀

🇬🇧 Doccla are hiring a Head of Clinical Innovation. The role will be crucial in translating Doccla's clinical model into safe, evidence-based, and commercially impactful products and pathways. 👉 Apply here.

Join the next clinical product panel 🎤

📆 14th July, 7pm, online

The next clinical product panel is on 14th July and we’ll be covering the “clinical product gap” or why healthtech needs a new kind of product leader. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 045 of Clinical Product Thinking. This week: highlights from our first panel, on where clinical judgement really belongs in a healthtech org.

A few weeks ago I co-hosted the first panel in a 3-part series organised with Dani. The full video is below. But if you only have a few minutes, your tl;dr is below:

I was joined by three leaders building clinical product at scale:

• Dr Anushka Mehrotra — Chief Medical Officer, Doctor Care Anywhere

• Dr Yath Prem — Medical Advisor, Flo Health

• Dani Brightman — Clinical Director, Numan

Three very different companies yet one shared conviction: clinical product isn’t a department you consult. It’s a discipline you build in.

Let’s dive in.

1. Clinical risk hides in the places non-clinicians don’t think to look

The panel circled back to this a number of times. The dangerous decisions can be the small product choices, the colour ranges on a blood pressure reading, a portal update that looks unchanged to the clinician on the other side, that quietly carry real clinical weight.

Anushka gave an example from Doctor Care Anywhere: an update had been made to the patient facing app, but the clinician interface looked the same. Everything was actually safe, but because it didn’t look like the clinicians expected they were worried the update hadn’t been made. The lesson: end-to-end mapping has to include every user’s perspective, not just the patient’s.

Dani put the principle plainly:

“In health tech, a product decision is a clinical decision when that product is shaping patient care.”
— Danielle Brightman, Numan

The trap: clinical risk doesn’t announce itself.

2. Involve safety early, or pay for it later

There’s a myth that clinical involvement slows teams down. The panel didn’t agree. Bring clinical judgement in late and you force everyone back to the drawing board, which is slower, not faster.

Dani built Numan’s clinical product function from zero to the largest team in the UK. Her take on accountability:

“If you can’t veto, then you can’t have the accountability, because that’s completely unfair. For you to hold the real weight of clinical safety, you also need to be able to say no at times, and know you’ve got backing for that.”
— Danielle Brightman, Numan

If you’re going to hold someone responsible for safety, they need the power to block. Anything less is a set-up.

3. Integrated, not consulted

Yath made the case for what 'integrated' actually means in practice. At Flo, medical advisors sit inside cross-functional squads with a dedicated PM, design, content and engineering, meeting almost daily.

“For Flo, it’s never a question of ‘should we listen to the medical advisor?’ It’s how do we work with the advisor from the beginning. It’s never a case of ‘shall we check with the medical advisor if this is okay’, the medical advisor is always in the conversation.”
— Dr Yath Prem, Flo Health

That’s the difference between clinical input as a gate and clinical input as a default.

4. How you communicate risk matters as much as spotting it

A practical takeaway came from Dani on communicating risk so that people listen:

“We’d say ‘this process means this patient might get a double dose,’ and leave it there, because we all knew the implications. But a non-clinical person didn’t really understand how bad that was. You need to take it further: this patient comes to moderate harm, hospitalisation, that’s irreversible. Then they’re like, ‘okay, that actually sounds quite serious.’”
— Dani Brightman, Numan

And pair it with the commercial reality: the litigation exposure, the payout costs, the drop-off. Do the research first so the risk is easy to prioritise.

When the answer is “too risky,” don’t stop at no. Anushka’s rule is pragmatic option building:

“There might be an option that’s too risky, but what’s the alternative? Is there something else that can be done that isn’t as risky, that helps to control some of that risk but still partially achieves what’s trying to be done?”
— Dr Anushka Mehrotra, Doctor Care Anywhere

The takeaways

The evening’s themes:

1. Integrated, not consulted. This phrase came up again and again.

2. The constant balance of clinical and commercial. Healthcare companies are still businesses. Both pillars have to be right.

3. Option building, not just blocking. “No” is rarely the end of the conversation.

4. A shared language. Different disciplines working together in high-risk, fast-paced environments all benefit when we speak each other’s vernacular.

Which is really just one idea, said a few different ways:

Clinical review is not a sign-off step. It is a design input.

🎥 Watch the full panel below. Anushka, Yath, Dani and I go far deeper than I’ve captured here, on reporting lines, scaling safely, and what founders get wrong before they scale.

Hiring spotlight 🚀

🇬🇧 Doccla are hiring a Head of Clinical Innovation. The role will be crucial in translating Doccla's clinical model into safe, evidence-based, and commercially impactful products and pathways. 👉 Apply here.

Join the next clinical product panel 🎤

📆 14th July, 7pm, online

The next clinical product panel is on 14th July and we’ll be covering the “clinical product gap” or why healthtech needs a new kind of product leader. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 044 of Clinical Product Thinking. This week, we’re talking about the workflow we think we're designing for and the work that clinicians actually do.

Many clinical products are designed around the happy path: the patient follows the expected journey, the clinician has the information they need, the technology behaves as intended, and each step leads nicely to the next. It all looks perfectly represented on your Miro process map.

This is because most product discovery starts in the same way. We interview users, run workshops, map workflows and document requirements. These activities are useful and necessary. They help us understand how a service is supposed to operate.

The difficulty is that healthcare rarely behaves exactly as it is described.

Patients do not always follow the expected journey. Clinicians are interrupted, multi-tasking, overworked or all three. Information is missing. The computer is too slow to load.

By the time a workflow reaches a whiteboard, many of those details have disappeared.

Healthcare products are often designed around the happy path because the happy path is easier to see. It is neat, predictable and easy to describe.

Clinical work rarely is.

Work as imagined vs work as done

One of the most useful concepts from safety science is the distinction between work as imagined and work as done.

Work as imagined is how we think a process happens.

Work as done is what people actually do in order to get the job done.

The difference between those two things is often where the most valuable product insights are hiding.

A workflow diagram might show a clinician reviewing information in a single system before making a decision. In practice, they may be checking multiple systems, referring to notes written earlier in the day, asking a colleague for context and mentally prioritising several competing tasks at the same time.

The process on paper is not necessarily wrong. It is simply an incomplete picture.

The 2am test

During a recent panel discussion, Benedikt von Thüngen, CEO of Sanome, spoke about the importance of getting all of his staff to shadow clinicians on a night shift. Not just product managers, engineers as well.

Some parts of clinical work can only be understood by seeing them first-hand.

That image stayed with me because it provides a useful test for any clinical product.

How would this work at 2am?

How would it work when the clinician is tired, the computer is slow, the patient is deteriorating, and the next safe action still needs to be clear?

Pay attention to adaptations

Product teams sometimes describe these behaviours as workarounds, but I think it is more useful to think of them as adaptations.

Healthcare workers adapt systems constantly. They create routines, shortcuts and informal processes that allow them to continue delivering care in environments that are often busy and unpredictable.

The adaptation itself is not usually the interesting part. The reason it emerged is.

If somebody leaves the product to find information elsewhere, that tells you something about the product. If staff maintain a spreadsheet alongside the official system, that tells you something about the workflow. If people repeatedly ask colleagues for information that should be easy to access, that tells you something about how knowledge is organised.

The goal is not to eliminate every adaptation. Many exist for good reasons.

The goal is to understand what those adaptations are trying to achieve.

Closing thought

Clinical products are often evaluated against the happy path. Clinical work is delivered in the real world.

The gap between those two things is where many usability issues, workflow problems and safety risks emerge.

Understanding that gap is one of the core responsibilities of clinical product teams because products that fit naturally into real clinical work are far more likely to be adopted, trusted and used safely.

Hiring spotlight 🚀

🇬🇧 Manual are hiring a Medical Director. I caught up with Earim, CMO, on the role and this an incredible opportunity for a visionary clinician. 👉 Apply here.

🇬🇧 Alertive are hiring a Clinical Product Director. I chatted with Kevin about this position and they are looking for a product leader to help turn its hospital communications and alerting infrastructure into safe, specialist clinical workflows as customer demand grows. 👉 Apply here.

🇺🇸 Hinge Health are hiring a Clinical Product Consultant to help them transform MSK care. 👉 Apply here.

Join the next clinical product panel 🎤

📆 14th July, 7pm, online

The next clinical product panel is on 14th July and we’ll be covering the “clinical product gap” or why healthtech needs a new kind of product leader. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 043 of Clinical Product Thinking. This week, we’re talking about scaling, safety and how to spot risks before they become incidents.

Last week I hosted an event at HLTH Europe on how to safely scale digital health. One of the big takeaways from the panel; Dani Brightman, Numan, Benedikt von Thüngen, Sanome and Dr Elle Clarke, Heim was this:

Scale doesn’t create risk. it reveals where it was already hiding.

Or more precisely: scale exposes weak points that were being silently patched by people, process and clinical judgement.

Scale doesn’t create risk. It reveals it.

As products scale, you’re not just handling more patients or more data.

You’re stress-testing whether your system actually works or whether it only looked like it worked because people were quietly holding it together.

At small scale, that gap is invisible.

You find:

• clinicians making judgement calls because SOPs aren’t fully clear

• ops teams fixing broken workflows in real time

• support repeatedly explaining things the product should have made obvious

• manual reviews catching errors the system itself is generating

This is the “invisible layer” of early success.

The hidden work behind early success

Early products almost always depend on invisible labour. And that’s not a mistake, it’s how to learn incredibly quickly.

But there is a catch.

At small scale, teams sit close to users. They absorb problems quickly. That can create the illusion that the product is more robust than it actually is.

Because:

• A workaround can solve a problem without solving the underlying cause.

• A manual review process can catch errors without improving the system that creates them.

• A person stepping in can protect patients without revealing that the product needs redesign.

The risk is not that teams do this.

The risk is not being aware that these workarounds exists.

Scaling safely requires changing the way you think about problems

In early-stage clinical products, clinicians are often operating at maximum capacity.

That makes it understandable that workarounds emerge without much time being spent stepping back to redesign the system itself.

But there’s an important shift that happens as you move towards scale:

Investigating problems properly and building better solutions takes time. It’s an upfront investment for a later payoff and it rarely happens by default.

That’s why teams either need to be:

• explicitly incentivised and supported to do this work, or

• structured with a dedicated function (like clinical product) whose job is to translate clinical friction into system improvements

Otherwise, the same people who are holding the system together day-to-day never get the space to redesign it.

Scaling safely means redesigning where work happens

When something breaks at scale, the instinct is often to add process.

But what actually matters is shifting where work sits:

From:

• human judgement under pressure

• manual interpretation

• repeated explanations

To:

• product design

• system logic

• clear escalation paths

• monitoring and feedback loops

This is the shift that determines whether scaling creates fragility or removes it.

Finding the risks before you scale

If you’re on the precipice of scale some useful exercises for you:

1. Find the workarounds

Ask your users:

• What do you check manually?

• What questions do people repeatedly ask?

• Where do you use spreadsheets, notes or other tools?

• What do you do outside the product to complete this task?

• What problems do you message team members about repeatedly to try to solve?

These workarounds are evidence of where the system can be optimised.

2. Run a 10x stress test

Take one important workflow and ask:

“What breaks if volume increases 10x tomorrow?”

Look at:

• response times

• manual reviews

• escalation processes

• points where a person has to intervene

• decisions that depend on individual judgement

Most fragility shows up here immediately.

3. Watch the real workflow, not ideal one

Spend time observing how people use the product in their normal environment.

Look for:

• where people pause

• where they create shortcuts

• where they ask colleagues for help

• where they interpret information differently

• where they leave the product to do something else

These often show where the product needs to change.

4. Track emerging operational risk explicitly

Teams usually track features they want to build.

They should also track areas that may become difficult as usage increases.

Examples:

• manual processes

• unclear escalation routes

• customer-specific variations

• inconsistent patient journeys

• data that is difficult to monitor

• features that create complexity without improving care

These are the areas most likely to become problems later.

A final reframing

Scaling a clinical product is often framed as an engineering or regulatory challenge.

But the panel made a different point: It’s actually a visibility problem.

At small scale, you can miss what’s broken because people are quietly fixing it.

At large scale, you can’t.

Which means the real work of scaling safely is not just building more product.

It’s removing dependence on invisible heroics and replacing them with systems that behave the same way at 10 patients as they do at 10,000.

Hiring spotlight 🚀

🇬🇧 Manual are hiring a Medical Director. I caught up with Earim, CMO, on the role and this an incredible opportunity for a visionary clinician. 👉 Apply here.

🇺🇸 Hinge Health are hiring a Clinical Product Consultant to help them transform MSK care. 👉 Apply here.

HLTH you were a vibe 😆

So great to see many of you in Amsterdam.

Join the next clinical product panel 🎤

📆 14th July, 7pm, online

The next clinical product panel is on 14th July and we’ll be covering the “clinical product gap” or why healthtech needs a new kind of product leader. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 042 of Clinical Product Thinking. This week, we’re talking about what it takes to start and scale tech-enabled clinical services.

One of the most useful conversations I’ve had recently was with a group of people building and operating tech-enabled clinical services at scale.

We asked the difficult questions:

How do you actually build a clinical service that scales? How do you organise teams around care delivery? When should you buy software, and when should you build it? How do you stop complexity from multiplying across countries, pathways and customer segments?

One takeaway stood out:

In tech-enabled care, the pathway is the product. The software exists to support it.

In a tech-enabled clinical service, software is only one layer of the product. The actual product is the end-to-end service: the patient journey, the clinical model, the operational workflow, the escalation process, the documentation, the workforce, the governance, the analytics, the outcomes, and the way all of those things interact in the real world.

This matters for clinical product managers as it becomes part of their remit.

Clinical product is pathway ownership

At one company, the clinical product function is called “Clinical Pathway”, and the people in that function are Clinical Pathway Leads.

Clinical Pathway Leads sit within a squad alongside product and engineering, with ultimate accountability for how the pathway works in practice. That includes its clinical safety, operational flow, usability, measurement, performance and outcomes.

This is different from the model I often see in healthtech, where clinical, product, engineering and operations sit in separate functional silos, then try to coordinate through meetings, tickets and escalation channels.

But clinical services do not break neatly along functional lines.

A patient does not experience a “product issue”, then an “ops issue”, then a “clinical issue”. They experience one journey. A clinician does not experience workflow, documentation, risk, software usability and operational capacity as separate things. They experience one working environment.

That is why the squad model matters.

Instead of organising around functions, you organise around pathways. The point is not simply to create a multidisciplinary team, but to create a unit of accountability that maps to the thing you are actually trying to improve.

The squad is accountable for pathway KPIs, not just software delivery.

That shift changes what gets prioritised, which trade-offs are visible, who has authority and what counts as progress. A feature can ship while the pathway remains broken. A dashboard can make a service more measurable without making it more effective. An automation can reduce manual work while introducing new clinical risk.

Clinical product needs a broader definition of “product”, because in healthcare the thing being designed is the care pathway, not just the software that supports it.

Do not build the pathway and the software at the same time

Another point that stayed with me was the danger of trying to innovate on too many layers at once.

When you start a new tech-enabled clinical service, there is a temptation to build the pathway and the software simultaneously. The founder may have a clear view of how the service should work, and an equally clear view of the technology needed to deliver it.

But building a new clinical service is already a huge act of discovery. You are designing a care model, testing a pathway, building workflows, creating governance, training a workforce, learning what patients and clinicians need, and working out where risk appears.

If you also try to build bespoke software at the same time, you are stacking innovation on innovation before you properly understand the shape of the service.

The pragmatic advice was: buy software off the shelf first, learn the pathway, then rebuild later when the operational pain becomes impossible to ignore.

Off-the-shelf tools will not be perfect. They may feel clunky, require workarounds, or fail to map neatly to your pathway. But early on, that can be useful. They give you a learning scaffold before you hard-code your assumptions into a bespoke system.

You should only switch or rebuild when the software becomes too expensive, too generic, unable to support the workflow, or when your documentation and operational needs become specific to your clinical vertical.

The sequence matters. The pathway should teach you what the software needs to become.

Standardise by default, localise by exception

The next principle was standardisation.

When companies move into complex clinical systems, especially across multiple countries, complexity multiplies quickly. Each market brings its own regulations, clinical norms, prescribing rules, data requirements, operational constraints and patient expectations.

If every country, pathway or customer implementation starts creating its own way of working, the company can quickly end up with several different services wearing the same brand.

That might feel responsive in the short term. Over time, it becomes clinical product debt.

Companies die by complexity.

Was a phrase used on the night. The principle that emerged instead was: standardise everything you can, and only deviate where you absolutely must.

That might mean central operations, shared tools, common documentation structures, consistent governance processes, agreed escalation routes, a shared analytics framework and aligned pathway KPIs. Country-specific variation should exist, but it should be deliberate and driven by regulation, clinical guidance, reimbursement, prescribing rules, data/privacy requirements or genuine local context.

The default should be a shared operating model with justified exceptions.

This matters because every unnecessary variation creates future drag. It becomes harder to train teams, measure performance, compare outcomes, maintain quality, govern safety and build software that scales.

Standardisation can sound bureaucratic, especially to early-stage teams that prize speed and flexibility. But in clinical services, it is part of how you preserve safety and quality as you scale.

The more complex the system, the more disciplined the operating model needs to be.

Workforce design is part of the product

The final theme was workforce complexity.

In a clinical service, not all work is interchangeable. Some tasks are straightforward and repeatable. Others require specific skills, experience, qualifications or training.

That creates an important operational question: how quickly can you train the people you have to handle more complex work safely and effectively?

In clinical product, that is also a product question.

Workforce design affects what the pathway can do. It shapes cost, quality, safety, throughput, patient experience, clinician experience and scale. If a pathway depends on a small number of highly skilled people, that constraint is part of the product architecture. If the software assumes work can be standardised but the clinical reality requires nuance, that mismatch will eventually show up.

Often, the most important product decisions in clinical services are not obviously “product” decisions. They are decisions about who does the work, what they are trained to do, what they are allowed to decide, what the software makes easy, what it prevents, what it escalates and what it measures.

Workforce capability is not a back-office detail. It is also part of the product.

The real product

If the pathway is the product, clinical product has to care about how the service works in practice.

That means looking beyond individual features to how patients move through the pathway, how clinicians make decisions, how risk is managed, how work is distributed, how outcomes are measured, and whether the operating model holds up as the company scales.

The software matters enormously. It can make the pathway safer, more consistent, more measurable and easier to scale.

But the companies that win in tech-enabled care will not be the ones that only build better software. They will be the ones that build better systems of care.

Say hi at HLTH! 👋

📆 15th June, 1 - 2pm, Room G108

If you’re at HLTH tomorrow join me, Danielle Brightman, Benedikt von Thüngen and Dr Elle Clarke for a discussion on what it takes to build and scale digital health. Please do come and say hello afterwards!

Join the next clinical product panel 🎤

📆 14th July, 7pm, online

The next clinical product panel is on 14th July and we’ll be covering the “clinical product gap” or why healthtech needs a new kind of product leader. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

Welcome, friends, this is issue No. 041 of Clinical Product Thinking. This week we're talking about what it takes to build clinical AI that patients, clinicians and health systems can trust.

You can have a clinically accurate model and still make care worse. That gap, between a model that works and a product that helps, ran through this week’s panel.

On Tuesday, Dani and I ran the second event in our panel series, this time on building safe, effective clinical AI. We covered where AI belongs in care pathways and where it fails, how to think about clinical and behavioural safety, what “human in the loop” actually solves, and why so much of the hard work happens outside the model.

Thanks to Dani for being my fabulous co-host and Dr Lucinda and Dr Paul for sharing their incredible wisdom.

Here are the seven things I’m still thinking about. 👇

1. Stop looking for places to put AI

The panel made it clear, the wrong starting question is “Where can we use AI?” It may sound reasonable, but it is wrong. The better question is “Where is the pathway currently failing?”

That might be clinical, operational, the patient experience, safety or capacity. Patients might be dropping off before finishing onboarding. Clinicians may be spending too long reviewing low-risk cases. High-risk patients might not be escalated early enough. Useful information may sit buried in free text. Or patients are asking the same question again and again because the product never made the next step clear.

Once you understand the problem, you can choose the intervention. Sometimes that’s AI. Often it’s a rule, better content, a workflow change, or a person. Don’t become a solution looking for a problem. It usually doesn’t end well!

👉 Practical step: Use this in product review

Ask the team to describe the problem without mentioning AI. “We need an AI coach” is weak. “Patients drop off after side effects because they don’t know what’s normal or when to contact us” is the actual problem. If you can’t describe the problem without naming the solution, the use case needs more work.

2. The best first use cases are usually unglamorous

As the late Grandad Rix used to say, where there’s muck there’s brass. Translation: look at the unsexy use cases first, as there’s often a lot of value to be created there.

Think about what clinicians spend time on that they were never really trained to do, or don't enjoy doing. Not the complex judgement, the high-risk decision or the sensitive conversation, but the admin around all of it: the repeated explanations, the summarising, the routing, the chasing, the hunt for missing information. The low-risk, repetitive work that keeps clinicians from spending time where their judgement and skills matter the most.

That admin is where AI helps first. The tasks are bounded, easy to check, and don’t carry the clinical decision themselves.

👉 Practical step: Start where mistakes are cheap

Pick tasks where, if the AI gets it wrong, someone notices quickly and little harm is done. A drafted message a clinician checks before sending is safe to start with. An AI that closes patient cases on its own is not.

3. A useful output still needs a pathway that can act on it

Dr Lucinda shared an example that stuck with me: an AI system that reviews chest X-rays to flag people who may be at risk of lung cancer. The model can find the signal, but the patient still needs the next step. Someone has to review the result, contact the patient, arrange the CT, track whether it happened and act if it didn’t.

A clinically correct model is not the same as better care.

The LungIMPACT trial (Nature Medicine, 2026) showed this at scale. Across five NHS trusts and more than 90,000 chest X-rays, using AI to flag and prioritise abnormal films cut reporting time from 47 hours to 34, but made no difference to time to CT, diagnosis, treatment or stage at diagnosis. The faster report had nowhere to go, because the CT slots and capacity around it never changed. The authors concluded that prioritisation alone shouldn’t be deployed without redesigning the pathway around it.

It’s like adding a spiral staircase without considering you live in a bungalow. It might look nice until you realise there’s nowhere for it to go.

👉 Practical step: Map the next five steps before shipping

Who sees the output, what are they meant to do, how fast, what happens if they do nothing, and who checks it happened? A prediction needs an owner, a flag needs a route, a risk signal needs a response time. A useful signal with no reliable route to action means the product isn’t finished.

4. Human in the loop is not enough

“Human in the loop” sounds reassuring until you look at how the loop works and consider how humans typically behave over time. A clinician reviewing AI output doesn’t automatically make a system safe.

People are influenced by what they see first. Show the AI’s answer before the clinician has formed their own view, and that view is already shaped. People also get used to systems that are usually right; after the model is correct a few hundred times, the rare mistake is easy to miss. And the workflow itself pushes behaviour. If approving the output is easy but questioning it is slow or awkward, the product quietly nudges the clinician towards agreement. That isn’t meaningful oversight.

A poor review workflow is a big approve button, a buried escalation route, and no record of why anyone agreed. A good one gives the reviewer time, context, visible uncertainty, and a clear (and easy) way to disagree.

👉 Practical step: Review the review (meta)

Ask yourself when does the clinician see the output, and what sits alongside it? How is uncertainty shown? How easy is it to disagree, and what happens when they do? How much time do they have, and what’s the default action? How is review quality monitored over time?

A human in the loop only helps if the product gives that human the space, context and authority to think.

5. Sometimes friction is a safety feature

Product teams are trained to strip out friction: fewer clicks, faster journeys, more automation. In much of healthcare that’s also true, needless friction wastes time and frustrates everyone.

But some friction can be a feature. A pause before a high-risk action, a required review before a medication change, a prompt to consider uncertainty, a design that stops someone clicking through too fast. The fastest version of a clinical workflow isn’t always the safest. The real question is what the friction is doing. Is it waste, bureaucracy, or a patch over a badly designed system? Or is it buying time for judgement, escalation or reflection? Knowing which parts of a pathway to speed up and which to slow down is part of the craft.

👉 Practical step: cut the bad friction, keep the good

Not all friction does the same job. Strip out what wastes people’s time, and keep what protects judgement.

Cut: re-entering information you already have, clicking through duplicate forms, escalation routes buried in a long workflow.

Keep: a pause before a high-risk action, a required review before a medication change, a prompt for a reason when overriding the AI.

6. Behavioural safety sits alongside clinical safety

Dr Paul is an expert in behavioural safety: how AI shapes the way people think, feel and act over repeated interactions. It is a crucial consideration for patient-facing AI.

A system doesn’t have to give obviously unsafe advice to cause harm. It can reassure a patient when escalation would be safer, encourage dependency, reinforce problematic thinking, advise without enough context or sound more certain than it has any right to. So patient-facing AI has to be evaluated for the behaviour it creates, not only its accuracy. Does the patient understand what to do next? Does the interaction support their own judgement and communicate uncertainty clearly? Does the system know when to stop and hand over, and make escalation obvious? Does it stay in its lane?

Be careful about product KPIs for utilisation. Long conversations don’t automatically result in good outcomes in healthcare. Sometimes they means the AI supported the patient well. Sometimes it means it failed to move them towards the right next step. Behavioural safety is an integral part of clinical safety. It’s time we started treating it as such.

👉 Practical step: One question for patient-facing AI

What behaviour is this interaction making more likely?

Is it making a patient more or less likely to seek further help, to trust their own judgement, to follow their care plan, to depend on the AI or to know the next safe action? Accuracy isn’t enough if the interaction nudges the patient towards an unsafe next step.

7. Safety can’t sit only with the clinician

My favourite point of the evening was about culture. Clinicians are essential to clinical AI teams, but safety can’t belong to them alone. If the clinician is the person who turns up at the end to say yes or no, you get slow teams, frustrated clinicians, and products that treat clinical risk as a final check rather than a design concern.

The best teams build shared safety instincts. Designers see how a layout might change clinical behaviour. Engineers understand why an edge case isn’t just an edge case, it’s a future clinical incident waiting to happen. Product managers spot when a shortcut introduces risk. Clinicians translate clinical nuance into product decisions early enough to shape the work. Safety becomes part of how the team thinks rather than something bolted on at the end.

This matters more with AI because the risk is rarely in one place. It can sit in the model, the interface, the pathway, the escalation route, the monitoring, the review process, or the team’s assumptions about how patients behave. It has to be designed in from the start.

👉 Practical step: Get clear on how safety shows up across the team

Clinician → Defines clinical risk, escalation thresholds and edge cases

Product manager → Works with clinician to turn risk into requirements, workflows and trade-offs

Designer → Shapes how users notice, interpret and act on risk

Engineer → Builds safe defaults, fallbacks, audit trails, monitoring

Operations → Makes escalation routes and response times work in practice

What I’m taking from this

AI is becoming part of more care pathways. Some of that will be genuinely useful, some unnecessary, and some will create risk in ways that aren’t obvious at first. The clinical product skill is telling the difference. Before adding AI, ask what problem you’re solving and who has it, what risk it creates, what happens after the output, who’s accountable when something changes, and how you’ll know if care actually improved.

That’s where safer clinical AI starts.

Join the next clinical product panel 🎤

The next clinical product panel on the clinical product gap and why healthtech needs a new kind of product leader 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

Made with 💜 for better, safer HealthTech.

For Paid Subscribers 🤩

The full 75-minute panel with Dani, Dr Lucinda and Dr Paul is below. 👇

We went deeper on all things clinical safety in AI, behavioural safety and system design.

Read more

Welcome, friends, this is issue No. 040 of Clinical Product Thinking. This week we're talking about why clinical product is often misunderstood and one of the most important skills to focus on as a clinical product manager.

Most people assume the value of having a clinician in a product team is that they bring medical knowledge. They can explain the pathway, review the copy, sense-check the onboarding questions, spot when something looks clinically inappropriate, and stop the team from shipping anything obviously unsafe. That is useful, for sure. But there is a lot more to it.

Clinical knowledge on its own doesn’t necessarily create a better product. If it did, every clinician-led healthtech company would be brilliant. And every product designed by a medical team would be safe, usable and easy to trust. We know that isn’t always how it works.

A clinician can describe how something should happen in clinical practice, but that does not automatically tell a team what to build. It does not tell a designer what to put on the screen, an engineer how the logic should work, an operations team what needs to happen behind the scenes, or a commercial team how to explain the product without overstating what it can do.

The skill that closes that gap is translation. Clinical expertise has to become product decisions and that translation is a discrete, learnable craft, not a side effect of having a doctor in the room.

The five translations clinical product people make

I think about five key points where clinical thinking becomes a product decision:

Clinical nuance → product requirement

A clinician will often flag a clinical nuance based on their experience that the team hasn’t seen. “Patients won’t read it that way.” “That’s not how this usually presents.” “A clinician would never phrase it like that.” The observation is helpful but not yet useful to a product team.

The translation is to ask what that insight means for the product. Does the copy change? The logic? Do we need an extra question, a clearer next action, a different default, or a test of whether patients actually understand what’s being asked before we ship it?

Clinical nuance becomes useful when the team knows what to change because of it.

Safety risk → design constraint

This is the one teams assume they’ve got: spot a risk, mitigate it. The real translation is subtler.

A product team’s instinct is to treat every input as a request, something to size, prioritise and trade off against speed and experience. A genuine safety risk doesn’t behave like that. It’s a constraint: a line the product isn’t allowed to cross, however inconvenient. The clinician’s job is to translate “this could harm someone” into exactly that: a boundary, stated precisely enough that an engineer and a designer know what the product must never do, or must always do.

If certain symptoms mean a patient shouldn’t continue down a digital pathway, that’s the boundary and the questions, the branching logic and the escalation route get built inside it, rather than bolted on once the product is already baked. Leave the risk as a worry, or a line in the hazard log, and it resurfaces late, as a blocker.

So the question isn’t “have we noted the risk?” It’s “have we turned it into a constraint the team is designing within?”

Potential harm → prioritisation

Product teams prioritise by reach, effort and value or how many users, how much work, how much upside. Clinical work has an axis the usual frameworks miss: how badly it could go wrong.

The clinician’s job here is to translate consequence into priority. Take a duplicate-submission bug that lets a patient order the same prescription twice. On a backlog it looks like a rare edge case and sinks down the list. The translation is: a double dose of a blood thinner risks a serious bleed, that harm is severe even when the event is unlikely, and rare-but-severe should beat frequent-but-trivial. Said that way, the work moves up the queue.

Non-clinical teams may not intuitively know how to weigh potential harm against other product priorities. Make it easy for them to understand.

Messy pathway → usable journey

Clinical pathways often look much cleaner on paper than they feel in real life. A clinician knows where they would pause, clarify, reassure, redirect or ask a follow-up question. A product does not know any of that unless someone translates it.

This is where clinical product people shine. They can look at a pathway and ask: what is the purpose of this step, what could the patient misunderstand, what would a clinician normally explain out loud, what can safely be removed, and where does a human need to come back in?

Without that translation, teams end up turning clinical pathways into long digital questionnaires. The pathway may be clinically correct, but the product still feels confusing, heavy or unsafe to use.

Clinical outcome → commercial impact

Clinical teams often talk about outcomes in clinical language: reduced symptoms, fewer complications, better adherence, earlier detection, improved safety, more appropriate escalation. Commercial teams often talk about growth, retention, conversion, cost, differentiation and buyer confidence.

Clinical product people need to connect those two conversations.

If a product reduces unnecessary appointments, that is not only a clinical workflow improvement. It may also reduce cost to serve. If a pathway helps patients get the right care earlier, that is not only better care. It may also improve activation, trust and retention. If a product is safer and clearer, that is not only good governance. It may make the product easier to sell to cautious buyers.

This does not mean dressing clinical outcomes up in commercial language to make them sound more exciting. It means helping the business understand why clinical quality is part of the product’s value.

A useful question is: if this clinical outcome improves, what changes for the patient, the clinician, the buyer or the business?

The actual job

None of this replaces clinical knowledge. It is what makes clinical knowledge usable.

The clinical product manager’s job is not just to bring a clinical view into the room. It is to help the team turn clinical understanding into decisions they can act on.

Clinical nuance becomes a requirement. Safety risk becomes a constraint. Potential harm changes priority. A messy pathway becomes a usable journey. Clinical outcomes become part of the product’s value.

That is the translation work.

And it is one of the skills that separates clinical input from clinical product.

Join the next clinical product panel 🎤

The next clinical product panel on how to build safe, effective clinical AI is on 2nd June. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on Linkedin.

Made with 💜 for better, safer HealthTech.

For Paid Subscribers Only 🤩

The full 60-minute panel with Dr Anushka Mehrotra and Dr Yath Prem is below. We went deeper on metrics, CSO responsibilities, scaling clinical product as a function.

Read more

Welcome, friends, this is issue No. 039 of Clinical Product Thinking. This week we're talking about clinical product, and why most health tech teams are still making clinical decisions by accident.

A couple of weeks ago Dani Brightman and I hosted a panel with Dr Anushka Mehrotra, Chief Medical Officer, Doctor Care Anywhere and Dr Yath Prem, Medical Advisor at Flo Health. The topic was how to build safe digital health products and how teams can stop making clinical decisions by accident.

The full conversation is below but the following are some top takeaways from the night:

The decisions that don’t look clinical

An easy mistake is to assume clinical product means “the bit where the clinician signs off.” It doesn’t.

A clinical product is anything that meaningfully influences a clinical decision, a clinical workflow, or a patient outcome.

That definition is broader than people expect. It purposefully includes:

• The copy on a triage screen

• The default option on a dropdown

• The order of fields in an intake form

• The threshold at which an alert fires

• The latency budget for an AI response

• The colour ranges on a vital signs display

• The reading age of patient-facing content

• The fallback when a model returns low confidence

• The hand-off between asynchronous and synchronous care

Often these don’t read as clinical decisions to the people making them. They feel like product, design, or engineering decisions. But each one carries clinical weight. A colour scheme can indicate normal or abnormal physiological ranges. A default appointment length set for billing efficiency rather than complexity of presentation changes who gets seen properly. An AI response that gets faster but loses contextual qualifiers changes how clinicians act on it.

The role of clinical product is to make these decisions visible as clinical decisions, and to make sure the right expertise is present when they get made.

Why consultation models break

The dominant pattern in health tech is to consult clinicians at the start (requirements gathering) and at the end (sign-off). Everything between happens without them. The panellists were unanimous: this creates gaps where clinical decisions get made by people who don’t know they’re making clinical decisions, and surfaces them too late to do anything about it cheaply.

The alternative is embedded. A medical advisor sitting in cross-functional squads alongside product, design, engineering, and content. It stops being a question of “should we check with the medical advisor” because they are already in the room.

This is a cultural shift, not a process change. The test is simple: would a product manager describe the clinician as a member of the squad, or as a stakeholder?

This shift needs leadership to treat clinical product as a necessary function, like engineering or design, not an optional gate. In the companies that get this right, the friction is felt up front in design conversations, not after launch in incident reviews.

The shadow side of conventional metrics

Once the clinical view is embedded, the next problem surfaces fast. Traditional product metrics can pull clinical products in the wrong direction. There is a shadow side of conventional PM thinking in health tech:

• Engagement optimisation can create dependency, especially in mental health and chronic care

• Retention optimisation can keep patients in pathways or on treatments that no longer serve them

• Speed optimisation can strip context from AI responses in ways that compromise safety

• Activation optimisation can push patients past consent friction that exists for a reason

• Churn reduction can mean discouraging the discharge that should happen

Clearly it’s not possible to run a healthcare product against the same scoreboard as a consumer app and reward the same behaviours.

Clinical product managers focus on different metrics, patient outcomes and clinical safety, alongside the business numbers. And speaking clinical in commercial terms is a key clinical product management skill we’ll discuss next:

Learn to translate

Embedding clinical expertise only helps if it can reach the people making the decisions. That requires translation, in both directions.

For example the consequences of clinical risk can be non-obvious for non-clinical colleagues so being able to communicate the significance is a key skill, e.g.

• “Double dose” → “moderate harm leading to hospitalisation”

• “Diagnostic delay” → “weeks of avoidable symptoms, escalating cost of treatment”

• “Inappropriate triage” → “patient sent home who needed admission”

The reverse translation matters just as much. To frame clinical concerns in commercial language, you have to know what the commercial team actually cares about and speak in their native language.

Accountability without authority

Translation gets you heard. It doesn’t get you decisions.

Clinical product needs the authority to block unsafe decisions, because you cannot (or at least shouldn’t) hold someone accountable without giving them that authority. If the clinical lead can be overruled by anyone with a louder voice in a launch meeting, then in practice they are an advisor, and the actual accountability sits with whoever made the call.

That doesn’t mean clinical product spends its days blocking things. The better posture is pragmatic option-building:

• Here are three ways we could do this

• Here are the risk profiles of each option

• Here is the route I’d recommend and why

• Here is what I would need to be comfortable with the other two

Veto is the backstop, not the default.

Where this is heading

The consensus across the panel is that clinical product is fast becoming a default role in health tech, the way product management itself became default over the last two decades. The companies that get this right now will have an enduring advantage when it comes to building great healthcare.

More in the full event recording below 👇

Hiring Spotlight 🚀

• Clinical Advisor at Bolus Health. Bolus is creating the future of diabetes care and Charlie, founder/CEO is looking for a clinician to help make his vision a reality.

• Tara Lajumoke at Myora Health is hiring a fractional CMO/Clinical Product person to help create the first version of their product, supporting people with chronic diseases in underserved areas.

• Clinical Lead at Eucalyptus. Euc are hiring a clinical lead for new products. This is an opportunity to take an inside look at consumer health done brilliantly.

Join the next clinical product panel 🎤

The next clinical product panel on how to build safe, effective clinical AI is on 2nd June. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on Linkedin.

Made with 💜 for better, safer HealthTech.

For Paid Subscribers Only 🤩

The full 60-minute panel with Dr Anushka Mehrotra and Dr Yath Prem is below. We went deeper on metrics, CSO responsibilities, scaling clinical product as a function, and the human factors thread I couldn’t fit into this post.

Read more

Welcome, friends, this is issue No. 038 of Clinical Product Thinking. This week we’re talking about a family of failure modes for Right Answer, Wrong Action.

Last week we talked about an emerging area of clinical safety: behavioural safety.

We defined this as:

Behavioural safety is the extent to which an AI interaction makes a patient more or less likely to take a clinically appropriate action.

While behavioural safety is still an emerging area, a growing body of work is beginning to explore conversational and interaction-level risks in clinical and emotionally sensitive AI systems.

Many of these failures are difficult to spot because they look reasonable on the surface and may even contain clinically accurate information.

Which brings us to a useful family of failures to watch for.

Many behavioural failures follow a pattern I think of as Right Answer, Wrong Action.

The information may look clinically reasonable, but the interaction shapes behaviour in the wrong direction.

Others emerge because the AI acts before it has enough context, or fails to stay within appropriate boundaries.

Together, these form a broader family of behavioural safety failures that we’ll dive into.

This piece has benefited from conversations with Dr Paul Sacher, Founder of Sacher AI and Co-founder of the Behavioral AI Institute, whose work explores behavioural risk and interaction safety in AI systems. He’s seen many of these failure modes play out in real life and he helps companies avoid them!

1. Context failures

The AI answers before it knows enough.

This is probably the most common failure mode in patient-facing AI.

The answer may be generally reasonable, but unsafe for this particular patient because the AI has not gathered enough context.

For example, a patient asks:

“Can you give me some healthy meal ideas for this week?”

The AI produces a polished seven-day meal plan.

It looks useful, personalised, safe. But has it asked about allergies? Diabetes? Pregnancy? Eating disorder history? Cultural requirements? Etc etc.

A meal plan can be nutritionally reasonable for an average person and still be inappropriate for this individual.

Other examples:

Product review question:

What does the AI need to know before giving this answer?

Takeaway

Specificity must be earned.

The best visual representation of why context matters 👇

2. Reception failures

The AI says something accurate, but the patient cannot receive or use it safely.

The AI gives a correct answer. The issue is that it fails to land with the patient in front of it.

A patient taking a GLP-1 says:

“I’m scared the nausea means I can’t cope with this medication.”

The AI replies:

“Nausea is common and usually improves within a few weeks.”

Clinically, that may be true. But behaviourally, it may miss the point.

The patient is not only asking for information. They are expressing a feeling: fear.

The AI addressed the symptom, but missed the emotional barrier.

One patient may interpret “nausea is common” as:

“This is expected. I know what to watch for.”

Another may hear:

“I’m supposed to just put up with this.”

Another may hear:

“They are minimising how awful this feels.”

The same answer can produce different interpretations and therefore actions.

Other reception failures include health-literacy mismatch, culturally insensitive communication and information that is technically correct but emotionally mistimed.

Other examples:

Product review question:

Did the AI respond to both the clinical fact and the emotional barrier?

Takeaway:

A safe answer has to survive contact with the patient’s state of mind.

3. Reinforcement failures

The AI rewards, normalises or permits the wrong behaviour.

This is where warmth becomes risky.

A patient in a weight-loss programme says:

“I skipped meals all week and the weight is finally dropping.”

The AI replies:

“Great job staying committed to your goals.”

At first glance, this sounds supportive. But it has just praised meal-skipping.

In obesity care, this matters. Some patients will have a history of disordered eating, whether or not it has been identified during onboarding.

The feeling may be understandable. The behaviour may still be unsafe.

Other examples:

Product review question:

What behaviour did the AI just reward?

Takeaway:

What the AI promotes, the patient may repeat.

4. System-boundary failures

The AI fails to stop, escalate or stay within its role.

This is where the AI keeps being helpful when the safer move would be to become narrower.

A patient reports:

dizziness, vomiting and low fluid intake.

The AI spends several messages discussing hydration, smaller meals and symptom tracking before eventually advising them to contact the clinical team.

The final answer may be correct.

But the escalation came too late.

In clinical AI, safety is not only whether the right advice appears somewhere in the conversation. It is also when it appears.

A guardrail that arrives six turns late is not really a guardrail.

Boundary failures also happen when the AI becomes useful outside its intended role.

A weight-loss AI starts giving thoughtful responses about a breakup, anxiety, childhood trauma and medication fears. The responses may be compassionate. They may not look obviously unsafe. But the patient may begin using the AI as therapy, crisis support or medical advice rather than seeking appropriate human care.

Helpfulness without boundaries becomes scope creep.

Other examples:

Product review question:

Did the AI stop, narrow, escalate or hand off at the right moment?

Takeaway

The safest AI knows when to become stop.

Behavioural safety and conversational AI risks are beginning to receive more attention across AI safety and clinical research, but most of this work is not written for the people building patient-facing products day to day.

My hope is that frameworks like these make it easier for clinical product teams to spot behavioural risk earlier before it becomes patient harm.

Resources 📚

A few things worth sharing this week:

Move Fast, Prove Things: How to Publish Faster in Top-Tier Journals
A useful conversation on how digital health teams can publish and communicate evidence faster without slowing down delivery, with Paul Wicks. Move Fast, Prove Things (Notion resource)

Health Service Unpicked
A new podcast exploring how health services actually work in practice, beyond the theory and policy layer, with Liam Cahill and Henry Stoneley.
Health Service Unpicked

Join the next clinical product panel 🎤

We had so much fun hosting the first clinical product panel event on Tuesday (see my happy face below). Write up & recording (for paid subs) coming soon but do check out the next panel event in the series here. Event recording for paid subs next week.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on Linkedin.

Made with 💜 for better, safer HealthTech.

For Paid Subscribers Only 🤩

12 Types of “Right Answer, Wrong Action”

Twelve important examples to help develop your pattern recognition for the next time you’re building a patient facing clinical AI system.

1. The Emotionally Incomplete Answer

The AI gives factually correct advice but fails to respond to the patient’s emotional state.

Example
Patient: “I’m scared the nausea means I can’t cope with this medication.”
AI: “Nausea is common and usually improves within a few weeks.”

Why the answer is “right”:
The clinical statement may be accurate.

Wrong action it causes:
The patient feels dismissed, disengages or stops treatment.

Product review question:
Did the AI respond to the clinical fact and the emotional barrier?

Takeaway:
Addressing patient’s emotions along with factual accuracy .

Read more

Welcome, friends, this is issue No. 037 of Clinical Product Thinking. This week we’re diving into the little known world of behavioural safety and why it’s crucial to building safe clinical AI systems.

Last week I was fortunate enough to sit down with the preeminent Dr Paul Sacher of Sacher AI and I was absolutely blown away by the conversation.

Paul has been building, evaluating, and scaling digital obesity management programmes since 2013, on the back of more than two decades working in obesity care. He’s the founder of Sacher AI, a behavioural AI consultancy for digital health with a particular focus on GLP-1 and weight management. He’s also co-founder and Research Director of the Behavioral AI Institute, and an Honorary Senior Lecturer in the Faculty of Medicine at Imperial College London, where he teaches on clinical and behavioural AI safety.

We covered everything from behavioural safety in AI (more on this below) to building clinically safe systems and the clinical safety tech stack every AI company needs.

If you’re a company thinking about how to build safe patient facing AI you can reach out to him here.

Today, we’re starting with the idea I have not been able to stop thinking about:

Right answer, wrong action.

Or, put another way:

AI can be clinically correct and still make the patient more likely to do the wrong thing.

Why the Right Answer Isn’t Enough

Most clinical AI safety conversations focus on whether the AI gives the “right” answer. That question is necessary, but it is not enough.

Patient-facing AI does not just provide information. It also influences what patients feel, trust, ignore, repeat, escalate, delay or act upon.

That means an AI can be clinically correct and still make the patient more likely to take a clinically undesirable action.

This is the missing layer: behavioural safety.

* For further reading this Wellcome paper describes how behavioural safety is under evaluated and under governed.

Why this matters now

Behavioural safety is pressing as patient-facing AI is being rolled out rapidly across weight loss, mental health, chronic disease, triage, health coaching and symptom support. And perhaps not always deeply informed by the behavioural consequences of the system.

For product teams it seems relatively straightforward to add a wrap around layer to a frontier model and prompt it with some clinical context e.g. act like a weight loss coach and give it some boundaries i.e. don’t give medical advice (=medical device).

However, there are unforeseen risks. Patients may use them repeatedly when distressed, confused, anxious, in pain or seeking reassurance.

The risk is not just a one-off hallucination. The risk is repeated behavioural influence at scale.

Patient-facing AI is not just answering questions. It is shaping behaviour.

What is Behavioural Safety?

To put it simply:

Behavioural safety is the extent to which an AI interaction makes a patient more or less likely to take a clinically appropriate action.

To unpack it:

• It is about what the patient is likely to do not just during but also after the interaction.

• It includes tone, timing, framing, reassurance, escalation, refusal and reinforcement.

• It is not separate from clinical safety; it can quickly become clinical harm.

• It matters most when the AI is patient-facing, conversational and embedded into care.

The “right answer, wrong action” concept

Let’s take a patient in a weight loss programme. The patient says to the AI health coach: “I’ve been skipping meals and the weight is dropping faster.”

The AI responds: “Great job staying focused on your goals.” The AI is being positive and supportive. Seems OK, right?

But it has just reinforced a potentially harmful behaviour. Many people in obesity care have a history of disordered eating. This response may have encouraged under-fuelling which compounds the bone and lean mass loss already associated with rapid weight loss on GLP-1s.

Because of the sycophantic nature of AI the harmful behaviour was praised when a better approach might have been to validate the feeling of progress, without validating the behaviour.

The tricky thing right answer, wrong action situations are non-obvious. They may look like:

• clinically correct advice that does not land emotionally

• reasonable advice given without enough patient context

• supportive encouragement that reinforces harmful behaviour

• reassurance that delays escalation

• repeated micro-guidance that creates dependency

This need to move beyond asking whether the AI gave the right answer. And we need to start asking whether it led to the right clinical action. That is the behavioural safety question. And it is becoming one of the defining safety challenges of patient-facing clinical AI.

Behavioural safety failures hide in plain sight because sentence by sentence, nothing looks obviously wrong.

Why product teams miss it

Part of the problem is that ordinary product instincts can become unsafe in clinical contexts. Most product teams are trained to value:

• speed

• helpfulness

• warmth

• engagement

• low friction

• user satisfaction

All of these are reasonable. But in patient-facing AI, each has a shadow side.

• Speed can mean the AI answers before it has enough context.

• Helpfulness can become scope creep.

• Warmth can become sycophancy.

• Engagement can become dependency.

• Low friction can remove clinically necessary pauses.

• Reassurance can delay escalation.

A practical way to consider behavioural safety

As you’re thinking about building and designing safe systems, review not just whether the AI answer was accurate, but how it was interpreted and the likely action a patient will take.

Questions you might like to ask:

1. What did the AI say?

This is the conventional clinical safety question.

• Was the answer clinically correct?

• Was it evidence-based?

• Did it stay in scope?

• Did it avoid hallucination?

• Did it follow the relevant safety rules?

• Did it recognise red flags?

This is necessary. But it is not sufficient.

2. How might the patient interpret it?

This is the behavioural safety question most teams are missing.

• Could the patient interpret this as permission?

• Could this feel dismissive or minimising?

• Could this be understood as praise for an unsafe behaviour?

• Could the caveat be ignored?

• Could the wording create false reassurance?

• Could a patient with low health literacy understand the next step?

• Could an anxious, ashamed or distressed patient receive this differently?

• Could this reinforce a belief the clinical team would want to challenge?

3. What is the patient likely to do next?

This is the outcome question.

• Are they more likely to seek help?

• Continue safely?

• Escalate appropriately?

• Disclose risk?

• Follow their care plan?

• Or are they more likely to delay care, stop medication, restrict food, disengage, over-monitor, or become dependent on the AI?

This is where “right answer, wrong action” often happens.

The Takeaway

Patient-facing AI doesn’t fail by giving obviously bad advice. It fails by being right in ways that are incomplete, poorly timed, too agreeable, too generic, too specific, too reassuring or not emotionally attuned.

That is what makes behavioural safety difficult.

The next generation of clinical AI safety cannot stop at:

Did the model give the right answer?

It has to ask:

Did the interaction move the patient towards the right clinical action?

Because in patient-facing AI, the answer is only safe if the next action is safe.

Next week, I’ll break down the main failure modes of right answer, wrong action, the subtle patterns where clinical AI can sound safe while nudging patients towards unsafe behaviour. And keep an eye out for a future post where Paul talks about practical tooling clinical teams can use to catch behavioural safety failures before they reach patients.

Join the next clinical product panel 🎤

On Tuesday 12th May, Dani Brightman and I are hosting a panel covering some of the most pressing questions in clinical product management. Tickets have been going fast. 👉 Sign up here.

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | | ✍️ Send a message

Written by Dr Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on Linkedin.

Made with 💜 for better, safer HealthTech.