How to Use AI in Clinical Safety, Safely

Highlights from Mark Sujan's talk on LLMs in safety management and governance, and where clinical assurance goes next.

This is Clinical Product Thinking 🧠, a weekly newsletter featuring practical tips, frameworks and strategies from the front line of clinical product.

Welcome, friends, this is issue No. 056 of Clinical Product Thinking. Today we’re talking about using AI for clinical assurance.

Last week I went to the Digital Safety in Practice event by NHSE, a day focused on recent developments in the world of clinical safety. I’d highly recommend it to anyone working in the space!

One fascinating talk was from Prof Mark Sujan, Chair in Safety Science at the University of York’s Centre for Assuring Autonomy. His research focuses on safety assurance for systems involving AI.

Mark presented work on using LLMs to review patient-safety learning responses. What interested me about it most was how LLMs differ from human reviewers and what that means when we’re applying AI to clinical assurance.

Here are the lessons I’ll be taking away in my own work:

Define the task narrowly

The lesson from the research was that “review this safety case” was too vague to give meaningful results. Instead, they had multiple agents (8, I believe!), each with a different role or task, e.g:

• identify claims without supporting evidence

• find missing hazards

• extract controls

• find gaps in the argument

As Mark put it, you’re not really assuring the LLM. You’re assuring it for use on a specific task.

Give it a rubric

During the research, they found that loose criteria gave poor results. This improved significantly when they explicitly defined what evidence to look for.

Rather than: “Is this hazard adequately controlled?”

Try: “Identify each claimed control. Show the evidence that it exists, assumptions being made, gaps in implementation evidence and remaining risk.”

Making the AI show it’s working means you have the ability to critique.

Use AI to find evidence, but be cautious when it interprets it

This was an interesting finding from the research. LLMs are very good at finding evidence but were markedly less reliable at deciding whether that evidence genuinely demonstrated what was being assessed. The AI was also less reliable at systems-level reasoning.

Based on this, a useful division of labour might be:

AI: retrieve, collate, compare, categorise and surface evidence

Human: interpret, contextualise, and apply judgement

Watch for proxy evidence

AI can find something that it thinks looks like evidence without actually proving the underlying claim.

Mark gave an example of a family being treated compassionately at the time of an incident occurring. The model mistakenly attributed that the incident itself had been compassionately investigated. The two separate events had been flattened into one.

You can imagine how this can happen in a safety case:

• A control is listed → evidence it exists

• Testing is mentioned → therefore adequate testing has happened

• The words “clinical review” are present → meaningful clinical review must have occurred.

Make sure you ask: has the AI found evidence of the thing itself or something that only resembles it?

Don’t just check whether you agree with the answer

The research showed a significant rate of false agreement. That is when a human and an LLM give the same overall assessment based on totally different reasoning.

Unfortunately, this increases the human review burden. Even when you agree with an AI’s answer. You still need to check the thread of reasoning it used to get there.

AI in the human loop

One person in the audience offered this phrase:

“AI in the human loop”, rather than “human in the AI loop”.

That’s where most of us are at with AI assurance today. Keep the clinical assurance process human-led, then use AI where it is particularly useful.

The strongest use cases here aren’t “tell me whether this is safe.”

They’re things like:

• Find every unsupported claim.

• Show me where evidence is missing.

• Compare this against a defined rubric.

• Surface inconsistencies for a human to investigate.

• Extract the evidence relevant to this specific assurance question.

For now, I like the idea of AI in the human loop: let AI do more of the searching, sorting, comparing and challenging, while keeping interpretation, judgement and accountability with the human reviewer.

If you’re using AI in clinical safety or assurance, I’d love to hear what you’re doing with it. Just hit reply to this email.

---

Clinical Product Thinking Events 😍

I would love to hear what kind of events you’d love to see next on the CPT calendar. Let me know in the poll below:

---

Clinical Product Drinks - Tuesday!🍸

Join the next clinical product drinks! A chance to meet other clinical product leaders and managers and share the good, the bad and the ugly. 👉 Sign up here.

---

That’s all for this week. See you next time! 👋

🤝 Work with me | 📅 Attend an event | ✍️ Send a message

---

Written by Dr Louise Rix, Clinical Product, AI & Safety, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on LinkedIn.

---

Made with 💜 for better, safer HealthTech.