The 5 Principles Every Clinical Product Manager Needs to Build Safe AI
Your weekly newsletter on all things clinical product and building better healthcare 🏥
This is Clinical Product Thinking 🧠, a weekly newsletter featuring practical tips, frameworks and strategies from the frontlines of clinical product.
Hello friends, this is issue No. 013. Today, we’re staying on our regulation theme and cracking open one of the biggest focus areas in clinical product right now: AI as a medical device.
Last Wednesday, a small group of clinical product leaders and founders gathered for dinner to swap real stories about what it takes to build, ship and operate AI safely. Dr Dom Pimenta (CEO, Tortus) shared frontline lessons, and the table brought perspectives across safety, product, engineering and regulation.
It was direct, energising and occasionally spicy. Here are the biggest takeaways you need to know.
1. The Device Is the System, Not Just the Model
An important theme of the evening:
‘The medical device is the system as a whole.’
Not the AI model. Not the prompt. Not the UI. Not the dataset.
The system.
That includes the:
UI and workflows
Prompts
LLM(s)
Monitoring layer
Infrastructure
Risk controls
Once you think this way, how you make product decisions changes. ‘Tweak the prompt’ stops being a casual UX experiment and becomes a controlled design change in your QMS. Some prompt changes will be minor, but anything that shifts behaviour or risk may need re‑evaluation and, in some cases, regulator or Approved/Notified Body involvement.
👉 What CPMs should do tomorrow
Map or review your full ‘AI medical device system’: UI → prompts → model → monitoring → infra → safety controls.
Version-control prompts, not just code.
Decide which prompt changes are ‘minor’ vs. ‘needs re-verification’.
Put a governance workflow in place for any model or infra update.
2. If AI Touches the Patient, it Will Likely Become a Medical Device
One of the clearest lines of the night:
‘As soon as AI touches the patient, assume it’s a medical device.’
Even if you built it for efficiency.
Even if it’s ‘just a summary’.
Even if the user is a clinician.
Regulators in the EU, UK and US are tightening how they interpret risk.
They are moving from “what you say it does” → ‘what harm it could cause if it goes wrong.’
This means more AI is likely to fall inside MDR/MHRA classification, often in Class IIa or above, particularly when it:
Influences diagnosis
Influences treatment
Summarises clinical notes
Extracts findings
Reinforces clinical decisions
Interacts with patient-specific data in any meaningful way
👉 What CPMs should do tomorrow
Ask these four questions for every AI feature:
Could a clinician act on this?
Could a patient be harmed if it’s wrong?
Does it generate or transform patient-specific clinical information?
Could a regulator reasonably interpret this as clinical decision support?
If the answer is ‘yes’ to any: Treat it as a device until proven otherwise.
3. Monitoring Isn’t Optional. It Is the Product
LLM outputs are non-deterministic and change over time. Even small changes or updates can alter behaviour in unpredictable ways.
‘You cannot test a model once and assume it stays safe.’
You may need:
Live performance monitoring
Hallucination detection (classified by harm severity)
Drift detection
Automated threshold alerts
A rollback mechanism
Human review pathways
This isn’t analytics. This is your safety layer and regulators increasingly expect it.
👉 What CPMs should do tomorrow
Define your hallucination severity rubric.
Decide when an alert is triggered and who receives it.
Write your rollback procedure.
Add ‘monitoring’ as a formal design input in your QMS.
4. Compliance Built Early Saves Teams
This story probably sounds all too familiar for people in the industry:
‘Backdating a year of work to be compliant… people were crying at their desks.’
If you don’t implement your QMS early, and actually build within it, you’ll end up retro-fitting months of decisions, prompts, risks and tests just to meet basic design control requirements. Most of that work has to be reconstructed from Slack threads, outdated Notion pages, and people’s memories.
Teams that take this approach lose weeks, sometimes months, and it’s painful, demoralising work. A lightweight QMS from day one avoids all of this and keeps teams moving quickly and safely.
👉 What CPMs should do tomorrow
Build the QMS early
Write verification plans early
Define evidence plans early
Treat infrastructure and prompts as design artefacts
Keep decisions documented
5. Evidence Takes 10x More Work
A key takeaway:
‘One unit to build the AI; ten units to evaluate it.’
Evaluation means:
Labelling
Multiple reviewers
Disagreement resolution
Adjudication
Metrics design
Task-specific test sets
Repeatability checks
Most early AI teams underestimate this by months and potentially even by hundreds of thousands of pounds.
👉 What CPMs should do tomorrow
Choose task-specific metrics that reflect real clinical use
Version-control your test sets so they’re stable and auditable
Budget time, reviewers and money for evaluation, it may cost more than expected
Treat evaluation as a core product workstream, not an afterthought
Clinical Product Calendar 2026 💁♀️
Incredibly excited to be planning out the Clinical Product Thinking 🧠 calendar for 2026.
We kick off with a panel event on 29th Jan, followed by intimate Clinical Product Dinners on 4th Feb and 4th March. The next round of Clinical Product Drinks lands on 25th March. Save the dates and subscribe to the CPT Substack for early access codes. More details here.
Content & Community Associate 😍
Clinical Product Thinking 🧠 is looking for a part-time Content & Community Associate to support content ops, events and early testing of new Clinical Product AI tools. Ideal for someone early in their healthtech or clinical product career who wants mentorship, exposure and hands-on experience in a fast-growing discipline. 👉 Details and how to apply here.
From the Community 💡
A few highlights from the Clinical Product community this week 👇
25 Nov | Hash It Out: Regulating AI in Healthcare | London, UK: Hosted by Hale House, in partnership with Assuric. This event will cover the future of AI healthcare regulation. (👋 Say hi to me there!)
11 Dec | An introduction to Generative AI in Healthcare | Online + Belfast, UK An overview on how to roll out GenAI safely across the NHS, with Dr Keith Grimes. (👋 Say hi to me online!)
Post | Guiding Principles for Technical and Clinical Evaluation of LLMs as a Medical Device: By Scarlet, a medical device regulator focusing on AI.
Webinar | A Framework to Assess Clinical Safety and Hallucination Rates of LLMs: An incredible talk by Dom for The Alan Turing Clinical AI Interest Group. A must-watch!
That’s all for this week. See you next time! 👋
🤝 Work with me | 📅 Attend an event | | ✍️ Send a message
Written by Dr.Louise Rix, Head of Clinical Product, doctor and ex-VC. Passionate about all things healthcare, healthtech and clinical product (…obviously). Based in London. You can find me on Linkedin.
Made with 💜 for better, safer HealthTech.
Excellent analysis. Thinking of the entire AI system, not just the model, is realy crucial. Anyone who's debugged knows this.