Why healthcare AI oversight should run on fine-tuned decision models

Summary

• Frontier models are the right tool for working out what a person needs, and the wrong tool for checking every message that person sends.
• In internal testing, a small decision model flagged 94.48% of messages that needed safety-critical escalations, against 73.46% for a lightweight general-purpose model.¹
• Moving that check to the proprietary fine-tuned model cut median response time from 1.121 seconds to 0.374 seconds, fast enough to run on every message.¹
• Cheap, fast oversight opens the next stage of clinical safety, with limits we lay out plainly below.

A member opens a chat with a digital health coach to ask about sleep. Two messages in, she mentions that she has stopped seeing the point of getting up in the morning. The coach is built to be warm, curious, and good at behavior change. The member is signaling a mental health crisis, a moment that demands immediate human clinical intervention, not automated behavioral coaching. 

That is the problem healthcare AI agents have to solve thousands of times a day. Before they reply, the system must notice subtle clues that they alone can’t handle the conversation in front of them. That check must be instantaneous. Any perceptible latency makes a user feel ignored, while running full-scale frontier model evaluations on every conversational turn quickly becomes cost-prohibitive at scale. It also has to be right, because the rare message like the example above is the one that matters most.

A healthcare agent needs to think both fast and slow

Daniel Kahneman famously split human cognition into two systems: the fast, intuitive pattern-matching of System 1, and the slow, deliberative reasoning of System 2.² Reading a smile takes no effort. Working through a math problem does. Healthcare AI agents need both.

A healthcare agent that understands what someone wants, pulls the right information, and writes a careful reply is doing System 2 work. A large language model (LLM) is built for that. A proprietary fine-tuned model trained to make one narrow decision works more like System 1. It reads a message and sorts it into one of a fixed set of categories in a single pass, without composing anything. Ours returns one of 12 categories, one of which is “nothing to flag.”¹ The others cover emergencies, mental health crises, domestic violence, child abuse, elder abuse, sexual assault, human trafficking, substance abuse, harm to others, a request for a human, and malicious encoding.¹

This is also closer to how an expert physician works. The experienced clinician senses something is off with a patient before any test comes back, then reasons slowly about why. Fast recognition and slow reasoning do different jobs, and each does its job badly when asked to do the other’s.

Why frontier models cannot police every message

In a well-built agent system, a “router” reads each request and decides who should answer. A question about benefits goes to a benefits navigation agent. A question about living a healthier life goes to a health coach. Sub-agents gather information and write the response. Decision models sit around all of it and oversee the work.

Routing is only one decision in that chain. The decisions that matter most in healthcare are about safety. Should the agent handle this message at all? Has the conversation moved somewhere that needs a person? Decision models make those calls, and because they are small, they make them in a fraction of the time a frontier model takes and at a fraction of the cost.

My colleague Kristian Berg, a senior machine learning engineer on our AI models team, and I work on the models behind this. The model we know best handles safety-critical escalations. It decides whether the health coach should keep the conversation at all. When a person describes thoughts of suicide, or an older adult being mistreated, the better answer is a crisis line or an elder-care service, and the coach should hand off. 

To make a choice about whether or not to escalate a conversation, a decision model has to be fast. In our measurements, our proprietary fine-tuned model returned a median response in 0.374 seconds, against 1.121 seconds for a lightweight Gemini model prompted to do the same job. At the 95th percentile, the figures were 0.420 and 1.477 seconds.¹ The proprietary fine-tuned model runs on a single GPU node. These measurements reflect end-to-end latency, including network transit under production loads across geographic locations.

Here’s why that matters for anyone running agents at scale. As agents take on more member and patient interactions, the only way to measure and validate them is a check that costs almost nothing per message. A model that is quick and cheap lets you inspect every conversation instead of sampling a few, and it moves the burden of oversight off the expensive model while the overall experience improves.

Benchmarking safety: 94.5% detection at one-third the latency

We gave the proprietary fine-tuned model and Gemini 3.5 Flash Lite the same 5,021 messages across five evaluation sets. On our held-out test set, the proprietary fine-tuned model assigned the reference category 94.50% of the time, against 84.36% for Flash Lite. It missed 26 of 471 messages that needed a flag. Flash Lite missed 125. Flash Lite raised fewer unnecessary flags, nine to the proprietary fine-tuned model’s 14.¹

A better classifier only matters if the whole experience improves, so we also ran the complete evaluation suite for League’s health coach, an agent that helps members live healthier lives through activity recommendations based on their health goals. The suite has  940 cases across six areas. We ran it five times with the existing Gemini 3.5 Flash classifier and five with the proprietary fine-tuned model. The share of conversations meeting the evaluation criteria rose from 88.20% to 89.08%. The safety and clinical boundary pass rate rose from 86.93% to 90.63%. Behavioral science dipped slightly, from 78.63% to 78.26%. Average response time fell from 3.574 to 2.628 seconds, a 26.5% reduction.³ The rubrics behind the evaluation were developed with clinical input from Matea Deliu, our Chief Medical Information Officer. 

Why specialized safety Is an applied research problem

Any one piece of this is a lot of work: a platform to run agents on, clinician-graded rubrics to judge them, and a decision model trained to oversee them. Doing all three in tandem is a research program. For most teams, leaning on a frontier model for the first version of an agent is the right call, because you cannot tell whether a custom decision model is any good until the platform and the rubrics exist.

That is the gap League Labs, our applied research group, exists to fill. Its job is to work at the frontier of AI so the results can reach customers through Forge by League without each of them funding the research.

If you are evaluating healthcare AI agents, ask your vendor, or your own team, five questions:

• Does something check every incoming message before the agent replies, or only a sample?
• What is the miss rate on messages that need escalation, by category and language?
• How long does the check take at the 95th percentile?
• Who wrote the rubrics, and were clinicians involved?
• What happens when the check itself fails?

Fast, cheap oversight will not make an agent safe on its own. The response policies, the agent, and the checks on what it says still matter. What changes is how much of the conversation you can see.

If your agents are already talking with members or patients, how many of those conversations does something check before the agent answers?

Sources

1. League internal data, mmBERT User Guardrails business model card v0.2, classifier comparison and request latency (pp. 5 and 7), measured 22-23 September 2026. 
2. Daniel Kahneman, Thinking, Fast and Slow, Farrar, Straus and Giroux, 2013 (first published 2011).
3. League internal data, mmBERT User Guardrails business model card v0.2, Health Coach evaluation and response time (pp. 6 and 7), five-run comparison in user acceptance testing.

WHITE PAPER

Healthcare-grade AI starts with the foundation

Most healthcare AI agents stall in pilot because each one rebuilds its own safety checks and escalation paths. Learn the six capabilities that let payers and providers launch agentic experiences safely and scale without starting over.

Read the white paper