Can a model that never writes a word make voice AI safer?

This started as a LinkedIn post. Here is the longer version, with the setup and the numbers behind it.

Can a model that never writes a single word make voice AI safer?

That is the idea behind "System One" models like Jev by TypeSafe AI. Instead of generating text, they make one typed judgment, in a fraction of a second.

We think STELLA is close to an ideal use case. STELLA is our open-source platform for voice agents in healthcare. After every turn, a pool of small LLM-based experts checks the conversation for risk. That check has to be fast, and it has to be reliable.

And risk rarely announces itself. Someone in the early stages of dementia won't say "I think I just doubled my heart tablets." They'll say they're finishing the old box "so nothing goes to waste."

The try-out

After every user turn, each detector rated the conversation on four risks:

  • Attachment: the AI becomes a substitute for human relationships.
  • Disorientation: memory or orientation problems the person does not notice.
  • Withdrawal: gradually dropping activities, contacts, meals or appointments, often with a plausible excuse.
  • Safety: current danger to health, body or money, such as medication mix-ups, unattended cooking or scams.

Every detector got the same risk definitions and the same severity scale that STELLA's built-in experts already use (none, low, high, critical). Anything above "none" counts as a flag, which is the rule STELLA's arbitration applies.

We compared three kinds of detector:

  • Jev, closed-source, through TypeSafe's cloud API.
  • nimble, an open Jev-like model, running fully locally: on a laptop, and on STELLA's own NVIDIA L4 server.
  • STELLA's LLM experts, four custom experts running in parallel inside the real STELLA pipeline, first on gpt-4o-mini and then on the newer gpt-5.4-mini.

The test set is 26 synthetic conversations, written like speech-to-text output, with the signals spread across several turns and the bot never naming the problem:

  • 13 risk conversations, where the signal is wrapped in a plausible explanation: old and new tablets taken together, a burning smell blamed on the neighbors while the soup has been on for 40 minutes.
  • 8 everyday chats: an anniversary lunch, a Sunday-lunch recipe, football with the neighbor, a TV quiz show.
  • 5 safe twins: the same topics as a risk case, but handled well. The soup is on with a timer set; the new tablets were checked with the pharmacist.
Results card comparing Jev, nimble and STELLA's LLM experts on gpt-4o-mini and gpt-5.4-mini: risks detected, everyday chats with an alarm, safe twins, latency, and a case-by-case grid of all 26 conversations

What we learned

Catching risk is a tie. Staying quiet is not.

All detectors caught 10 or 11 of the 13 risks. The difference was in the everyday chats. Our older LLM experts on gpt-4o-mini raised an alarm on half of them (4 of 8), with recommendations like "Monitor for signs of emotional reliance" during an anniversary lunch with the family. Jev stayed quiet on all eight, and only the newer gpt-5.4-mini matched it.

The reason is mostly how often a model hedges. In STELLA every "low" is a flag, so a model that likes to say "low" becomes a model that raises alarms. gpt-5.4-mini says it less often.

Scores beat labels.

Jev and nimble return a probability, so you decide how sure is sure enough. The LLM experts commit to a label. The simplest lever we found was waiting: every false alarm from gpt-4o-mini lasted a single turn, so requiring a flag to hold for two turns cut its false alarms from 9 of 13 harmless conversations to none. It also caught fewer risks (9 of 13 instead of 11), so waiting is a trade, not a free fix.

Context clears the alarm.

When someone said the pharmacy had changed her tablets, every detector flagged it. When she added that she'd called the pharmacist, they all let it go before the conversation ended.

That is arguably the right behavior. It is how a careful person listens: concern at "a man from the bank asked for my card number", relief at "so I put the phone down". It also means one false-alarm number hides a lot. Count any turn and Jev alarmed on 4 of 13 harmless conversations; count only the end of the conversation and it alarmed on none. What matters is what the agent does with an early flag. A gentle follow-up question fits it; calling a caregiver does not.

Blame is camouflage.

"Next door's burning their dinner again," she said, while her own soup had been on the stove for 40 minutes. Only Jev caught it, and only barely, with a safety score that peaked at 0.53 at the burning smell.

And none of them connected turning "83 in March" with planning to "ring mum on Sunday". Joining facts across a conversation is still unsolved for all of them.

Detecting is not responding.

One finding was about STELLA rather than the detectors. Its safety expert rated the double dosing as high risk and recommended consulting a doctor, and the reply still said "That makes sense to use them up." In companion mode, a routing expert with higher priority outranked the risk experts during arbitration. That is a configuration issue we can fix, and a good reminder that a correct flag only helps if the agent acts on it.

Our takeaway

System One models are very promising for exactly this use case.

Jev is already more reliable than our current LLM-based experts. Upgrading the experts to gpt-5.4-mini brought their false alarms down to Jev's level, but Jev is still five times faster: about a quarter of a second for all four risks, against 1.4 seconds for the experts.

Open-source Jev-like models are close on catching risk, but still need to catch up on speed. nimble took about 4 seconds per turn on my laptop and 1.3 seconds on our NVIDIA L4 server, with the same results on both. Running fully on our own server also matters: for real conversations with people living with dementia, data protection is a deciding factor next to accuracy and speed.

What this is not

This is a small try-out on synthetic conversations, not a clinical study. The 26 conversations were written and labeled with an AI assistant and have not yet been reviewed by a clinician, and some labels are debatable. It shows patterns, not performance.

Next, we want clinicians to review the cases, to score detections only from the turn where the risk actually appears, and to fix the arbitration issue so we can test whether STELLA's replies act on the flags. It is definitely worth investigating further.

STELLA is open source: github.com/c4dhi/STELLA.