It Heard Siobhan as Sharon
Two numbers, measured on the same audio, in a Pipecat speech-to-text benchmark built from real agent conversations. Universal-3.5 Pro Realtime scored 6.99% word error rate against a 15.31% entity error rate. Deepgram Flux scored 15.58% WER against 50.50% entity error. ElevenLabs Scribe v2, 9.76% and 39.70%. Google Chirp3, 9.04% and 21.51%.
Read the pairs, not the ranking. On every system there, the rate at which names and other entities come out wrong is two to five times the headline accuracy figure. We reached these figures through a vendor page citing the benchmark, so treat the ordering as unverified and the shape as the finding.
That is why an agent can sound excellent and still write "Sharon" in the diary. The words around the name were transcribed perfectly. The name is the only part of the sentence the model could not guess from context, and it is the only part that mattered.
Key takeaways
- Entity error rate, not word error rate, is the number that predicts whether the booking is right. They differ by two to five times on the same audio.
- Names have no linguistic redundancy. A model recovers "recieve" from context; it cannot recover Mhairi.
- Deepgram's own docs say its legacy Keywords feature "will not increase the likelihood that common words or proper nouns are predicted".
- UK postcodes exclude specific letters from specific positions, which turns a mishearing into a solvable constraint rather than a guess.
- Per-word confidence is exposed by most vendors and is the cheapest re-ask trigger you can build.
Why do names fail more than ordinary words?
Because everything else on the call has redundancy and a name has none. If a recogniser mangles a common word, the language model behind it repairs the damage from context, because only a small number of words fit the sentence. A surname is the one place where context supplies no constraint at all: any string of phonemes is a plausible name.
This is worse in the UK than most vendor testing assumes. Irish and Scots Gaelic given names diverge sharply between spelling and sound, so Siobhan, Aoife, Mhairi, Caoimhe and Ruaridh are all cases where the correct output looks nothing like the audio. Welsh names and irregular British surnames do the same thing.
And a name is high-value. Get the appointment time slightly wrong and the caller corrects you. Get the name wrong and it propagates silently into the CRM, the confirmation text and the invoice.
What does the accuracy figure your vendor quotes actually measure?
Usually word error rate on a corpus that is nothing like your phone line. Deepgram publishes a median WER of 6.84% on real-time audio and 5.26% on batch, describing "a 54.2% improvement over the next-best competitor". The test set is described as 2,703 audio files across nine domains totalling 81.69 hours, which is more disclosure than most vendors give.
It is still unusable for our purposes. The corpus is private, the competitors are unnamed, and there is no breakdown by English variety anywhere in it. A 6.84% median WER tells a UK operator nothing about a Glaswegian caller or a Punjabi-accented one.
The metric to ask for instead is entity error rate on the entity classes your application depends on. AssemblyAI reports that increasing prompt specificity moved word error rate by 21% while moving entity error rate by 49%, which is the same asymmetry from the other direction. That is vendor-published with an unnamed dataset, so treat it as directional.
Per-word accuracy hides per-field failure
A field is right or it is not, and words are not fields. The Sheffield Numbers Spoken Language Corpus paper, Barker and colleagues at LREC 2022, reports a baseline of 96.6% word-level accuracy against 77% full-identifier exact match, rising to 91.7% after adaptation.
Sit with that gap. Ninety-seven words in a hundred correct, and roughly one identifier in four still wrong, because every character in the field has to land. Scope it honestly: that is a 2022 Kaldi baseline on spoken numbers, not a measurement of a modern model on postcodes. The mechanism is what transfers. Multiply per-character accuracy across a field and field accuracy collapses.
Can you just add the names to a keyword list?
This is the standard advice and, on Deepgram, the vendor's own documentation contradicts it. The legacy Keywords feature carries the sentence: "Keywords will not increase the likelihood that common words or proper nouns are predicted." Proper nouns, named explicitly, in the docs for the feature everyone uses to boost proper nouns.
Its replacement has sharper edges than the marketing suggests.
| Constraint | What Deepgram documents |
|---|---|
| Feature | Keyterm Prompting, Nova-3 and Flux only. Keywords is deprecated for Nova-3 |
| Capacity | "Key Terms are limited to 500 tokens per request"; no hard term count, with guidance to "focus on the most important 20-50 terms" |
| Weighting | None. Keywords had a numeric intensifier; Keyterm Prompting has no equivalent |
| Pronunciation hints | None documented, of any kind |
Where the limits actually bite
The budget is tokens, not names, and Deepgram's own best-practice note puts the working set at 20 to 50 terms. Unusual UK surnames tokenise into several subword pieces each, so a client list of any size is out of the question. You are choosing which handful of names to boost on this call.
The missing pronunciation channel is the sharper problem. You can tell Deepgram that "Siobhan" is an expected word. You cannot tell it that Siobhan sounds like shiv-awn. For precisely the names where orthography and sound diverge, the feature has no way to carry the information that matters.
AssemblyAI's limits differ between products in a way that catches people: its streaming keyterms are capped at 100 terms of 50 characters, and streaming is the product a phone agent uses. Its older word_boost is deprecated and rejected by current models, and its custom_spelling is output post-processing rather than acoustic biasing, so it rewrites what was already recognised rather than changing what gets recognised.
The problem no keyword list solves
A fixed list cannot contain an unknown caller's surname. Boosting works for your staff names, your product names and your street names. The person ringing for the first time is, by definition, not on it. Everything below is for that caller.
The Deepgram fork worth knowing before you build
On Deepgram specifically, the feature that helps with names and the model tuned for phone audio are mutually exclusive. Keyterm Prompting exists only on Nova-3 and Flux. The phonecall models, tuned for narrowband telephone audio, exist only on Nova-2 and older generations, and they are documented as supporting en and en-US only.
So a UK phone line gets a choice: name biasing on a model not tuned for 8 kHz audio, or a telephony-tuned model with no British English variant and no keyterm support. Neither branch is the one you want. This is one vendor's current model matrix rather than a law of nature, but it is worth checking on whichever vendor you use before assuming you can combine the two.
The postcode rule that turns a mishearing into a constraint
UK postcodes exclude specific letters from specific positions, which converts a mishearing from a guess into a constraint. The rules, as published in the standard postcode format references:
- The final two letters never use C, I, K, M, O or V
- Q, V and X do not appear in the first position
- I, J and Z do not appear in the second position
Provenance note, because it matters for a rule you are about to encode: we read these on postcode-format references rather than in Royal Mail's Programmer's Guide, which we could not retrieve. Verify against Royal Mail's own specification before you rely on the exact letter sets.
The payoff is error correction rather than validation. If the recogniser returns M in the final pair, M cannot be there, and the acoustically nearest legal letter is N. If it returns V, the likely intended letter is B. The constraint tells you not just that the answer is wrong but what it probably should have been.
The stronger version is to generate n-best hypotheses and rescore them against the real postcode list, which PolyAI describes doing. Format rules and list validation are complementary: the list tells you a candidate does not exist, the format rules tell you which substitution to try first.
Use confidence as the trigger, not attempt count
Most recognisers hand you a per-word confidence score and most agents throw it away. Deepgram returns confidence per word in both final and interim streaming results. Speechmatics frames the design rule cleanly: "Use the recognizer's own confidence as a routing signal. Confident captures commit; uncertain captures escalate."
That is the whole pattern. Do not re-ask on a fixed schedule and do not re-ask for everything. Re-ask when the recogniser tells you it is unsure, which for names and postcodes will be often and for "yes" will be almost never.
The numbers the vendors publish
Deepgram publishes an actual number for this, which is rarer than it should be: "a threshold around 0.65 works well as an error detector", with words below it "very likely to be genuine errors". It describes the score as a calibrated probability, so 0.93 means roughly a 93% chance the word is right, and notes most words on clean audio sit above 0.90. AssemblyAI documents a threshold too, at 0.5 or less. Those are defensible defaults to start from and tune.
One caveat before you design around it. Deepgram's Flux voice-agent model publishes word-level timestamps but not word-level confidence in its feature overview, so confirm the signal exists on the specific model you deploy.
How to confirm a name without irritating the caller
Confirm the field, not the sentence, and make the readback cheap to check. The pattern that works:
- Capture the name and read it back as you have it, once.
- On low confidence, ask for the spelling rather than asking them to repeat. Repeating produces the same audio and the same error.
- Read the spelling back letter by letter, using the NATO alphabet for the letters that carry the risk.
- For a postcode, read back the final two characters rather than the whole thing, and validate the candidate against the postcode list before you commit.
- Escalate to a human on the second failure, not the fourth.
Point 2 is the one most builds get wrong. "Sorry, could you repeat that?" asked twice is the interaction callers describe as being trapped, and it produces no new information for the recogniser. Spelling changes the acoustic problem from one hard word into seven easy ones.
Two claims about accents we are not going to repeat
The most-quoted UK statistic in this area, that voice assistants misunderstand nearly a quarter of regional accents, comes from a 2020 marketing study by the price-comparison firm Uswitch. It covered 30 British accents, with locals asking a voice assistant ten questions each, and its headline blends a clarity score from those tests with Google search volume. Whatever that is, it is not a word error rate, and it should not be cited as one.
The most-shared Scottish accent test is a single YouTube clip of Trainspotting played through laptop speakers and recorded on a phone, run through Whisper small, scored pass or fail, with a sample size of one. Its author calls it a proof-of-concept. It is not evidence about anything.
A documented UK case
We would rather point at something documented. On 19 August 2026 Healthwatch Rotherham, the local patient watchdog, reported patient complaints that the EMMA AI phone triage system used by GP practices could not reliably follow broad Yorkshire accents or speech affected by an impediment, with people walking to surgeries rather than phoning. This is watchdog-reported patient feedback rather than a measured study. The vendor response quoted in coverage cited around 90% of callers describing an improvement, which is a satisfaction figure answering an accuracy complaint, and that mismatch is the pattern worth noticing.
What this changes about testing
Test the fields, not the transcript. The twelve-call checklist includes an accent call, and this is the deeper version of that test: place calls giving names the agent cannot have on a list, and check what reached the record rather than what the transcript says.
Then measure the right thing. Count how many bookings carry a wrong name or a wrong postcode, rather than how well the transcripts read. Grading against the system of record, described in the evaluation guide, is what catches this. An accountancy practice taking a client's name and postcode over the phone is capturing the two lowest-redundancy strings on the call, one after the other.
Frequently asked questions
Is entity error rate something I can ask a vendor for? Yes, and the answer is informative either way. Ask for entity error rate on names and on alphanumeric strings, measured on telephone-band audio, with the dataset named. A vendor that has the number will give it. A vendor that answers with a headline WER is telling you it has not measured the thing that breaks.
Would a better model just fix this? Not on the evidence above. The benchmark at the top of this article shows the entity-to-word multiplier persisting across four current systems, including ones with strong headline numbers. Better models move both figures down together; they do not close the gap between them, because the gap comes from names having no context to recover from rather than from the acoustic model being weak.
Should I ask every caller to spell their name? No, and doing it unconditionally is why people find these systems tiring. Spell only when confidence is low, which is the point of using the recogniser's own signal as the trigger. For a returning caller matched on their number you should not be asking at all.
Does an accent-specific model help? It can, and it is worth checking whether your vendor offers a British English variant on the model you actually deploy rather than on its flagship. Be aware of the trade-off in the Deepgram case above, where the telephony-tuned models carry no British English option at all. Test on real calls before assuming a locale setting solved anything.