Why voice AI still trips up on accents, noise and interruptions

Voice AI can hold a sensible phone conversation. It still mishears a Glaswegian caller on a building site with an angle grinder running. That isn't a bug someone forgot to fix — it's the shape of the technology, and it affects every product in the category, ours included. Here's what's actually going wrong, when it matters and how to set a phone line up so the failures are small ones.

The three-step pipeline, and why step one decides everything

A voice agent isn't one thing. It's three jobs chained together:

1. Speech to text. The audio coming down the phone line gets transcribed into words.
2. Reasoning. A language model reads that text and decides what to say or do next — answer the question, take a name, book the slot.
3. Text to speech. The reply is synthesised back into a voice and played to the caller.

Notice what step two receives. Not the caller's voice. Not their tone, not the hesitation, not the fact they clearly said "Hawthorne" and not "four-time". It receives a line of text that another system guessed at. If that guess is wrong, the clever part of the system is now reasoning confidently about the wrong words.

That's the whole story in one sentence: accuracy problems mostly start before the AI understands anything. Everything else in this post follows from it.

Accents: the problem isn't "strong", it's "different from the training data"

Transcription models learn from enormous amounts of recorded speech. They get good at the patterns they've heard a lot of. Standard southern English, read clearly into a decent microphone, is extremely well represented. A broad Geordie accent, a Doric speaker from the north-east of Scotland, a Birmingham caller speaking quickly, or someone whose first language isn't English — all of these are further from the centre of the distribution, so the model's guesses get shakier.

It's worth being precise about where this actually bites, because it isn't everywhere:

So the headline isn't "AI can't understand accents". It's that an accent widens the gap between easy content and hard content. A Scouse caller saying yes is fine. A Scouse caller spelling out "Fazakerley" is where you'd want a human, or a very careful confirmation step.

Background noise: a workshop is harder than a motorway

Noise suppression is standard in any serious voice product — ours and everyone else's. It works by separating what looks like speech from what looks like steady background hum. That means it handles some noise far better than others.

• Steady, predictable noise — road rumble, a van engine, a fan, air conditioning — is the easiest case. It's consistent, so it can be modelled and stripped out.
• Sudden, sharp noise — a dropped scaffold pole, a nail gun, a dog — is harder. It swamps a fraction of a second of speech, and that fraction might be the digit in a phone number.
• Other human speech is the hardest of all. Noise suppression is tuned to keep speech. A colleague talking two metres away, a radio with a presenter on it, a busy market stall — the system has no clean way to know whose voice it should care about.

This is why a caller phoning from a train carriage often comes through fine while a caller in a quiet-sounding café comes through badly. It isn't volume. It's whether the interfering sound looks like speech.

Mobile networks add their own layer. Phone audio is compressed hard — it always has been — and a weak signal mangles it further. If a caller sounds robotic to your ear, the transcription model is working from the same degraded audio you are, with no ability to say "sorry, you're breaking up a bit" unless it's been built to.

Interruptions: the turn-taking problem

Humans interrupt each other constantly and barely notice. We overlap, we back-channel ("mm", "yeah", "right"), we start a sentence before the other person finishes. Phone conversation is full of this.

A voice agent has to make a decision roughly every few hundred milliseconds: has the caller stopped talking, or just paused? Get it wrong one way and the agent talks over someone mid-sentence. Get it wrong the other and there's a dead gap that makes the caller say "hello? hello?" — and then they've spoken over the agent's reply anyway, and now two audio streams are tangled.

Barge-in handling — letting the caller cut the agent off mid-sentence — makes this better and is widely implemented. But it introduces its own failure: a cough, a passing siren or a loud "yep" can trigger it, cutting the agent off mid-question. You've probably experienced this with a human on a bad line too. The difference is a human recovers gracefully and the machine needs to be explicitly designed to.

The three mitigations that actually exist

There's no secret technique here, and anyone claiming a solved problem is selling you something. What serious products do is the same three things:

1. Noise suppression on the incoming audio. Cleans up steady background noise before transcription. Helps a lot with engines and hum, less with other voices.
2. Confirm rather than guess. When something is ambiguous — a number, a name, a postcode — ask the caller to repeat it or read it back for confirmation. Slightly slower. Far fewer wrong callbacks.
3. Short, structured call flows. Yes/no questions, a name, a callback number, a one-line description of the job. Structured answers are much easier to get right than open-ended ones.

Point two is the one worth insisting on when you're comparing providers. A system that guesses silently produces a message log that looks perfect and is wrong. A system that confirms produces a slightly longer call and a number you can actually ring back. Ask any provider what happens when the agent isn't sure — the answer tells you a lot. There's a fuller list of things worth asking in our nine questions to ask a call handling service.

Do the maths on what an error actually costs you

The same arithmetic runs the other way. A line that mis-hears one caller in twenty, but answers twenty callers you'd otherwise have sent to voicemail, is still ahead — because voicemail's accuracy on a missed call is zero. We've laid out that comparison properly in AI receptionist vs answering service vs voicemail. The point of this post isn't that voice AI is unreliable. It's that you should know which parts are unreliable, so you can design around them.

How to test a voice agent properly before you commit

Most people test by ringing their own line from a quiet office and saying "hello, I'd like to book an appointment" in their clearest voice. That tests nothing. Try this instead:

1. Ring from the noisiest place you actually work — the van with the engine on, the workshop, the salon floor mid-blow-dry.
2. Give a mobile number quickly, the way a real caller would, and check whether the agent reads it back.
3. Say a genuinely awkward local street name or surname and see what lands in the message.
4. Interrupt the agent mid-sentence. See whether it stops cleanly or ploughs on.
5. Get someone with a different accent from yours to do the whole thing again.
6. Ring from a spot with one bar of signal.

Then look at the logged messages rather than trusting your memory of the call. If you can see what the agent wrote down, you can see exactly where it degrades. Our walkthrough of what actually happens when an AI receptionist answers covers what that log looks like in practice.

Where voice AI is the wrong tool

Honestly: there are calls you should not route to any voice agent, ours included.

• Callers who are distressed or in an emergency. People in a panic speak fast, overlap, repeat themselves and don't answer the question asked. That's the exact combination of conditions described above. These calls need a human, and they need a fast route to one.
• Environments where the caller is always in heavy noise. If most of your enquiries come from people standing on an active site or a market, you're fighting the physics on every single call rather than occasionally.
• Highly specialist vocabulary you haven't configured. Clinical terms, part numbers, breed names, obscure product codes. A model that's never seen the word will transcribe something phonetically close and plausible, which is worse than a blank.
• Long, open-ended consultations. If the value of the call is the conversation itself — a detailed clinical triage, a complex spec discussion — structure can't save you, because the whole point is that it's unstructured.
• Customers who simply don't want it. Some people hang up on a machine regardless of how well it works. We've written about that at some customers really don't want to talk to an AI.

The honest framing is that voice AI is good at a specific job: catching calls that would otherwise ring out, taking the handful of facts you need, and getting them to you quickly. It is not good at being the only way anyone can ever reach you. If a caller is struggling, the right outcome is a clean handover or a prompt call back — not a cleverer attempt at guessing.

What will and won't improve

Transcription models keep getting better on accented speech as training data broadens, and they will keep improving. That's a genuine trend and you'll feel it over the next few years without doing anything.

What won't change much is the physics. A phone line carries compressed, band-limited audio. When two people speak at once, the information about who said what is partly gone before anything digital touches it. No vendor gets around that, and when you read comparisons of the various voice agent platforms, it's worth holding that in mind — they're all drinking from the same pipe. The differences between products are mostly in how gracefully they fail: whether they confirm, whether they hand over, whether they tell you when they weren't sure.

That's the thing to judge on. Not whether a system ever mis-hears — it will — but whether it admits it.