Barge-In, and Why Your Voice Agent Talks Over People

Interruption handling fails in two opposite directions, and most stacks expose one dial for both. Make the agent easier to interrupt and it stops for coughs, backchannels and the television. Make it harder and it steamrolls a caller who is saying "no, wait". Teams tune that dial for weeks because the dial is the wrong abstraction: interrupting is a classification problem on the way in, and a cancellation problem on the way out, and the second half is the one almost nobody engineers.

This is the pipeline we work on at DeskCaller, and this guide covers both halves with the actual parameter names and defaults, because the folklore in this space is thick.

Four sounds, one microphone

While your agent is speaking, the caller's line can carry four different things, and only one of them means "stop talking".

What the mic hears	What it means	What the agent should do
"No, wait, actually..."	A real interruption	Stop within a syllable, yield the floor
"Mm-hmm", "yeah", "okay"	A backchannel, the caller signalling keep going	Keep talking, never stop
Someone else in the room	Third-party speech, not addressed to the agent	Ignore it, and keep it out of the transcript
A cough, a chair, the TV	Noise	Nothing

Backchannels are not an edge case. The term goes back to Yngve's 1970 linguistics paper, and in the Switchboard corpus of American telephone conversation, backchannel acknowledgements are roughly 19% of all utterances, the second most common thing people say. A stack that treats every voiced sound as an interruption stops mid-sentence a fifth of the time for callers who were agreeing with it. The linguistics literature even splits overlap by intent, cooperative versus competitive, which is exactly the classification your agent is being asked to make in real time.

The failure modes split the same way. Stopping for noise and third-party speech is an acoustic false positive, fixed on the input side. Talking over a real interruption is usually an output failure, and no input threshold fixes it. One dial cannot tune both, which is why tuning it never converges.

Barge-in was standardised in 2004

Interruption handling is not a new problem, and the old solution is instructive. The W3C's VoiceXML 2.0 recommendation, from March 2004, gave IVR prompts a bargein attribute and a bargeintype with two documented modes: speech, where any voice stops the prompt, and hotword, where only recognised phrases do. The acoustic groundwork is older still: adaptive echo cancellation, subtracting a synthesised copy of the outgoing audio from the incoming line, was published by Sondhi at Bell Labs in 1967, and research on intelligent barge-in in dialogue systems dates to at least Strom and Seneff in 2000.

Why it got hard again

Classic barge-in was tractable because the prompt was a known, pre-recorded file: cancel its echo, detect voice, cut the prompt in the DSP. An LLM agent made the problem harder in three ways at once. The outgoing speech is generated on the fly, the reply is streaming through several services with their own buffers, and the conversation state has to be repaired after the cut. Echo cancellation itself is mostly a solved input on phone calls, handled by the handset and network, and by getUserMedia's echo cancellation on the web. Without it, an agent hears its own voice, transcribes it, and interrupts itself.

The input side, deciding that a real interruption happened

What a VAD can and cannot tell you

A voice activity detector answers one question, is someone making speech-like sound, and nothing else. Silero VAD, the open-source default in much of this stack, is a roughly 2MB model classifying 32ms chunks, deliberately blind to meaning and speaker. WebRTC's VAD is older, a Gaussian mixture model from the 2011-era codebase, and its documented weakness is exactly the one that hurts here: good at telling silence from sound, poor at telling speech from noise. A VAD can tell you the line went loud. It cannot tell you whether that was "stop" or "uh-huh" or the television.

The two upgrades that actually help

The first is cleaning the input before anything judges it. Speaker isolation ahead of the STT removes third-party voices and background speech so the VAD and transcriber only ever see the caller. Krisp, the best-known vendor here, claims 3.5 times fewer VAD false triggers with its background voice cancellation in front of the pipeline, a vendor figure, but the architecture point stands independently: fix the input before tuning any threshold downstream, because thresholds tuned against a dirty input are compensating, not converging.

The second is judging turn completion semantically rather than acoustically. LiveKit's open turn detector is a 0.5B-parameter transformer reading the live transcript to score whether the utterance is complete. Pipecat's smart-turn v3 is an 8M-parameter open model on native audio, inferring in about 10 to 12ms on CPU. Deepgram's Flux builds end-of-turn into the ASR model itself, and AssemblyAI exposes an end-of-turn confidence score. These sit alongside the VAD, not instead of it. The silence thresholds and endpointing arithmetic they replace are covered in the latency budget guide, because every millisecond of turn detection is also a millisecond of response delay.

What the platforms actually expose

The settings exist, under five different names. All defaults below are from each vendor's current API documentation.

Platform	The barge-in controls	Defaults
Vapi	stopSpeakingPlan: numWords, voiceSeconds, backoffSeconds, plus acknowledgementPhrases and interruptionPhrases lists	0 words, 0.2s of voice, 1s backoff
Retell	interruption_sensitivity, 0 to 1, plus enable_backchannel	1
ElevenLabs Agents	interruption client event toggles barge-in entirely, interruption_ignore_terms list, turn_eagerness, disable_first_message_interruptions	eagerness normal, first-message interruptions on
Bland	block_interruptions, interruption_threshold, interruptibility 0 to 3	off, 500ms, 2
OpenAI Realtime	turn_detection: server_vad (threshold 0.5, silence_duration_ms 500) or semantic_vad (eagerness), interrupt_response	server_vad, interrupts on

Two details in that table deserve more attention than the dials. Vapi's acknowledgementPhrases ships with "yeah", "uh-huh" and "mm-hmm", words that never interrupt, while interruptionPhrases ships with "stop", "no", "wait" and "actually", words that always do. That is a backchannel filter and a hotword list, VoiceXML's 2004 bargeintype design reborn with better ears, and it is the first barge-in feature worth checking any platform for. The caller who says "wait, actually" is not adding to their sentence, they are restarting it, and a phrase list catches that where no threshold can.

And ElevenLabs' disable_first_message_interruptions, with Bland's block_interruptions, answer a question the tuning debate skips: some speech should not be interruptible at all. A legal disclaimer or a safety instruction is exactly where you want the floor held.

The output side, where interrupts go to die

Here is the failure that no input tuning touches. The classifier fires correctly, the decision is made, and the agent keeps talking anyway, because seconds of synthesised audio are already sitting in buffers between your code and the caller's ear. Your interrupt latency includes every downstream buffer, and few of them are documented.

The cancellation chain

Cancellation is a chain, and every link is a real API. Stop the LLM generation. Stop the TTS stream. Flush the audio already handed to the transport: on Twilio bidirectional media streams that is the clear message, a one-line JSON event that discards everything Twilio has buffered for playout, and the mark message is how you learn what had already been played. On OpenAI's Realtime API it is response.cancel. Frameworks that take this seriously keep the egress buffer tiny in the first place: Pipecat's output transport buffers four 10ms chunks by default, 40ms of committed audio, precisely so there is almost nothing to flush.

The pattern worth adopting where you control the chain: duck the output the instant the acoustic signal fires, then decide with the transcript a beat later. Lowering the volume is cheap to undo. Having talked over someone for two seconds is not.

Context repair, the invisible half of an interrupt

After a cut, your transcript says the agent delivered a sentence the caller never heard the end of. If the conversation context keeps the full sentence, the model believes it said things it did not, and two turns later it references them. The caller experiences an agent confidently building on words that were never spoken.

The serious runtimes repair this explicitly. OpenAI's Realtime API has a dedicated event for it, conversation.item.truncate, which cuts the stored item back to what was actually played, and its docs note the split: over WebRTC the server truncates automatically, over WebSocket your client must do it. LiveKit truncates the agent's chat history to the played portion using the playback position reported by its audio output. Pipecat synchronises by playback too, with a rule worth writing on the wall: text that never played never reaches the assistant context. If your stack cannot say where playback stopped, it cannot do this, and that question belongs in your platform evaluation.

Silence is not one signal either

The mirror problem. The caller goes quiet for five seconds, and the agent must choose between waiting, prompting, and hanging up, with three indistinguishable causes: thinking, reading a card number off a card, or gone.

The platforms expose patience, not telepathy. ElevenLabs waits 7 seconds by default before re-engaging, with a separate disabled-by-default timer for ending the call on extended silence, and a spelling_patience setting for exactly the card-number case. OpenAI's semantic_vad trades eagerness against wait time, up to 8 seconds at its most patient. The context-aware move is better than any global value: if the agent just asked for a card number, a postcode or a spelling, lengthen the wait, because the pause is the caller complying. A short spoken check-in beats both waiting forever and barging in, and if the silence turns out to be a side conversation with someone in the room, the answer it produces must be kept out of the model's context, or the agent will respond to a conversation it was never part of.

The duplex horizon, honestly stated

The architectural criticism is fair: a cascaded pipeline listens, then thinks, then speaks, and a mid-generation interruption discards the thought. Full-duplex models attack that directly. Kyutai's Moshi models both sides of the conversation as parallel audio streams with no turn gate at all, and NVIDIA's PersonaPlex, often misattributed to Kyutai, is a 7B model built on Moshi's architecture. Microsoft's Azure Voice Live documents semantic barge-in control and automatic truncation as platform features, and OpenAI's Realtime API is speech-to-speech end to end.

The honest state: duplex models dissolve the turn-taking gate, and production phone stacks still overwhelmingly run cascaded pipelines, where tool calling, guardrails and observability live. For those, the engineering in this article is the game. Classification on the way in, small buffers and fast flush on the way out, context repaired to match what was heard.

Test it like callers actually behave

A polished demo proves nothing about barge-in, because nobody talks over a demo. Run these eight against any agent, yours or a vendor's, and score what actually happens on the wire.

1. Talk over the greeting from the first word. Does it stop, and how fast?
2. Say "mm-hmm" and "yeah" while it explains something. It must not stop.
3. Clear your throat mid-sentence. It must not stop.
4. Have a TV or radio talking in the background for the whole call. Count false stops.
5. Have someone else in the room ask you a question, and answer them. The agent must not respond to that exchange, and it must not appear in the context.
6. Say "wait, actually" and change your request halfway through. Does it treat that as a restart, or bolt it onto the old request?
7. Interrupt the agent, then ask "sorry, what were you saying?" A context-repaired agent resumes cleanly; a broken one refers to words it never finished saying.
8. Ask it to wait, put the phone down for ten seconds, come back. Does it hold, prompt gently, or hang up?

Tests 5 and 7 separate real engineering from a tuned demo, because they test the transcript and the context, not the ears. What the agent should do once it correctly yields to "let me speak to a person" is the transfer guide's territory.

Frequently asked questions

Why does my agent keep talking for a second after being interrupted? Because the interrupt decision only stops new audio from being generated, and the audio already handed downstream keeps playing. The fix is at the transport: flush the buffered audio explicitly, Twilio's clear message being the canonical example, and keep the egress buffer small so there is little to flush. Measure from caller-starts-speaking to agent-audio-actually-stops, not to when your code decided.

What stops the agent halting every time the caller says "yeah"? A backchannel filter. Vapi ships acknowledgementPhrases, ElevenLabs ships interruption_ignore_terms, Retell ships enable_backchannel for the agent's own listening noises. If your platform has none of these, short affirmations will keep tripping barge-in and no sensitivity dial will fix it, because the words are genuinely speech.

Should some messages be uninterruptible? Yes, sparingly. Legal disclaimers, safety instructions, and the readback of a booking you are about to commit. ElevenLabs exposes disable_first_message_interruptions, Bland exposes block_interruptions. Everything else should yield, since a caller interrupting is information.

Will raising the interruption threshold fix false stops? Usually not, because false stops and talked-over callers are different failures sharing one dial. Stops on noise and background voices are an input-cleanliness problem, solved ahead of the detector with speaker isolation. Talking over real interruptions is an output-buffer problem. Raising the threshold trades one failure for the other, which is why threshold tuning never seems to finish.