The Latency Budget of a Voice AI Phone Call
Every voice agent answers with the same equation, whether its builders know it or not:
response delay = how long the system waits to decide you finished speaking, plus time to the LLM's first token, plus time to the TTS engine's first byte of audio.
LiveKit's documentation states it in exactly those three terms, and it is the only formula in this article. Nothing about model quality, nothing about transport protocols. When a voice agent feels slow, one of those three terms is where the milliseconds went, and it is almost never the one people optimise first. This is the pipeline we build and tune at DeskCaller, and this article is the map we wish had existed when we started.
What a caller's brain expects
The human baseline
The tolerances are documented in peer-reviewed work, and they are brutal. Stivers and colleagues measured turn-taking across ten languages in PNAS in 2009: the most common gap between one speaker finishing and the next starting falls in the 0 to 200 millisecond band, with cross-language means running from +7ms in Japanese to +469ms in Danish. Heldner and Edlund's corpus work in the Journal of Phonetics found the most common between-speaker interval is a gap of about 200ms, and roughly 40% of transitions are actually overlaps.
Humans hit those numbers by cheating. Levinson and Torreira showed that producing speech takes 600ms or more, so people predict the end of your turn and pre-plan their reply while you are still talking. A voice agent that waits for silence before it even starts thinking is already structurally behind the species it imitates.
Where tolerance breaks
The tolerance ceiling is documented too. Roberts and Francis found listeners' comfort with post-request silence breaking down around 700ms. Bögels, Kendrick and Levinson found that by 1,000ms of silence after a question, listeners' brains are already reinterpreting what the delayed answer will mean. And Shiwa's human-robot work found preference peaking at about one second, with spoken fillers buying tolerance beyond it, a result replicated for contextual fillers in a 2021 human-agent interaction study. Fillers are a legitimate engineering tool, not a hack.
The rule that traces to nothing
One number you will see everywhere deserves flagging: the "300ms rule", claiming users perceive a system as broken beyond 300ms. We tried to trace it to a paper or standard and could not. It appears only in vendor marketing citing other vendor marketing. The honest published anchors are 200ms (the human norm), roughly 700ms (tolerance starts degrading) and 1,000ms (interpretation changes).
The pipeline, stage by stage
A phone call to a voice agent crosses six stages each turn. The budget question is which stages hold the milliseconds.
| Stage | What happens | Where the time goes |
|---|---|---|
| Telephony ingress | Carrier delivers audio to your provider | Fixed by geography, tens of ms |
| Media transport | Provider streams audio to your app | Packetisation floor of 20ms per RFC 3551 |
| Speech-to-text | Transcription, then the endpointing decision | Transcription is fast. The decision to stop waiting is not |
| LLM | Reads the turn, starts generating | Time to first token, 100ms to nearly a second |
| Text-to-speech | First audio byte from the reply text | 40 to 300ms first-byte, vendor-claimed |
| Egress | Audio back through provider and carrier | Mirror of ingress |
The published end-to-end targets cluster tightly. Daily, who build Pipecat, publish an 800ms median voice-to-voice target and have demonstrated 500ms. Vapi's engineering blog sets a sub-500ms p50 with p95 under 800ms and states plainly that above 1,200ms conversational flow breaks. Retell publishes roughly 600ms in production against a 700ms "feels human" threshold. All vendor-published figures about their own stacks, but four independent stacks converging on the same half-second-to-a-second window, against the peer-reviewed 700ms tolerance ceiling, is as close to consensus as this field has.
Endpointing, the stage that actually dominates
Speech-to-text engines transcribe faster than people assume. Deepgram's own measurement documentation puts server transcription latency at 150 to 300ms with total client-observed latency of 200 to 500ms. AssemblyAI publishes roughly 300ms median word-emission latency for Universal-Streaming, from its own benchmark. Vendor figures both, but the scale is right: transcription is a few hundred milliseconds behind the audio.
The bet every detector makes
The expensive part is deciding that you have stopped, not transcribing what you said. An endpointing detector is a bet: wait longer and you interrupt nobody but add dead air to every single turn; wait less and you cut callers off mid-sentence. That wait is pure, deliberate latency, spent on every turn of every call.
The configured values, from the vendors' own docs
The configured values dwarf the transport layer. Deepgram's endpointing parameter defaults to a 10ms silence threshold, tuned for fast transcript finalisation rather than conversation, and its own conversational example raises it to 300ms, with voice-agent guidance commonly at 300 to 800ms. Its utterance_end_ms word-gap signal has a documented floor of 1,000ms, because interim results only arrive about once a second. AssemblyAI's turn-detection defaults are a 400ms minimum silence with a 1,280ms acoustic fallback, and AssemblyAI states outright that endpointing decision time, not transcription compute, is the dominant dead-air component for voice agents.
Two traps in the defaults
Two traps hide in the defaults. Deepgram's smart_format, which formats numbers and dates, is documented to hold finalisation for up to 3 seconds while it waits to see whether an entity is finished. A caller reading out a phone number is the worst case, and a phone agent hears phone numbers all day. Turn it off on live calls and format downstream. And Whisper deserves a specific caution: it is an offline model consuming 30-second windows, not a streaming engine, and the published Whisper-Streaming paper measured 3.3 seconds of average latency. Whatever its transcription quality, that is an order of magnitude outside a conversational budget.
The semantic turn detectors
The newer semantic approaches attack exactly this. Deepgram's Flux claims end-of-turn detection in 100 to 500ms and a 200 to 600ms saving against traditional STT-plus-VAD, and AssemblyAI's hybrid model reads meaning as well as silence. Vendor claims, but aimed at the correct stage.
The LLM and TTS terms, and what streaming changes
Once stages stream into each other, full generation time stops mattering. Only the leading edges count: the LLM's time to first token and the TTS engine's time to first audio byte. LiveKit's docs define agent latency as exactly that sum, and Pipecat instruments those two metrics by name.
Time to first token
Time to first token is the least controllable term. On ArtificialAnalysis, an independent benchmark under its own serverless conditions, GPT-4o measures about 0.89s to first token, GPT-4o mini about 0.94s, Claude 4.5 Haiku about 0.84s, and Gemini 2.5 Flash about 0.45s. Platform budgets assume 100 to 400ms, which is achievable with dedicated capacity and tight prompts, and the gap between those two sets of numbers is worth understanding before you promise anyone a latency figure. Vapi's engineering post names the LLM as "almost always" the bottleneck.
Time to first audio byte
Text-to-speech first-byte claims are the healthiest part of the market: ElevenLabs publishes about 75ms model latency for Flash with realistic websocket first-byte of 100 to 200ms by region, Cartesia markets sub-90ms for Sonic, and PlayHT documents 190ms for Play 3.0 mini. All first-party claims.
Streaming the two together
The technique that ties the terms together is streaming the LLM's output into the TTS engine as it generates, sentence by sentence or token by token, rather than waiting for the full reply. ElevenLabs' websocket input endpoint, Cartesia's continuations and Pipecat's text-frame aggregation are each vendor mechanisms for exactly this. One honesty note our research forced on us: no vendor publishes a measured A/B saving for this technique. Everyone documents the mechanism, everyone recommends it, nobody publishes the before-and-after. The structural logic holds anyway, since a two-sentence reply that takes 2 seconds to generate fully can start speaking after its first clause.
The transport myth
The stage that attracts the most optimisation enthusiasm is the one with the least room in it.
The delta nobody has measured
Twilio's Media Streams delivers 8kHz mulaw audio base64-encoded inside JSON over a WebSocket. Telnyx's equivalent sends a base64 RTP payload in JSON. Going to raw SIP/RTP instead removes the framing overhead and a TCP hop. How much does that actually buy? Here is the uncomfortable answer: nobody has published a measurement. We went looking for a credible engineering write-up quantifying the WebSocket-versus-RTP delta for voice agents and found none. The commonly repeated figure of a few tens of milliseconds traces to nothing.
The verifiable ceiling
What is verifiable is the ceiling. RFC 3551 sets the default audio packetisation interval at 20ms and states verbatim that the packetisation interval determines the minimum end-to-end delay. Audio arrives in 20ms increments regardless of what wraps it, and base64-plus-JSON framing on 20ms frames costs microseconds of CPU and some bandwidth, not tens of milliseconds of wall clock. The transport stage is bounded by small numbers while endpointing is configured in hundreds of milliseconds. Rebuilding your media layer to shave the former while utterance_end_ms sits at its 1,000ms floor is optimising the wrong end of the equation.
What you give up in the swap
And the swap is not free. A managed media stream arrives with jitter buffering, packet reordering and loss concealment already done. Run raw RTP yourself and an adaptive jitter buffer, typically holding around 40ms and growing under network stress, becomes your code, along with every reordered and dropped packet it was silently absorbing. You take on real operational surface to chase an unmeasured saving.
Where to spend your effort, in order
Instrument before touching anything. Pipecat exposes the exact metrics to watch, time to first answer token and time to first audio, plus an observer for user-stop-to-bot-start, and whatever your stack, that last number is the one the caller feels.
- Measure voice-to-voice at p50 and p95, from caller-stops-speaking to agent-audio-starts. This is the only number that matters and most teams have never measured it.
- Tune endpointing first. Check what your STT's silence threshold and turn-detection values actually are, not what you assume. Check
smart_formatstyle entity-formatting delays are off. - Check the LLM term. Measure your real time to first token under production prompts, not the benchmark number. Shorter prompts and smaller models move this term more than anything else you control.
- Stream everything. LLM tokens into TTS, first sentence out while the rest generates.
- Add fillers where the budget cannot shrink. A spoken "let me check that for you" is peer-reviewed latency tolerance, and it costs nothing.
- Leave the transport layer alone until everything above is measured and tuned. It is the smallest term and the most expensive to rebuild.
The same logic runs the other way when a turn needs verification before the agent speaks: checking a fact costs milliseconds from this same budget, and the hallucination guardrails guide covers how to spend them without leaving dead air. What the agent should do while a human's phone rings is the transfer guide's version of the same problem.
Frequently asked questions
What is a good end-to-end latency for a voice agent? The published targets from platform vendors cluster between 500 and 800ms voice-to-voice, with everyone agreeing conversation degrades near 1,200ms. The peer-reviewed human side says tolerance starts breaking around 700ms of silence and interpretation changes by 1,000ms. Treat under 800ms at p50 as competent and under 500ms as excellent, and always ask a vendor for p95, not the demo number.
Why does my agent feel slow when every component claims fast benchmarks? Because component benchmarks measure compute and your delay is dominated by waiting. Endpointing silence thresholds sit at 300 to 1,000ms or more by configuration, and they apply to every turn. Add an LLM time to first token measured under real conditions rather than a marketing page, and a pipeline of individually fast parts produces a slow conversation.
Does switching from Twilio WebSocket streaming to raw SIP/RTP reduce latency? By some amount that, as far as we can find, nobody has publicly measured. The framing overhead it removes is real but bounded by RFC 3551's 20ms packetisation floor, while the jitter buffering and loss handling you give up are real operational costs. If your endpointing and LLM terms are not already tuned and measured, this is the wrong place to start.
Is Whisper suitable for a real-time phone agent? Not in its standard form. Whisper is an offline model that processes 30-second windows, and the published streaming adaptation measured 3.3 seconds of average latency. Streaming-native engines with tunable endpointing are built for the job Whisper is being bent into.