Voice Agent Latency: Where the Seconds Go
Jaseem Abbas
- Published

Voice agent latency is the silence a caller hears between finishing a sentence and hearing the agent's first word. It is made of several parts, and most of them sit outside the AI. This guide walks through one reply in order, shows how to time it from the caller's own phone, and says what to do about each part.
Picture a two-wheeler dealer in Pune whose agent calls a lead about a test ride. The lead says, "Haan, Saturday chalega." On the dealer's dashboard the agent "replied fast". On the lead's phone there is a long silence. He says, "Hello? Hello?", and the agent's answer lands right on top of his second hello. The dashboard was not lying. It never saw the time spent on the phone line in either direction. The dealer, the lead and the call are an example we made up. They are not customer data.
What is latency in voice AI?
Latency in voice AI is the time from the end of the caller's speech to the start of the agent's first audio, as heard on the caller's phone. You will also see it called reply time or response time. The phone industry says mouth-to-ear delay, and Wikipedia's page on audio latency splits a phone call's delay into codec delay, playout delay and network delay (Wikipedia on latency in audio).
The important word is "heard". A server can log when audio arrived and when audio left, and that number can look good while the caller waits much longer. For a short overview of how one turn is split between listening, thinking and speaking, see the latency budget in our guide to what conversational AI is. This post is the deeper version, with the phone network included.
How fast do people expect a reply?
Faster than an agent can easily match. A study of ten languages by Stivers and colleagues found that most gaps between turns fell between 0 and 200 milliseconds, with a mean offset of about 208 milliseconds across the whole data set (Stivers et al., 2009). None of the ten languages was an Indian language, so do not assume Indian callers follow that average.
People manage this by not waiting. Levinson and Torreira note that gaps are about 200 milliseconds, yet producing speech takes over 600 milliseconds, so listeners must predict the end of a turn and start planning their reply while the other person is still talking (Levinson and Torreira, 2015). A voice agent mostly cannot start until it believes you have finished. That is why it needs tricks such as streaming, a short spoken acknowledgement or a hold line, and why the aim is "no gap the caller notices", not a magic number.
What is acceptable latency for voice?
The phone standard gives two reference points. ITU-T G.114 recommends that one-way delay not exceed 400 milliseconds for general network planning. It also says highly interactive tasks, such as many voice calls, can be affected by much lower delays, and that if delay stays below 150 milliseconds most applications will experience "essentially transparent interactivity" (ITU-T G.114).
Read that carefully. It describes the delay of the line itself, one way, before any AI is involved. An agent then adds its own time on top. So the standard tells you how much of your budget the phone network can eat. It does not give you a target for a whole agent reply.
Where do the seconds go on a phone call?
They go into seven parts, in this order. The table lists each one, what makes it slow and what you can change. We give no millisecond figure for any part, because we could not verify a figure for Indian phone calls and we will not invent one. Your own measurements, taken as described below, are the numbers that count.
Part: 1. Phone line in
- What happens
- The caller's voice is encoded, crosses the mobile network, the carrier and a gateway, and reaches your platform
- What makes it slow
- Codec conversions, extra hops between carrier and platform, jitter buffers
- What you can change
- Fewer hops, less transcoding, an agent close to the carrier's network
Part: 2. Wait for the caller to finish
- What happens
- The agent waits to be sure the caller is done
- What makes it slow
- A long wait set to protect callers who pause
- What you can change
- Set the wait per step, longer only where callers read numbers
Part: 3. Speech to text
- What happens
- The last words are turned into text
- What makes it slow
- Long final phrases, noisy or narrowband audio, a slow provider
- What you can change
- Test providers on your own calls, swap only the slow one
Part: 4. The model
- What happens
- The language model reads the text and starts its reply
- What makes it slow
- A long prompt, a long history, a slow time to first token
- What you can change
- Shorter prompts, shorter replies, a quicker model for simple steps
Part: 5. The voice
- What happens
- Text becomes speech, and the first audio is ready
- What makes it slow
- Waiting for a whole reply instead of its first sentence
- What you can change
- Stream the reply, make the first sentence short
Part: 6. A tool call (optional)
- What happens
- The agent asks another system for data, such as a slot or an order
- What makes it slow
- A slow API, several calls in a row
- What you can change
- Faster endpoints, fewer calls, a hold line while it runs
Part: 7. Phone line out
- What happens
- The agent's audio crosses the same path back to the caller
- What makes it slow
- The same hops, codecs and buffers as part 1
- What you can change
- The same fixes as part 1
Why does the phone line count twice?
Because G.114 describes one-way delay, and a turn crosses the line twice. The caller's last word travels to the agent. The agent's first word travels back. Every millisecond on the line is paid in both directions before any AI has run.
This is also why the dashboard misleads. A server's log starts when audio reaches it and stops when audio leaves it. Parts 1 and 7 happen outside that window. That is our reasoning from how the delay is defined, not a measurement of any platform. It is the reason to time a reply from the caller's side, as shown below.
What do codecs and jitter buffers add?
A codec squeezes voice into a stream and a jitter buffer smooths out the packets that arrive unevenly. Both help quality, and both cost time.
- The codec. G.711 is the classic phone codec: it samples at 8,000 times a second, uses 8 bits per sample, runs at 64 kilobits a second and passes roughly 300 to 3,400 Hz of sound (ITU-T G.711, Wikipedia on G.711). It is usually set up for the least encode-and-decode delay and is widely used on the phone network. Mobile calls use other codecs, such as AMR narrowband, which adds delay in encoding and decoding, and a call between two networks crosses two radio networks (Wikipedia on latency in audio). If your callers phone from mobiles, expect this on those calls. The same narrow phone audio also hurts accuracy, as our guide to Hindi speech to text on calls explains. Narrowband audio also gives the speech recogniser less to work with.
- The jitter buffer. Packets of audio do not arrive at an even pace, and a jitter buffer holds a few so playback is smooth. The largest jitter it can absorb is equal to the delay it adds, and adaptive buffers change that delay as the network changes (Wikipedia on jitter). RTP, the protocol that carries the audio, defines jitter as the variation in packet spacing at the receiver compared with the sender (RFC 3550).
The trade is simple: a bigger buffer means fewer gaps in the audio and more delay. A smaller one is quicker and may break up on a bad mobile line. Do not shrink it blindly. For how the audio is carried between the caller's device and the agent, see WebRTC vs WebSocket for voice agents.
One more point on the phone leg. SIP sets the call up, and the audio itself rides on RTP. Our glossary explains the difference between a SIP trunk and a phone line. On the carrier side, SIP (RFC 3261) is the signalling protocol for creating and ending sessions.
Why is the wait to be sure the caller is done a big slice?
Because the agent waits on purpose. It cannot tell a finished sentence from a pause, so it holds for a short time before it replies. That wait is time the caller hears as silence, and it is often the largest deliberate slice of the reply.
It also cannot be zero for Indian callers. People say "ek second, card dekh raha hoon" and then read a policy number in pieces, with pauses between the groups. An agent that replies at the first pause cuts them off. We cover how to set this wait, and the habits that trip it, in our guide to endpointing and turn detection. We stop there so we do not repeat it.
What do speech to text, the model and the voice each add?
Each one adds the time it takes to produce its first output, and none of them needs to finish its whole job before the next one starts. This is the part most vendor pages focus on. It is real, but it is only about half the story.
- Speech to text. The recogniser turns the caller's last words into text. Streaming speech to text can transcribe while the caller is still talking, so most of its work can be done before they stop and what remains is finishing the final phrase. Long final phrases, noise and narrowband audio make it slower and less accurate.
- The model. The measure here is time to first token: how long the language model takes to produce the first piece of its reply. The total time to write the whole reply matters less, because the voice can start on the first sentence. A long prompt and a long history make the first token later.
- The voice. The measure is time to first byte, here meaning the first audio: how long the text to speech voice takes to produce its first sound. If the voice waits for the whole reply to be written, the caller waits for all of it. If it starts on the first sentence, the caller waits for one sentence.
Two habits matter more than any provider choice. First, stream every stage, so each one passes work forward as soon as it has a first piece. Second, keep the first sentence short. A short reply such as "Ji, Saturday ko 11 baje" can start sooner than a long opening that restates the question. Our voice agent prompt guide covers how to write short replies. For the choice between one model for the whole turn and separate stages, see speech-to-speech vs cascaded voice agents.
Each agent on Bananaflow picks its own speech-to-text, language model and voice provider, and you can change any one of them without rebuilding the agent. That matters here because the slowest part is not always the same part, so you can swap the slow one and leave the rest.
How do you measure voice agent latency?
Measure it on a real phone call, from the caller's side. This is our suggested method, not a standard, and the sample size is a starting point.
- Call the agent from an ordinary mobile phone. Use the network your callers use, not office wifi.
- Record the call on the caller's end. A second phone on speaker, or a recorder, works. The recording must hold both voices.
- Mark two points in an audio editor. The end of the caller's last word, and the start of the agent's first audio. The gap between them is the latency the caller heard.
- Do 20 to 30 turns for each type of turn. A plain answer, a turn that triggers a tool call, the first greeting, and a number read aloud in groups. Each type has a different shape.
- Report the median and the slowest one in twenty. The median is the middle result: half the turns were faster, half slower. The slowest one in twenty, often written p95, shows the bad turns your callers remember.
- Compare with your platform's own logs. The difference between the two is the part the phone line adds, in and out.
Latency is one check inside a wider test plan. Our guide to how to test a voice agent covers test cases, run counts and scoring, and its section on what to monitor after launch is where repeat latency checks belong.
How do you cut latency without breaking the call?
Find the slowest part, change that part, and measure again. The table lists the common changes and what each one costs. It is advice from how these systems work, so test each change on your own calls.
Change: Keep the agent near the carrier's network
- Which part it helps
- Phone line in and out
- What it costs
- Less choice of where you host
Change: Use fewer hops between carrier and platform, and avoid transcoding
- Which part it helps
- Phone line in and out
- What it costs
- May limit which carrier routes you can use
Change: Leave the jitter buffer alone unless you can test on bad mobile lines
- Which part it helps
- Phone line in and out
- What it costs
- A smaller buffer is quicker but can break up the audio
Change: Set the endpointing wait per step
- Which part it helps
- The wait for the caller
- What it costs
- Too short and the agent interrupts, too long and the call drags
Change: Swap only the slowest provider
- Which part it helps
- Speech to text, the model or the voice
- What it costs
- You must retest turn-taking and accuracy after a swap
Change: Write short prompts and short first sentences
- Which part it helps
- The model and the voice
- What it costs
- Less room for long, careful replies
Change: Stream the reply so the voice starts on the first sentence
- Which part it helps
- The model and the voice
- What it costs
- Awkward if the first sentence must change after the rest is known
Change: Say a short hold line first when a tool runs
- Which part it helps
- A tool call
- What it costs
- It does not speed up the tool, it only explains the wait
Now the warning. Do not buy speed with turn-taking. If you shorten the wait until the agent speaks over a caller who is reading a number, the call feels faster for the callers who answer in one word and much worse for everyone else. A faster agent that cuts callers off is worse than a slower one that listens. Fix the parts that are slow for no good reason first, such as extra hops and long prompts, and leave the wait for last.
What should the agent say while a tool runs?
A short, honest hold line. Silence while a lookup runs sounds like a dropped call, and callers say "hello?" into it. A line such as "Ek second, main Saturday ka slot check kar rahi hoon" tells the caller the agent is working, and the caller waits.
Back to the Pune dealer. His agent checks the test-ride calendar with a tool before it confirms Saturday. With no hold line, the lead hears nothing while that request runs, and he says "hello?". With a hold line, he hears the agent say it is checking, so the wait feels intended. The hold line does not make the calendar faster. It makes the wait understandable.
On Bananaflow, the HTTP tool test panel shows how many milliseconds a request took, so you can see which tool is slow before a caller does. A hold line, typed or recorded, plays while the request runs on HTTP tools and app tools. You can also mix pre-recorded audio with generated speech. A recorded line needs no speech synthesis, which is general reasoning and not a promise of any speed gain. See the integrations page for how tools are set up.
What questions should you ask a voice AI vendor about latency?
Ask how the number was measured, because a figure with no method behind it tells you little. Use this checklist, whether you build or buy:
- Was it measured on real phone calls or in a browser?
- Is it a median, or also the slowest one in twenty?
- Which turn types does it cover: greeting, plain answers, tool turns?
- Which carrier route was used, and is it the route I will use?
- Does the number include the endpointing wait?
- Can I listen to an unedited call?
Then run the method above on your own calls. A vendor's best case and your callers' typical case are different numbers. Our guide to what conversational AI is makes the same point in one line: ask every vendor for its reply time, and then listen to a live call yourself.
How Bananaflow fits
Bananaflow is an AI voice agent platform. Each agent picks its own speech-to-text, language model and voice provider, and can change any one without rebuilding. Turn-taking is set per step: barge-in is chosen for each step, and there is a turn judge and a sound sensitivity setting, covered in our endpointing guide. You can test any agent in the browser by voice or text, and after each call you get a transcript and a recording.
Telephony runs through the carriers on our telephony page, and you bring your own numbers. A figure measured in a browser would hide the phone line, which is the reason for the method above. Test on your own calls.
What should you remember about voice agent latency?
- Latency is the gap the caller hears. Time it from the caller's phone, not from a dashboard.
- One reply has seven parts: the phone line in, the wait to be sure the caller is done, speech to text, the model, the voice, an optional tool call and the phone line out.
- The phone line counts twice per turn, and G.114 calls one-way delay below 150 milliseconds essentially transparent and 400 milliseconds the planning limit.
- Codecs and jitter buffers add delay on the phone leg. A bigger buffer means fewer gaps and more delay.
- People answer in about 200 milliseconds by predicting the end of a turn. An agent needs tricks: streaming, short first sentences and hold lines.
- Report a median and the slowest one in twenty, on 20 to 30 turns of each type.
- Never cut the endpointing wait so far that the agent interrupts callers.
FAQ
Frequently asked questions
What is voice AI latency?
Voice AI latency is the silence a caller hears between finishing a sentence and hearing the agent's first word. It is also called reply time or response time. It is the sum of the phone line in, the wait to be sure the caller is done, speech to text, the language model, the voice and the phone line out.
What is acceptable latency for a voice agent?
There is no single number that fits every call. People take turns in about 200 milliseconds on average, but they predict the end of a turn, which an agent cannot do as well. The phone standard ITU-T G.114 calls one-way delay below 150 milliseconds essentially transparent and 400 milliseconds the general planning limit, and an agent adds its own thinking time on top. Aim for no gap the caller notices, and judge it by listening to real calls.
Why is my voice agent slow on phone calls but fast in the browser?
A browser test skips the phone network. On a real call, the caller's voice crosses a mobile network, a carrier and a gateway before the agent hears it, and the reply crosses the same path back. Codecs and jitter buffers add delay on that path too. Your platform's logs usually start when audio arrives and stop when audio leaves, so they miss both legs.
How do you measure voice agent latency?
Call the agent from an ordinary mobile phone and record the call on the caller's side, for example with a second phone. In an audio editor, measure from the end of the caller's speech to the start of the agent's first audio. Repeat for 20 to 30 turns of each kind and report the median and the slowest one in twenty. This is our suggested method, not a standard.
How do you reduce voice agent latency?
Find the slowest part first, then change that part. Keep the agent close to the carrier's network, avoid extra hops and transcoding, tune the endpointing wait per step, keep replies short so the first sentence is quick, and swap the slowest provider instead of the whole stack. Do not trade away turn-taking for speed: an agent that cuts callers off is worse than one that is a little slower.
What is time to first token?
Time to first token is how long a language model takes to produce the first piece of its reply. It matters more than the total time to write the whole reply, because a voice can start speaking as soon as the first sentence is ready. The matching measure for the voice is time to first byte, here meaning the first audio.
Does a hold line help?
Yes, when a tool call makes the agent wait. A short line such as "Ek minute, main aapka order dekh rahi hoon" tells the caller the agent is working, so silence does not read as a dropped call. It does not make the lookup faster. It makes the wait understandable. On Bananaflow you can type a hold line or record one for HTTP tools and app tools.




