Endpointing and Turn Detection in Voice AI
Jaseem Abbas
- Published
- Updated

Endpointing is how a voice agent decides that the caller has finished speaking, so it can reply. Get it wrong in one direction and the agent talks over people. Get it wrong in the other and callers sit in silence and say "hello?". This guide explains endpointing, turn detection, voice activity detection and barge-in in plain words, shows why Indian callers trip them up, and gives advice on tuning and testing.
Here are two calls from the same agent. A clinic in Lucknow calls Mr Verma to confirm a cashless visit. He says, "Haan... ek second, card dekh raha hoon," and starts reading his policy number: "4410... 27..." At the pause the agent says, "Sorry, I didn't catch that. Could you repeat your policy number?" He sighs and starts again. On another call, a patient says "Yes, Thursday" and then hears two seconds of nothing before the agent answers. She says, "Hello?"
The cause is the same on both calls: one setting decides when a turn ends. It was too eager for Mr Verma and too patient for the second caller. The clinic, the patients and the numbers are an example we made up. They are not customer data.
What is endpointing?
Endpointing is the decision that the caller's turn is over. It is also called endpoint detection or end-of-turn detection. The agent runs it continuously while the caller speaks, and the moment it says "done", the agent starts its reply.
It can fail in two ways:
- Too early: the agent starts talking while the caller is still thinking or reading out a number, so it cuts in.
- Too late: the agent waits well after the caller finished, so the call has dead air and the caller wonders whether anyone is there.
Both are about timing, not about what the agent knows. For the short version of this idea inside a wider overview, see the turn-taking section of our guide to what conversational AI is. This post is the deeper version.
What is the difference between VAD, endpointing and turn detection?
They are three layers of one job. VAD asks "is there speech right now?". Endpointing asks "has this speaker finished?". Turn detection asks "whose turn is it, and should the agent talk or stay quiet?".
Voice activity detection checks short slices of audio and labels each one as speech or not speech. Wikipedia describes it as detecting the presence or absence of human speech, and notes that real algorithms trade latency, sensitivity, accuracy and computing cost (Wikipedia on voice activity detection). It is old technology on phone lines. ITU-T G.729 Annex B, from 1996, is a silence compression scheme for a phone voice codec, built on the same idea (ITU-T G.729 Annex B). Our voice activity detection glossary entry has the full definition, so we do not repeat it here.
Endpointing uses VAD's output, and sometimes more, to decide the turn is over. The simplest endpointer says: if there has been no speech for a set time, the turn has ended.
Turn detection is the widest layer. It includes endpointing and also covers what happens when the caller starts speaking while the agent talks, and small sounds like "hmm" that are not a real turn.
Layer: Voice activity detection
- The question it answers
- Is there speech in this slice of sound?
- What it uses
- The sound only
- How it fails
- Counts traffic or a TV as speech, or misses a soft voice
- What helps
- Adjust how much sound counts as speech
Layer: Endpointing
- The question it answers
- Has this speaker finished?
- What it uses
- VAD plus a silence wait, sometimes the words
- How it fails
- Cuts in at a pause, or waits too long after the end
- What helps
- A longer wait where callers pause, a words-so-far check
Layer: Turn detection
- The question it answers
- Whose turn is it, and should the agent stop or go on?
- What it uses
- All of the above plus the caller starting to speak
- How it fails
- Stops for a "haan", or keeps talking over a "ruko"
- What helps
- Decide per step who may cut in, and which words count
A turn decision can combine three signals, and the better systems use more than one.
Why do voice agents interrupt callers or wait too long?
Because a pause is not an ending, and a silence timer cannot tell them apart. Callers stop speaking for many reasons: to think, to find a card, to read a number, to check with someone in the room. The audio looks the same as a finished answer.
The table maps the symptoms a business owner usually hears to the likely cause. This is advice from how these systems work, not a diagnosis of your setup.
What you hear: The agent talks over a pause in the middle of an answer
- Likely layer
- Endpointing: the wait is too short
- What to change
- Lengthen the wait on that step, and add a words-so-far check
What you hear: Dead air after a short answer like "Yes, Thursday"
- Likely layer
- Endpointing: the wait is too long, or there is no check that the sentence is complete
- What to change
- Shorten the wait on steps that expect short answers
What you hear: The agent stops for "haan" or "hmm"
- Likely layer
- Turn detection: backchannels count as interruptions
- What to change
- Make sure acknowledgement words never stop the agent
What you hear: The agent stops when nobody spoke
- Likely layer
- Voice activity detection: noise counts as speech
- What to change
- Lower the sound sensitivity, then retest on a noisy line
What you hear: The agent keeps talking when the caller says "ruko"
- Likely layer
- Turn detection: real stop words are not recognised
- What to change
- Add the stop words, in the languages your callers use
What you hear: The caller says "hello?" after their answer
- Likely layer
- Endpointing or VAD: the end was never detected
- What to change
- Check noise first, then the wait
How fast do people take turns?
Faster than most people think. A study of ten languages, by Stivers and colleagues, found that most transitions between turns fell between 0 and 200 milliseconds, with a mean response offset of about 208 milliseconds across the whole data set (Stivers et al., PNAS 2009). Japanese was the fastest at +7 milliseconds and Danish the slowest at +469, and each language's mean sat within about 250 milliseconds of the overall mean.
There is a catch for an agent. Levinson and Torreira point out that gaps are short, about 200 milliseconds, but producing speech takes far longer, over 600 milliseconds. So listeners must predict the end of the other person's turn and plan a reply while still listening (Levinson and Torreira, 2015). People do not wait for silence. They see the end coming.
Two lessons follow. First, a silence timer is a weak copy of what people do, so it needs more help than a timer alone. Second, be honest about what the number means. None of the ten languages in the Stivers study was an Indian language, so do not assume Indian callers follow its average. We found no published measure of turn gaps for Hindi, Tamil or Hinglish phone calls.
How do Indian callers' speech habits trip up turn detection?
Several everyday habits look like an ending to a silence timer. These are patterns we see in calls and in our tests, not statistics, and the fix depends on your callers, so test for them.
- Asking for time: "ek second", "one minute", "hold on". The caller is not ending a turn. They are asking you to wait. A good agent agrees in a word ("Sure, take your time"), then listens, and answers whatever comes next, such as "ek second... fees kitni hai?".
- Fillers that hold the floor: "haan... matlab...", "woh... kya hai na...", "actually...". The sentence is not finished, but there is a gap. A silence-only endpointer jumps in. A words-so-far check sees an unfinished thought and keeps listening.
- Numbers read in groups. A policy, loan or order number is often read off a card or an SMS in pieces: "560... 0... 34...". Each pause between pieces looks like an ending. Steps that collect numbers need a longer wait than steps that collect a yes or no.
- "Haan", "hmm", "accha", "ji" while the agent talks. These are backchannels, the small sounds that show someone is listening. They are not interruptions. "Ruko", "wait" and "nahi nahi" are real stops. Telling these apart is a word-level job, so it is not something VAD can do.
- Noisy lines. Traffic, a TV, a shop counter or a speakerphone in a car. Noise can look like speech, so the agent stops for nothing, or it never hears silence and answers late. We cover the noise case in our glossary entry on voice activity detection.
- Mixing languages in one sentence. A caller may say "mera policy number hai four four one zero" in one breath. A check that decides whether a sentence is finished has to understand Hinglish, not only English. In our tests, a turn check given English-only examples still misjudged Hindi turns.
Our guide to Hinglish voice agents covers the language side, including how the agent follows a caller who switches between Hindi and English.
What is semantic turn detection?
Semantic turn detection decides whether a turn is over by looking at the words so far, not only at the silence. "Thursday works for me" is a finished thought. "My number is 4410" is not, whatever the silence after it. It is sometimes called end-of-utterance or end-of-speech detection.
The idea has research behind it. Ekstedt and Skantze's TurnGPT paper showed that whether a sentence is syntactically and pragmatically complete helps predict where a turn shifts, and that a language model of the dialogue so far beat two baselines (TurnGPT, 2020). A later paper from the same authors modelled turn-taking by predicting both speakers' voice activity a little ahead of time, trained without labels, and including backchannels (Voice Activity Projection, 2022). Both are research, not a product promise.
Semantic turn detection has a cost, and you should weigh it:
- It adds a little time, because something has to read the words.
- It must understand every language the agent speaks. A check that is good in English and weak in Hindi will keep listening when it should reply, or reply when it should keep listening.
- It still needs a fallback. If a caller says something odd, the agent cannot wait forever, so a time limit still applies.
What is barge-in, and when should a caller be able to cut in?
Barge-in is when the caller speaks while the agent is talking, and the agent stops to listen. In voice AI, that is the meaning, and it has nothing to do with the dictionary phrase. When it works, the agent stops, listens and answers what the caller just said. For a short definition, see our glossary answer on what barge-in is. For tuning it in practice, read barge-in on voice agents.
The hard part is deciding what counts. A real barge-in is the caller taking the floor. A backchannel is the caller saying "haan" to show they are following. If every "haan" stops the agent, the agent sounds jumpy and loses its place. If nothing stops it, the agent talks over a caller who is trying to correct it.
Bananaflow keeps per-language lists of words that never interrupt, like "haan" and "mhm", and words that stop the agent, like "ruko". There are starter lists for all 11 languages, and you can edit them. We explain the lists in our haan and ruko section.
When should you stop a caller from cutting in?
Allow barge-in on most steps, and make it stricter on a few. This is our advice:
- A required disclosure. A line that must be heard in full, such as a recording notice or a payment term, is a good place to restrict cutting in.
- The opening line. A short greeting that gets talked over on the first second can leave both sides confused.
- Steps where the caller asks questions. Leave these open, because the caller will want to interrupt when they already know the answer.
On Bananaflow, barge-in is set for each step of the call, so you can make these choices step by step. That is a product fact. The rules above are our advice on how to use it.
How do you tune endpointing without making it worse?
Change one setting at a time, change it on the step that has the problem, and retest with real voices. Most failed tuning comes from changing everything at once, so no one can tell what helped.
There are four settings to think about, and each has a cost on both sides.
Setting: Wait before replying
- Too eager or too sensitive
- Cuts in at pauses
- Too patient or too strict
- Dead air after short answers
- Who suffers
- Callers who pause, or callers who answer fast
Setting: How much sound counts as speech
- Too eager or too sensitive
- Stops for noise
- Too patient or too strict
- Misses a soft voice
- Who suffers
- Callers on noisy lines, or callers who speak quietly
Setting: How easily the caller can cut in
- Too eager or too sensitive
- Stops for every "haan"
- Too patient or too strict
- Talks over "ruko"
- Who suffers
- Callers who are only listening, or callers who are correcting
Setting: A words-so-far check
- Too eager or too sensitive
- Answers an unfinished sentence
- Too patient or too strict
- Waits when the sentence was already done
- Who suffers
- Callers who read numbers, or callers with short answers
What steps should you follow to tune it?
- Find the step with the problem. A number step, a yes or no step and a greeting each need different waits. Do not apply one setting to the whole call.
- Listen to recordings and read transcripts. The transcript shows where the agent replied. The recording shows whether the caller had finished. You need both.
- Change one setting. Lengthen the wait on number steps. Shorten it on yes or no steps. Then run the same call again.
- Test with the voices you will meet. A fast speaker, a slow one, a soft voice, a call on speakerphone, a noisy shop, a number read in groups, "ek second" in the middle of a sentence.
- Give longer waits only where they help. A long wait everywhere makes the whole call feel slow. Our advice is to keep it for steps that collect numbers or addresses. The wait is one slice of the full reply time; see our voice agent latency guide.
- Retest the steps you did not touch. A change in one place can shift another.
For a full method to test an agent before it takes real calls, see our guide on how to test a voice agent. Our guide to how to build an AI calling agent puts testing before real calls as a step in the build.
What does the prompt have to do with it?
Less than people expect. Silence and turn-taking are settings, not lines in the prompt, and a prompt line like "wait for the caller to finish" does not change when the agent replies. We cover that rule, and what the prompt should do about silence and unclear answers, in our voice agent prompt guide.
In our tests across 286 turns in 11 languages, wrong waits fell from 8.6% to 0.7% of finished turns after we gave the turn check a short wait, a long wait for fillers like "hmm... uh...", the length of each number being collected, and examples in every language the agent speaks. That is a small sample, so read it as a sign that these changes help, not as a benchmark.
How Bananaflow fits
Bananaflow is an AI voice agent platform. Barge-in is set for each step of the call, and there is a turn judge and a sound sensitivity setting you can adjust. How long the agent waits after the caller pauses is a setting called reply eagerness, set for each agent, not for each step. Eager waits 0.4 seconds, Normal (the default) waits 0.6 seconds and Patient waits 1.0 second. You set it on the agent's settings page under "Let callers finish", or through the MCP server. An optional AI end-of-turn check is also available, and it is off by default. When a caller cuts in, the agent stops, listens and answers what they said. The inbound calls and outbound calls pages show how this fits each type of call.
You can test any agent in the browser by voice or text before it takes real calls. After each call you get a transcript and a recording, and the QA step can flag problems. See Call QA for what it checks.
What should you remember about endpointing and turn detection?
- Endpointing decides the caller is done. VAD only hears sound. Turn detection decides whose turn it is.
- A pause is not an ending. Callers pause for numbers, for thought and for "ek second".
- People average about 200 milliseconds between turns, but they predict the end of a turn. An agent needs help to do the same.
- That research did not include an Indian language. Treat Indian speech habits as patterns to test.
- Let acknowledgement words like "haan" through, and make real stop words like "ruko" stop the agent.
- Tune one setting at a time, on the step that has the problem, and test with real voices before real calls.
FAQ
Frequently asked questions
What is endpointing?
Endpointing is how a voice agent decides that the caller has finished speaking, so it can start its reply. It is also called endpoint detection or end-of-turn detection. If it decides too early, the agent talks over the caller. If it decides too late, the caller hears dead air.
What is turn detection?
Turn detection is the wider job of working out whose turn it is. It covers endpointing, which asks whether the caller is done, and also the cases where the caller starts speaking while the agent talks. It can use how long the silence lasts, the words so far, or both.
What is VAD in AI?
VAD stands for voice activity detection. It listens to the sound in short slices and decides whether each slice holds human speech or not. It does not know whether a sentence is finished. Phone networks have used it for decades, for example to avoid sending silence. See our glossary entry on voice activity detection for the full definition.
What is semantic turn detection?
Semantic turn detection looks at the words so far and asks whether they make a finished thought. "Thursday works" is finished. "My number is 4410" is not. Research on turn-taking has found that whether a sentence is complete helps predict when a turn will end. It costs a little extra time, and it must understand every language the agent speaks, including Hinglish.
Why does my voice agent interrupt people?
Usually because it treated a pause as the end of the turn. Callers pause to find a card, read a number or think, and a silence-only rule cannot tell that from a finished answer. Noise that sounds like speech can also stop the agent for no reason. Give number steps a longer wait, add a words-so-far check, and test with real voices.
Can a caller interrupt a voice agent?
Yes. When a caller cuts in, the agent should stop, listen and answer what they said. This is called barge-in. On Bananaflow, whether a caller can barge in is set for each step of the call, so you can protect a line that must be heard in full and leave the rest open.
What is a good wait before the agent replies?
There is no single right number. In a study of ten languages, most gaps between turns fell between 0 and 200 milliseconds, and the average was about 208. People manage that by predicting the end of a turn while the other person is still talking. An agent cannot predict as well, so it needs a little longer. On Bananaflow the wait is the agent's reply eagerness setting: Eager waits 0.4 seconds after the caller pauses, Normal (the default) 0.6 seconds and Patient 1.0 second. If callers read out numbers, try Patient.
Do the turn-taking studies cover Indian languages?
Not the best-known one. The ten-language study did not include an Indian language, so do not assume Indian callers follow its average. We found no published measure of turn gaps for Hindi, Tamil or Hinglish phone calls. Treat the habits in this post as patterns to test for, not as statistics.




