Skip to content

Bananaflow MCP is live.

Read the docs

Endpointing and Turn Detection in Voice AI

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published
Updated
Cover for "Endpointing and Turn Detection in Voice AI": a white line diagram of the turn check: a sound wave, the caller speaking, runs into a lit node, and from the node a bold arrow leads to a panel with a speaker labelled Reply, while a dashed arc loops back from the node over the wave under a pill reading Keep listening, drawn over a soft, out-of-focus foxglove in muted aqua, its long bell-shaped petals stretched sideways, with a cool sky blue haze at the far left and a narrow buttercup glow at the far right.

Endpointing is how a voice agent decides that the caller has finished speaking, so it can reply. Get it wrong in one direction and the agent talks over people. Get it wrong in the other and callers sit in silence and say "hello?". This guide explains endpointing, turn detection, voice activity detection and barge-in in plain words, shows why Indian callers trip them up, and gives advice on tuning and testing.

Here are two calls from the same agent. A clinic in Lucknow calls Mr Verma to confirm a cashless visit. He says, "Haan... ek second, card dekh raha hoon," and starts reading his policy number: "4410... 27..." At the pause the agent says, "Sorry, I didn't catch that. Could you repeat your policy number?" He sighs and starts again. On another call, a patient says "Yes, Thursday" and then hears two seconds of nothing before the agent answers. She says, "Hello?"

The cause is the same on both calls: one setting decides when a turn ends. It was too eager for Mr Verma and too patient for the second caller. The clinic, the patients and the numbers are an example we made up. They are not customer data.

What is endpointing?

Endpointing is the decision that the caller's turn is over. It is also called endpoint detection or end-of-turn detection. The agent runs it continuously while the caller speaks, and the moment it says "done", the agent starts its reply.

It can fail in two ways:

  • Too early: the agent starts talking while the caller is still thinking or reading out a number, so it cuts in.
  • Too late: the agent waits well after the caller finished, so the call has dead air and the caller wonders whether anyone is there.

Both are about timing, not about what the agent knows. For the short version of this idea inside a wider overview, see the turn-taking section of our guide to what conversational AI is. This post is the deeper version.

What is the difference between VAD, endpointing and turn detection?

They are three layers of one job. VAD asks "is there speech right now?". Endpointing asks "has this speaker finished?". Turn detection asks "whose turn is it, and should the agent talk or stay quiet?".

Voice activity detection checks short slices of audio and labels each one as speech or not speech. Wikipedia describes it as detecting the presence or absence of human speech, and notes that real algorithms trade latency, sensitivity, accuracy and computing cost (Wikipedia on voice activity detection). It is old technology on phone lines. ITU-T G.729 Annex B, from 1996, is a silence compression scheme for a phone voice codec, built on the same idea (ITU-T G.729 Annex B). Our voice activity detection glossary entry has the full definition, so we do not repeat it here.

Endpointing uses VAD's output, and sometimes more, to decide the turn is over. The simplest endpointer says: if there has been no speech for a set time, the turn has ended.

Turn detection is the widest layer. It includes endpointing and also covers what happens when the caller starts speaking while the agent talks, and small sounds like "hmm" that are not a real turn.

  • Layer: Voice activity detection

    The question it answers
    Is there speech in this slice of sound?
    What it uses
    The sound only
    How it fails
    Counts traffic or a TV as speech, or misses a soft voice
    What helps
    Adjust how much sound counts as speech
  • Layer: Endpointing

    The question it answers
    Has this speaker finished?
    What it uses
    VAD plus a silence wait, sometimes the words
    How it fails
    Cuts in at a pause, or waits too long after the end
    What helps
    A longer wait where callers pause, a words-so-far check
  • Layer: Turn detection

    The question it answers
    Whose turn is it, and should the agent stop or go on?
    What it uses
    All of the above plus the caller starting to speak
    How it fails
    Stops for a "haan", or keeps talking over a "ruko"
    What helps
    Decide per step who may cut in, and which words count

A turn decision can combine three signals, and the better systems use more than one.

The signals a turn decision can combine: how long the silence lasts (voice activity detection), the words so far (is the sentence complete), and the caller starting to speak again (barge-in).

Why do voice agents interrupt callers or wait too long?

Because a pause is not an ending, and a silence timer cannot tell them apart. Callers stop speaking for many reasons: to think, to find a card, to read a number, to check with someone in the room. The audio looks the same as a finished answer.

One caller turn on a timeline: speech, a pause while finding a number, more speech, then the real end. An agent that ends the turn at the pause interrupts the caller, and one that waits long after the real end leaves dead air.

The table maps the symptoms a business owner usually hears to the likely cause. This is advice from how these systems work, not a diagnosis of your setup.

  • What you hear: The agent talks over a pause in the middle of an answer

    Likely layer
    Endpointing: the wait is too short
    What to change
    Lengthen the wait on that step, and add a words-so-far check
  • What you hear: Dead air after a short answer like "Yes, Thursday"

    Likely layer
    Endpointing: the wait is too long, or there is no check that the sentence is complete
    What to change
    Shorten the wait on steps that expect short answers
  • What you hear: The agent stops for "haan" or "hmm"

    Likely layer
    Turn detection: backchannels count as interruptions
    What to change
    Make sure acknowledgement words never stop the agent
  • What you hear: The agent stops when nobody spoke

    Likely layer
    Voice activity detection: noise counts as speech
    What to change
    Lower the sound sensitivity, then retest on a noisy line
  • What you hear: The agent keeps talking when the caller says "ruko"

    Likely layer
    Turn detection: real stop words are not recognised
    What to change
    Add the stop words, in the languages your callers use
  • What you hear: The caller says "hello?" after their answer

    Likely layer
    Endpointing or VAD: the end was never detected
    What to change
    Check noise first, then the wait

How fast do people take turns?

Faster than most people think. A study of ten languages, by Stivers and colleagues, found that most transitions between turns fell between 0 and 200 milliseconds, with a mean response offset of about 208 milliseconds across the whole data set (Stivers et al., PNAS 2009). Japanese was the fastest at +7 milliseconds and Danish the slowest at +469, and each language's mean sat within about 250 milliseconds of the overall mean.

There is a catch for an agent. Levinson and Torreira point out that gaps are short, about 200 milliseconds, but producing speech takes far longer, over 600 milliseconds. So listeners must predict the end of the other person's turn and plan a reply while still listening (Levinson and Torreira, 2015). People do not wait for silence. They see the end coming.

Two lessons follow. First, a silence timer is a weak copy of what people do, so it needs more help than a timer alone. Second, be honest about what the number means. None of the ten languages in the Stivers study was an Indian language, so do not assume Indian callers follow its average. We found no published measure of turn gaps for Hindi, Tamil or Hinglish phone calls.

How do Indian callers' speech habits trip up turn detection?

Several everyday habits look like an ending to a silence timer. These are patterns we see in calls and in our tests, not statistics, and the fix depends on your callers, so test for them.

  1. Asking for time: "ek second", "one minute", "hold on". The caller is not ending a turn. They are asking you to wait. A good agent agrees in a word ("Sure, take your time"), then listens, and answers whatever comes next, such as "ek second... fees kitni hai?".
  2. Fillers that hold the floor: "haan... matlab...", "woh... kya hai na...", "actually...". The sentence is not finished, but there is a gap. A silence-only endpointer jumps in. A words-so-far check sees an unfinished thought and keeps listening.
  3. Numbers read in groups. A policy, loan or order number is often read off a card or an SMS in pieces: "560... 0... 34...". Each pause between pieces looks like an ending. Steps that collect numbers need a longer wait than steps that collect a yes or no.
  4. "Haan", "hmm", "accha", "ji" while the agent talks. These are backchannels, the small sounds that show someone is listening. They are not interruptions. "Ruko", "wait" and "nahi nahi" are real stops. Telling these apart is a word-level job, so it is not something VAD can do.
  5. Noisy lines. Traffic, a TV, a shop counter or a speakerphone in a car. Noise can look like speech, so the agent stops for nothing, or it never hears silence and answers late. We cover the noise case in our glossary entry on voice activity detection.
  6. Mixing languages in one sentence. A caller may say "mera policy number hai four four one zero" in one breath. A check that decides whether a sentence is finished has to understand Hinglish, not only English. In our tests, a turn check given English-only examples still misjudged Hindi turns.

Our guide to Hinglish voice agents covers the language side, including how the agent follows a caller who switches between Hindi and English.

What is semantic turn detection?

Semantic turn detection decides whether a turn is over by looking at the words so far, not only at the silence. "Thursday works for me" is a finished thought. "My number is 4410" is not, whatever the silence after it. It is sometimes called end-of-utterance or end-of-speech detection.

The idea has research behind it. Ekstedt and Skantze's TurnGPT paper showed that whether a sentence is syntactically and pragmatically complete helps predict where a turn shifts, and that a language model of the dialogue so far beat two baselines (TurnGPT, 2020). A later paper from the same authors modelled turn-taking by predicting both speakers' voice activity a little ahead of time, trained without labels, and including backchannels (Voice Activity Projection, 2022). Both are research, not a product promise.

Semantic turn detection has a cost, and you should weigh it:

  • It adds a little time, because something has to read the words.
  • It must understand every language the agent speaks. A check that is good in English and weak in Hindi will keep listening when it should reply, or reply when it should keep listening.
  • It still needs a fallback. If a caller says something odd, the agent cannot wait forever, so a time limit still applies.

What is barge-in, and when should a caller be able to cut in?

Barge-in is when the caller speaks while the agent is talking, and the agent stops to listen. In voice AI, that is the meaning, and it has nothing to do with the dictionary phrase. When it works, the agent stops, listens and answers what the caller just said. For a short definition, see our glossary answer on what barge-in is. For tuning it in practice, read barge-in on voice agents.

The hard part is deciding what counts. A real barge-in is the caller taking the floor. A backchannel is the caller saying "haan" to show they are following. If every "haan" stops the agent, the agent sounds jumpy and loses its place. If nothing stops it, the agent talks over a caller who is trying to correct it.

Bananaflow keeps per-language lists of words that never interrupt, like "haan" and "mhm", and words that stop the agent, like "ruko". There are starter lists for all 11 languages, and you can edit them. We explain the lists in our haan and ruko section.

When should you stop a caller from cutting in?

Allow barge-in on most steps, and make it stricter on a few. This is our advice:

  • A required disclosure. A line that must be heard in full, such as a recording notice or a payment term, is a good place to restrict cutting in.
  • The opening line. A short greeting that gets talked over on the first second can leave both sides confused.
  • Steps where the caller asks questions. Leave these open, because the caller will want to interrupt when they already know the answer.

On Bananaflow, barge-in is set for each step of the call, so you can make these choices step by step. That is a product fact. The rules above are our advice on how to use it.

How do you tune endpointing without making it worse?

Change one setting at a time, change it on the step that has the problem, and retest with real voices. Most failed tuning comes from changing everything at once, so no one can tell what helped.

There are four settings to think about, and each has a cost on both sides.

  • Setting: Wait before replying

    Too eager or too sensitive
    Cuts in at pauses
    Too patient or too strict
    Dead air after short answers
    Who suffers
    Callers who pause, or callers who answer fast
  • Setting: How much sound counts as speech

    Too eager or too sensitive
    Stops for noise
    Too patient or too strict
    Misses a soft voice
    Who suffers
    Callers on noisy lines, or callers who speak quietly
  • Setting: How easily the caller can cut in

    Too eager or too sensitive
    Stops for every "haan"
    Too patient or too strict
    Talks over "ruko"
    Who suffers
    Callers who are only listening, or callers who are correcting
  • Setting: A words-so-far check

    Too eager or too sensitive
    Answers an unfinished sentence
    Too patient or too strict
    Waits when the sentence was already done
    Who suffers
    Callers who read numbers, or callers with short answers

What steps should you follow to tune it?

  1. Find the step with the problem. A number step, a yes or no step and a greeting each need different waits. Do not apply one setting to the whole call.
  2. Listen to recordings and read transcripts. The transcript shows where the agent replied. The recording shows whether the caller had finished. You need both.
  3. Change one setting. Lengthen the wait on number steps. Shorten it on yes or no steps. Then run the same call again.
  4. Test with the voices you will meet. A fast speaker, a slow one, a soft voice, a call on speakerphone, a noisy shop, a number read in groups, "ek second" in the middle of a sentence.
  5. Give longer waits only where they help. A long wait everywhere makes the whole call feel slow. Our advice is to keep it for steps that collect numbers or addresses. The wait is one slice of the full reply time; see our voice agent latency guide.
  6. Retest the steps you did not touch. A change in one place can shift another.

For a full method to test an agent before it takes real calls, see our guide on how to test a voice agent. Our guide to how to build an AI calling agent puts testing before real calls as a step in the build.

What does the prompt have to do with it?

Less than people expect. Silence and turn-taking are settings, not lines in the prompt, and a prompt line like "wait for the caller to finish" does not change when the agent replies. We cover that rule, and what the prompt should do about silence and unclear answers, in our voice agent prompt guide.

In our tests across 286 turns in 11 languages, wrong waits fell from 8.6% to 0.7% of finished turns after we gave the turn check a short wait, a long wait for fillers like "hmm... uh...", the length of each number being collected, and examples in every language the agent speaks. That is a small sample, so read it as a sign that these changes help, not as a benchmark.

How Bananaflow fits

Bananaflow is an AI voice agent platform. Barge-in is set for each step of the call, and there is a turn judge and a sound sensitivity setting you can adjust. How long the agent waits after the caller pauses is a setting called reply eagerness, set for each agent, not for each step. Eager waits 0.4 seconds, Normal (the default) waits 0.6 seconds and Patient waits 1.0 second. You set it on the agent's settings page under "Let callers finish", or through the MCP server. An optional AI end-of-turn check is also available, and it is off by default. When a caller cuts in, the agent stops, listens and answers what they said. The inbound calls and outbound calls pages show how this fits each type of call.

You can test any agent in the browser by voice or text before it takes real calls. After each call you get a transcript and a recording, and the QA step can flag problems. See Call QA for what it checks.

What should you remember about endpointing and turn detection?

  • Endpointing decides the caller is done. VAD only hears sound. Turn detection decides whose turn it is.
  • A pause is not an ending. Callers pause for numbers, for thought and for "ek second".
  • People average about 200 milliseconds between turns, but they predict the end of a turn. An agent needs help to do the same.
  • That research did not include an Indian language. Treat Indian speech habits as patterns to test.
  • Let acknowledgement words like "haan" through, and make real stop words like "ruko" stop the agent.
  • Tune one setting at a time, on the step that has the problem, and test with real voices before real calls.

FAQ

Frequently asked questions

  • What is endpointing?

    Endpointing is how a voice agent decides that the caller has finished speaking, so it can start its reply. It is also called endpoint detection or end-of-turn detection. If it decides too early, the agent talks over the caller. If it decides too late, the caller hears dead air.

  • What is turn detection?

    Turn detection is the wider job of working out whose turn it is. It covers endpointing, which asks whether the caller is done, and also the cases where the caller starts speaking while the agent talks. It can use how long the silence lasts, the words so far, or both.

  • What is VAD in AI?

    VAD stands for voice activity detection. It listens to the sound in short slices and decides whether each slice holds human speech or not. It does not know whether a sentence is finished. Phone networks have used it for decades, for example to avoid sending silence. See our glossary entry on voice activity detection for the full definition.

  • What is semantic turn detection?

    Semantic turn detection looks at the words so far and asks whether they make a finished thought. "Thursday works" is finished. "My number is 4410" is not. Research on turn-taking has found that whether a sentence is complete helps predict when a turn will end. It costs a little extra time, and it must understand every language the agent speaks, including Hinglish.

  • Why does my voice agent interrupt people?

    Usually because it treated a pause as the end of the turn. Callers pause to find a card, read a number or think, and a silence-only rule cannot tell that from a finished answer. Noise that sounds like speech can also stop the agent for no reason. Give number steps a longer wait, add a words-so-far check, and test with real voices.

  • Can a caller interrupt a voice agent?

    Yes. When a caller cuts in, the agent should stop, listen and answer what they said. This is called barge-in. On Bananaflow, whether a caller can barge in is set for each step of the call, so you can protect a line that must be heard in full and leave the rest open.

  • What is a good wait before the agent replies?

    There is no single right number. In a study of ten languages, most gaps between turns fell between 0 and 200 milliseconds, and the average was about 208. People manage that by predicting the end of a turn while the other person is still talking. An agent cannot predict as well, so it needs a little longer. On Bananaflow the wait is the agent's reply eagerness setting: Eager waits 0.4 seconds after the caller pauses, Normal (the default) 0.6 seconds and Patient 1.0 second. If callers read out numbers, try Patient.

  • Do the turn-taking studies cover Indian languages?

    Not the best-known one. The ten-language study did not include an Indian language, so do not assume Indian callers follow its average. We found no published measure of turn gaps for Hindi, Tamil or Hinglish phone calls. Treat the habits in this post as patterns to test for, not as statistics.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Similar articles

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot