Voice activity detection (VAD)
Jaseem Abbas
- Published
- Updated
How does voice activity detection work?
Voice activity detection listens to an audio stream in short slices and marks each one as speech or not speech, so the software around it knows when someone starts and stops talking. It is a different job from speech recognition, which turns the words into text. Wikipedia's entry on voice activity detection defines it as "the detection of the presence or absence of human speech, used in speech processing."
Older detectors judge each slice by features of the sound, such as how loud it is. Newer ones are trained models: Silero VAD, an open-source model, describes itself as a "pre-trained enterprise-grade Voice Activity Detector". Either way, the detector has to decide how much sound counts as speech, and that choice has a cost on both sides: set it too eager and a passing horn sounds like a caller, set it too strict and a soft voice is missed.
On a phone call that matters more than it seems. Picture an eye clinic in Kochi that confirms appointments by phone, about 300 calls a day. One patient calls from a busy road, where traffic noise rises and falls under her words; another is an elderly patient who speaks softly. The same detector has to catch the second voice without treating the first caller's traffic as speech. The clinic and its numbers are made up for this example.
How does AI change voice activity detection?
On a call with an AI voice agent, VAD decides when the caller is speaking, so it sets when the agent can reply and whether a caller can cut in while the agent is talking. When it marks the end of the caller's speech too early, the agent talks over a pause; when it hears noise as speech, the agent stops for nothing.
On Bananaflow, the agent's pipeline runs Silero VAD by default to detect the start and end of the caller's speech. Three settings shape how the caller and the agent take turns: barge-in, set for each step of the conversation, decides whether the caller can cut the agent off at that step; a turn judge decides when the caller has finished speaking; and sound sensitivity, low, medium or high, decides how readily sound on the line counts as the caller talking. Low ignores quiet or short sounds, which suits noisy lines but can miss a very soft voice; high picks up soft voices, but on a noisy line background sound can hold back the agent's reply. For what happens when a caller cuts in, read how a Bananaflow agent handles a caller who cuts in, and for the words that never interrupt and the words that stop the agent in each language, see the agent's languages.
What is barge-in?
Barge-in is when a caller starts speaking while the voice agent or phone menu is still talking, and the system stops to listen. It depends on voice activity detection: the agent can only stop for the caller if something notices, in the middle of its own speech, that the caller has started talking. For more on turn-taking, endpointing and barge-in in voice AI, read what conversational AI is and how it takes turns. For how an agent decides a caller has finished, see endpointing and turn detection.
Why does background noise cut an agent off?
Background noise cuts an agent off when the voice activity detector treats the noise as the caller speaking. Traffic, a television or another person nearby can cross the line it uses for speech, and the agent stops as if the caller had cut in. A less sensitive setting ignores more of that noise, at the cost of missing a very soft voice. For how Silero VAD behaves on narrowband calls, read Silero VAD on 8 kHz phone audio.
FAQ
Frequently asked questions
What is the full form of VAD?
VAD stands for voice activity detection, also called speech activity detection. It is software that detects whether human speech is present in a stream of audio, as against silence and background noise.
What is Silero VAD?
Silero VAD is an open-source, pre-trained voice activity detector, published on GitHub. Bananaflow's agents run it by default to detect when a caller starts and stops speaking.
What is barge-in?
Barge-in is when a caller speaks while the agent is still talking and the agent stops to listen. On Bananaflow, whether a caller can barge in is set for each step of the conversation.
What is endpointing?
Endpointing is deciding that a caller has finished their turn, so the agent can reply. Voice activity detection feeds it by marking where speech stops; on Bananaflow, a turn judge makes the call on when the caller has finished.
Why does background noise cut an agent off?
Because the voice activity detector heard the noise as speech. Traffic, a television or another person nearby can sound like a caller to it. A less sensitive setting ignores more noise, but can miss a very soft voice.
Can I change how sensitive VAD is?
On Bananaflow, yes: sound sensitivity is set per agent to low, medium or high. Low ignores quiet or short sounds and suits noisy lines; high picks up soft voices, though background sound on a noisy line can hold back the agent's reply.
