Skip to content

Bananaflow MCP is live.

Read the docs

Speech-to-Speech vs Cascaded Voice Agents

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published
Cover for "Speech-to-Speech vs Cascaded Voice Agents": a white line diagram of one speech model: a sound wave runs into a large glass panel with a spark, which forks into a solid arrow up to a panel with a speaker and a dashed arrow down to a panel with text lines, the transcript on the side, drawn over a soft, out-of-focus bluebell in muted sky blue, its nodding bell petals hanging in soft rows at both ends, with a pale blush glow at the left edge and a soft peach glow at the right.

A speech-to-speech voice agent uses one model that listens to audio and answers in audio. A cascaded voice agent chains three steps: speech to text, a language model, then text to speech. This guide compares the two for phone calls in India, using the official OpenAI and Google documentation for their live voice APIs, and says where each one is stronger and where you should test before you trust it.

Here is a made-up example. Dr Kavitha runs a small chain of dental clinics in Chennai. She wants an agent to call patients in Tamil and Hindi to confirm Thursday appointments. Her developer shows her two demos. The first answers almost instantly and sounds warm, but she cannot see what it understood. The second pauses a beat longer, and every call has a clean transcript she can search. Which is better? The honest answer is: it depends on what she needs to check, change and prove. Dr Kavitha and her clinics are made up for this guide.

What is a cascaded voice agent?

A cascaded voice agent passes each turn through three separate models in a row: speech to text turns the caller's audio into words, a language model writes a reply as text, and text to speech turns that reply into audio. Each step hands plain text to the next, so the words are visible between the steps.

Researchers describe the same set-up as a pipeline of independent components, namely voice activity detection, speech recognition, textual dialogue and text to speech (Moshi paper, arXiv 2410.00037). This is the classic design, and our guide to what conversational AI is shows each part. For the vocabulary, see the glossary entries on speech to text, the language model and text to speech.

Two ways to answer a caller. Above, a cascaded agent: audio goes to speech to text, then a language model, then text to speech. Below, a speech-to-speech agent: one model takes audio in and gives audio out, with a text transcript available on the side.

The strength of this design is the seams. Because words appear between steps, you can read them, log them, check them against rules, and change one part without touching the rest. Our guide to voice agent latency lists the parts and what each one adds to the wait.

What is a speech-to-speech voice agent?

A speech-to-speech voice agent uses a single model that takes audio in and produces audio out, with no separate speech-to-text or text-to-speech step in between. OpenAI describes this as voice-to-voice interaction "without an intermediate text-to-speech or speech-to-text step", which it says enables lower latency for voice interfaces and gives the model more data to work with, such as the tone and inflection of the caller's voice (OpenAI Realtime conversations guide).

The research idea behind it is older than the products. The Moshi paper argues that text pipelines add latency of several seconds, lose non-linguistic information such as emotion or non-speech sounds, and depend on turn segmentation that ignores overlapping speech, interruptions and interjections. Its answer is a model that treats spoken dialogue as speech-to-speech generation and models its own voice and the user's in parallel streams. The authors report a theoretical latency of 160 milliseconds and 200 in practice (Moshi paper). Those are the authors' figures for their own research model, not a promise about any product or any phone line.

For a short definition, see our glossary entry on the speech-to-speech model.

What do OpenAI Realtime and Gemini Live actually document?

Both document a live voice API with interruptions, tool calling and transcripts. We read these pages on 16 November 2026. Product names, models and limits change often, so check the current pages before you build.

  • Topic: What it is

    OpenAI Realtime API
    A stateful session with a session, a conversation and responses. Its guide says it suits agents that need barge-in, low first-audio latency, natural turn taking and realtime tool use
    Google Gemini Live API
    A low-latency, real-time voice and vision API that takes continuous streams of audio, images and text and replies with spoken audio
  • Topic: How you connect

    OpenAI Realtime API
    WebRTC, WebSocket or SIP, with an ephemeral key for browsers. A guide covers SIP trunks for phone calls
    Google Gemini Live API
    WebSockets, from your server or straight from the client. Google recommends ephemeral tokens for production. Partners connect it over WebRTC or WebSockets
  • Topic: Audio formats shown

    OpenAI Realtime API
    PCM audio at 24 kHz in its examples. For SIP, the audio format is negotiated by the call
    Google Gemini Live API
    Raw 16-bit PCM, 16 kHz in and 24 kHz out. Other input rates are resampled
  • Topic: Interruptions

    OpenAI Realtime API
    Its guide lists barge-in among the uses the API suits
    Google Gemini Live API
    Documented as barge-in: users can interrupt the model at any time
  • Topic: Tool use

    OpenAI Realtime API
    Functions can be set on the session or on a single response
    Google Gemini Live API
    Tool use, including function calling and Google Search
  • Topic: Transcripts

    OpenAI Realtime API
    Audio responses come with a transcript. The guide notes the model cannot precisely align a transcript with audio when a reply is cut short
    Google Gemini Live API
    Transcripts of input and output audio can be turned on in the setup
  • Topic: Languages

    OpenAI Realtime API
    Check the current language list for your languages
    Google Gemini Live API
    A supported-languages page lists Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil and Telugu, among many others

Read the table with care. OpenAI's and Google's pages are written for developers, they change, and a row in a table is a feature, not a result. The row that matters most for you is the last one. A language on a list means the model can take part in that language. It does not tell you how it hears a caller on a bad mobile line, or how it handles Hinglish. Only your own recordings tell you that. Our guide to Indian LLMs for voice agents and the Hinglish guide cover that test.

Which one is faster?

Speech-to-speech removes two hand-offs, so it has the chance to be faster, but the chance is not the same as the result on your phone line. A cascaded turn waits for speech to text to finish, then for the model to start, then for the voice to produce its first audio. A speech-to-speech model has no text hand-off between listening and speaking, which is where OpenAI says its lower latency comes from.

Three things still add time to both designs:

  • The phone line counts twice, in and out, before any AI runs.
  • The wait to be sure the caller has finished. Both designs must decide when the caller is done. OpenAI documents turn detection settings in a session, one of them called semantic_vad. Our guide to endpointing and turn detection explains why this wait is a big slice.
  • Tool calls. If the agent must look up an order, the lookup takes the same time in either design.

So do not pick on a headline number. Time both from the caller's ear: from the last word the caller says to the first sound the agent makes, on your own phone calls. We give no figure here because we have none we can verify for Indian phone calls.

Which one gives you more control?

Cascaded gives you more places to take control, and speech-to-speech gives you fewer things to run. These are two different kinds of control, so know which one you need.

  • You want to...: Change only the voice

    Cascaded
    Swap the text-to-speech step and keep the rest
    Speech-to-speech
    Pick from the voices the model offers, and the voice is fixed once the model has spoken in a session
  • You want to...: Change what hears Hindi

    Cascaded
    Swap the speech-to-text step, or run two and compare
    Speech-to-speech
    Stay with the model's own listening, or change the whole model
  • You want to...: Check the words before the agent speaks

    Cascaded
    Read, filter or rewrite the text between steps
    Speech-to-speech
    The model speaks as it generates, so any check has to run alongside it
  • You want to...: Use a different language model

    Cascaded
    Swap the model and keep the rest
    Speech-to-speech
    You are tied to a model that can speak
  • You want to...: Keep a record of what was said

    Cascaded
    Text exists at every step
    Speech-to-speech
    Ask for a transcript from the API, and check it against the audio
  • You want to...: Test one part alone

    Cascaded
    Test speech to text on audio, the model on text, the voice on text
    Speech-to-speech
    Test the whole model with audio in and audio out

The OpenAI row about the voice comes from its guide, which says the voice cannot be changed after the model has responded with audio once in a session (OpenAI Realtime conversations guide). The rest of the table is our reading of how the two designs work, so test it against the model you plan to use.

Four checks before you choose: have you tested your Indian languages on phone audio, do you need to read or change the words before the agent speaks, do you want to swap one part at a time, and have you measured cost per minute on a real call. Each answer points toward one design or the other.

What about transcripts, QA and call records?

Both designs can give you a transcript, but the transcript means different things. In a cascaded agent, the transcript is the text the model actually read, so what you see is what the agent acted on. In a speech-to-speech agent, the transcript is an extra output produced alongside the audio. Both APIs offer it: Google lets you switch on transcription of input and output audio (Live API capabilities), and OpenAI audio responses come with a transcript.

For quality checks this matters. If a QA score or a rule check runs on the transcript, you want to know the transcript is faithful. OpenAI's own guide says the model does not have enough information to align a transcript with audio precisely when you cut a reply short, so a transcript may not match exactly what the caller heard. Plan to spot-check transcripts against recordings, whichever design you pick. Our guide to testing a voice agent describes how.

What do phone calls add?

Phone audio is narrow, and the models were documented with wider audio formats, so something has to convert it. A phone call over a carrier usually carries 8 kHz audio. Twilio's Media Streams documentation shows the format as audio/x-mulaw at a rate of 8000 (Twilio Media Streams messages). Google's Live API takes 16 kHz audio in and gives 24 kHz out (Gemini Live API capabilities), and OpenAI's examples use 24 kHz (OpenAI Realtime conversations guide). Converting between rates is routine, but raising 8 kHz audio to 16 kHz does not add detail that the phone line threw away. That is our reasoning, and the reason to test on real calls and not on studio recordings or browser microphones.

OpenAI also documents a SIP connection, where your carrier's SIP trunk exchanges call audio with OpenAI, and lists an endpoint for SIP hosts located in India (OpenAI SIP guide). That is a different path from sending audio through your own server. If you are comparing paths, our guide to WebRTC vs WebSocket for voice agents lays out how phone audio travels.

What does each one cost?

Cascaded has three line items, and speech-to-speech has one, billed in a different unit. A cascaded call pays a speech-to-text service for audio, a language model for text and a voice service for the audio it creates. A speech-to-speech call pays one provider, in that provider's own units. We do not quote prices here, because they change. Read each provider's current pricing page.

The useful habit is to measure. Run the same sample of real calls through each design, add up what the provider billed for that sample, and divide by the minutes. Then add the phone carrier's charge, which is separate in either design. Our guide to the cost of a telecaller versus an AI calling agent shows where the voice stack sits in the bigger cost.

How do you choose between them?

Choose with a test, not a theory. Take about fifty real calls (or recordings, with the right permissions) from the calls you plan to make. Run them through each design. Score them on the things your business cares about.

  1. Understanding. Did the agent hear the number, the name, the date and the language correctly? Check Hindi, Hinglish and your other languages separately.
  2. Timing. From the caller's last word to the agent's first sound, on a phone line.
  3. Turn taking. Did it talk over callers, or wait too long?
  4. Tool calls. Did the lookup, booking or transfer work, with the right arguments?
  5. Records. Is the transcript faithful, and can QA run on it?
  6. Cost. What did a minute really cost, all in?
  7. Control. When something went wrong, could you see why and fix one part?

Use the table below as a first guess, then let the test overrule it.

  • Your situation: You must read and log every word the agent acts on

    Lean toward
    Cascaded
    Why
    Text exists between every step
  • Your situation: You want to swap the speech, model or voice on their own

    Lean toward
    Cascaded
    Why
    Each part is separate
  • Your situation: You want the most natural interruptions and the least to maintain

    Lean toward
    Speech-to-speech
    Why
    One model handles listening and speaking
  • Your situation: Your calls are in languages you have not tested

    Lean toward
    Test both
    Why
    A language list is not a result on phone audio
  • Your situation: You need records for QA, audits or disputes

    Lean toward
    Test both, with spot checks
    Why
    Check transcripts against recordings in either design

How Bananaflow fits

Bananaflow is an AI voice agent platform for calls in India. Each voice agent picks its own speech-to-text, language model and voice, and can change any one of them without rebuilding the agent. The agent supports English (Indian) and 10 Indian languages (Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil and Telugu), follows the caller's language and can switch mid-call. You can test any agent by voice in your browser before it takes real calls. After each call you get a transcript, a recording, extracted fields and a QA score, with tags, sentiment and a summary. See call QA and our languages page.

What should you remember about speech-to-speech and cascaded agents?

  • Cascaded is speech to text, a model, then text to speech. Speech-to-speech is one model, audio in and audio out.
  • Speech-to-speech can remove hand-offs and keep the caller's tone. Cascaded gives you text between every step and parts you can swap.
  • OpenAI and Google both document interruptions, tool calling and transcripts on their live voice APIs. Read the current pages, because they change.
  • A language on a list is not a test result. Check Hindi, Hinglish and your other languages on real phone audio.
  • Phone audio is 8 kHz. Raising it to a model's rate does not bring back detail the line lost.
  • Check transcripts against recordings, and measure cost per minute and delay from the caller's ear on your own calls.

FAQ

Frequently asked questions

  • What is a speech-to-speech model?

    A speech-to-speech model takes spoken audio in and produces spoken audio out with one model, instead of chaining speech to text, a language model and text to speech. OpenAI describes this as voice-to-voice interaction without an intermediate text-to-speech or speech-to-text step. Our glossary has a short definition of the speech-to-speech model.

  • What is a cascaded voice agent?

    A cascaded voice agent passes each turn through three models in a row. Speech to text turns the caller's audio into words, a language model writes the reply as text, and text to speech turns the reply into audio. Because the words appear between the steps, you can read, log and check them, and swap one part without touching the others.

  • Is speech-to-speech faster than a cascaded agent?

    It can be, because there is no text hand-off between listening and speaking, and OpenAI says this enables lower latency. But the phone line, the wait to be sure the caller has finished and any tool call add time to both designs. Measure from the caller's last word to the agent's first sound on your own phone calls, not from a headline number.

  • Do OpenAI Realtime and Gemini Live support Indian languages?

    Google's Live API documentation lists Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil and Telugu among its supported languages. Check OpenAI's current pages for the languages you need. A language on a list is not proof that it works on a noisy 8 kHz phone call, so test with recordings of your own callers.

  • Can a speech-to-speech agent call tools and APIs?

    Yes. OpenAI's Realtime guide shows functions configured on the session or on a single response, and Google's Live API documents tool use including function calling. The lookup itself takes the same time in either design, so a slow order-status API slows both a cascaded and a speech-to-speech agent equally.

  • Which should I choose for phone calls in India?

    Choose with a test. Run about fifty real calls through each design and score understanding in your languages, delay from the caller's ear, turn taking, tool calls, transcript quality and cost per minute. Cascaded tends to suit teams that must read the words and swap parts. Speech-to-speech tends to suit teams that want natural interruptions with less to maintain.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Similar articles

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot