Skip to content

Bananaflow MCP is live.

Read the docs

Speech-to-speech model: how it differs from a cascaded voice agent

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published

What is a speech-to-speech model?

A speech-to-speech model is a single AI model that listens to audio and answers in audio, so the words never pass through a written-text step on the way. The authors of Moshi, a research model published in 2024, describe it as "casting spoken dialogue as speech-to-speech generation". OpenAI's voice agents guide (read 16 November 2026) describes its Realtime API in the same spirit: one model that interprets audio, decides what to do and responds in speech.

Picture Anil, who rings an electricity helpline in Jaipur, annoyed about a wrong bill. He sighs, then says "theek hai, bata dijiye" in a flat voice. A pipeline that turns his speech into text keeps the words and drops the sigh and the tone. A speech-to-speech model hears the audio itself, so in principle it can use that tone. Anil and the helpline are made up for this example.

How is a speech-to-speech model different from a cascaded voice agent?

A cascaded voice agent runs three separate models in a row, while a speech-to-speech model does the whole job in one. In the cascaded design, speech-to-text writes down what the caller said, a language model decides the reply in text, and text-to-speech turns that reply into a voice.

A cascaded agent passes text between three models; a speech-to-speech model goes from audio to audio in one.

The Moshi paper names three problems it sets out to fix in the pipeline design: the chain of parts adds delay between turns, text as the middle step loses things like emotion and non-speech sounds, and splitting the call into turns ignores overlapping speech and interruptions. OpenAI's guide lists the trade from the other side: a chained pipeline gives "control over each speech and text stage", so you can inspect or change the text in the middle and swap any one part independently.

  • Question: What passes between steps

    Cascaded (speech-to-text, language model, text-to-speech)
    Written text
    Speech-to-speech
    Audio inside one model
  • Question: Can you read the text in the middle

    Cascaded (speech-to-text, language model, text-to-speech)
    Yes, and change it
    Speech-to-speech
    There is no text step between
  • Question: Can you swap one part

    Cascaded (speech-to-text, language model, text-to-speech)
    Yes, each part on its own
    Speech-to-speech
    You change the whole model
  • Question: What the model can hear

    Cascaded (speech-to-text, language model, text-to-speech)
    Words only
    Speech-to-speech
    The audio itself, including tone

How does AI change the choice for phone calls?

It turns the choice into a question about your own calls: how fast a turn must feel, which languages and accents you need, how much control you want over each part, and whether you need a written record of every call. Neither design wins on every call, and the only fair test is to run both on recordings of your own callers, on a phone line and not a studio microphone. Speech-to-speech vs cascaded voice agents walks through that comparison, and voice agent latency explains where the delay in a call comes from.

Each Bananaflow agent picks its own speech-to-text, language model and voice, and can change any one of them without rebuilding the agent. See how a Bananaflow agent is put together.

What are examples of speech-to-speech models?

Two named examples are Moshi and OpenAI's Realtime API. Moshi is a research model whose paper calls it the first real-time full-duplex spoken large language model, meaning it can listen and speak at the same time. The Realtime API is a hosted service for building voice agents with one model. Other providers offer similar services; check each provider's own documentation, because what they support changes often.

FAQ

Frequently asked questions

  • What is a speech-to-speech model?

    A speech-to-speech model is an AI model that takes spoken audio in and gives spoken audio out, with no separate text step between them. One model hears the caller and speaks the reply.

  • What is a cascaded voice agent?

    A cascaded voice agent chains three parts: speech-to-text writes down what the caller said, a language model writes the reply in text, and text-to-speech turns it into a voice. Text is what passes between the parts.

  • Is speech-to-speech faster than a cascaded pipeline?

    It aims to be. The Moshi paper says chained parts add delay between turns, and one model removes the hand-offs. The real figure depends on the model, the network and the phone line, so test it on your own calls.

  • What does text lose that speech-to-speech can keep?

    The Moshi paper says text as the middle step loses non-linguistic information that changes meaning, such as emotion or non-speech sounds. A model that works on audio directly can, in principle, still use it.

  • What do you give up with speech-to-speech?

    Some control. OpenAI's guide says a chained pipeline lets you inspect or change the text between stages and replace each component independently. With one model, there is no text in the middle to check.

  • What are examples of speech-to-speech models?

    Moshi, a research model from 2024, and OpenAI's Realtime API are two. The Moshi paper calls its model the first real-time full-duplex spoken large language model, one that can listen and speak at the same time.

  • What does full-duplex mean in a voice model?

    Full-duplex means the model listens and speaks at the same time, like people do, so it can handle overlapping speech and interruptions. The Moshi paper models its own speech and the caller's as two parallel streams.

  • How does a Bananaflow agent choose its speech parts?

    Each Bananaflow agent picks its own speech-to-text, language model and voice, and can change any one of them without rebuilding the agent. You can test the result in the browser before the agent takes real calls.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot