Skip to content

Bananaflow MCP is live.

Read the docs

How to Compare TTS Voices in Indian Languages

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published
Cover for "How to Compare TTS Voices in Indian Languages": a white line diagram of three round voice nodes in a row, each sending a short sound wave and each over a pill reading A, B or C, with a listener on the left whose bold arc points down at the middle voice, drawn over a soft, out-of-focus crocus in sage mint, its cupped petals blurred into veils, with a sky blue haze at one end and a buttercup yellow glow at the other.

To compare text to speech (TTS) voices in Indian languages, you play the same written lines in each voice, over a phone-quality line, to native speakers who do not know which voice is which, and you count what they say. The same lines go to every voice, and the listeners decide. This guide gives the method: what to write, how to play it, how to run a mean opinion score test or a head-to-head vote, and how to time the first sound. It gives no scores and does not rank any voice.

Here is an example we made up. Arjun runs patient calls for a chain of eye clinics in Bengaluru and needs a Kannada voice for appointment reminders. The demo pages of three voices all sound pleasant. He writes 30 reminder lines, plays each in every voice over a phone call, and asks five Kannada speakers to mark them. One voice reads amounts and dates clearly but sounds flat on questions. Another sounds warm but stumbles on clinic names. Arjun, the clinics and the voices are made up. The point is that a demo page cannot show you what a call line will.

What does it mean to compare TTS voices?

Comparing TTS voices means judging how well each one turns your written lines into speech a caller understands, and how natural it sounds, under the same conditions. Text to speech is the technology that reads text aloud. Our glossary entry on text to speech has the definition.

Quality has more than one part. The words must be right (a number read as the wrong number is a failure). The rhythm and tune, called prosody, must fit the sentence: a question should sound like one. The final sound comes from a vocoder, the part that turns the model's internal picture of the sound into audio you can hear. You do not need to judge each part separately, but it helps to know why two voices that read the same line can sound so different.

Research on Indian languages shows how much choice there is. In one study, Kumar and colleagues evaluated acoustic models and vocoders for Dravidian and Indo-Aryan languages, found that FastPitch with the HiFi-GAN V1 vocoder, trained on male and female speakers together, performed best among the setups they compared, and rated models for 13 languages by mean opinion score (Kumar et al., ICASSP 2023, abstract read 16 November 2026). That is research, and it says nothing about any product. It does show that quality is measured by asking people to listen.

For a buyer overview of what breaks in Indian-language voices, read text to speech for Indian languages on calls. This guide is the hands-on method for choosing between voices.

What lines should you test?

Test the lines your agent will really say, with extra weight on the ones that go wrong. Plan 30 to 40 lines per language. The mix below is our checklist, not a standard.

  • Group: Rupee amounts

    Why it breaks
    Digits can be read one by one, or in the wrong grouping
    A made-up example line
    Your EMI of Rs 12,499 is due on 5 December.
  • Group: Dates and times

    Why it breaks
    Day and month can swap, and 24-hour times read oddly
    A made-up example line
    Your visit is on 18/11 at 4:30 pm.
  • Group: Phone, order and PIN numbers

    Why it breaks
    A digit string needs a steady rhythm
    A made-up example line
    Your order number is 5 6 0 0 3 4.
  • Group: Names and places

    Why it breaks
    Long names can be split in the wrong places
    A made-up example line
    Thiruvananthapuram, Visakhapatnam, Narayanan
  • Group: English inside the local language

    Why it breaks
    The accent or spelling can change
    A made-up example line
    Aapki policy ka renewal premium due hai.
  • Group: Questions

    Why it breaks
    Intonation should rise or fall as in speech
    A made-up example line
    Kya aap kal 11 baje aa sakte hain?
  • Group: Short replies

    Why it breaks
    Very short lines can sound abrupt
    A made-up example line
    Ji, theek hai.
  • Group: Long sentences

    Why it breaks
    Breath and pauses decide naturalness
    A made-up example line
    One sentence with a name, a date and an amount

Write the lines in the script your agent will use, since script changes how a voice reads a line. Our guide on Hinglish voice agents covers writing mixed speech. Lines drafted with a language model, or by us, should be checked by a native speaker before any listener hears them. Our pages for each language, such as Hindi, Kannada and Bengali, show how callers speak in each.

How do you play the voices over a phone line?

Make a phone-quality copy of every sample and play both versions, because a voice that shines in headphones can sound different on a call. The classic phone codec, ITU-T G.711, is titled "Pulse code modulation (PCM) of voice frequencies" (ITU-T G.711, read 16 November 2026), and a call carries far less than a speaker does.

The best test is a real call: have each voice speak your lines to a real mobile phone on a normal network, and have listeners hear it there. If that is too slow, make a phone-style copy with ffmpeg. We ran these two commands on a test tone and the output had the codec and rate shown:

# studio sample -> phone-style 8 kHz mono (G.711 A-law)
ffmpeg -i voice_sample.wav -ac 1 -ar 8000 -c:a pcm_alaw voice_phone.wav

# back to 16 bit PCM so any player can open it
ffmpeg -i voice_phone.wav -ac 1 -ar 8000 -c:a pcm_s16le voice_phone_play.wav

A phone-style copy is a stand-in. Listen to a few real calls to confirm it matches what callers hear. Play both the wide and the phone version to your listeners and note which one each person heard.

One listening test: the same lines go to each voice, through a phone-quality line, to listeners who do not know which voice is which, and the marks are tallied by voice and by line type.

How do you run a listening test?

Use native speakers, hide the voice names, shuffle the order and write down what each person heard. Listeners decide, so choose them carefully, and give them one clear job per round.

  1. Pick listeners who speak the language the way your callers do. For Kannada calls in Bengaluru, use Kannada speakers from that region.
  2. Hide the voice names. Give each sample a code. Do not tell listeners which vendor or voice it is.
  3. Shuffle the order for each person, so one voice is not always heard first.
  4. Use the same lines for every voice. If one voice reads a harder line, the comparison is unfair.
  5. Keep sessions short. Tired listeners rate everything lower. Give a break between rounds.
  6. Record who heard what, and the date, so you can repeat the test later.

Any recording of a person's voice (such as a listener's comments) is personal data too. Tell them what you record, and keep it private. Our DPDP rules for call centers guide covers how Indian rules treat voice data. This is not legal advice.

What is a mean opinion score, and how do you run one?

A mean opinion score (MOS) is the average of the ratings a group of listeners gives to speech on a fixed scale. ITU-T P.800, titled "Methods for subjective determination of transmission quality", is the standard that sets out how such listening tests are done (ITU-T P.800, read 16 November 2026). We could open the recommendation's page but not read its full PDF text today, so check the P.800 document itself for the exact scale wording before you run a formal test.

To run a simple version:

  1. Ask each listener to rate each sample on one fixed scale, for example from 1 to 5, with a short description for each point that you write down and keep fixed.
  2. Ask one question at a time. "How natural does this sound?" is one question. "How clear is the amount?" is another.
  3. Average per voice and per group of lines. Show the spread as well as the mean.
  4. Compare only within your own test. A MOS from another test used other listeners, other lines and often other audio, so the numbers do not line up.
  5. Check the listeners. Repeat a few samples in a different place in the test and see whether the same listener gives a similar mark.

ITU-T P.808, "Subjective evaluation of speech quality with a crowdsourcing approach" (ITU-T P.808, read 16 November 2026), covers running such tests online. For Indian-language calls we suggest using your own native-speaker panel, because you can pick the region and check the lines.

When is a head-to-head vote better than a MOS?

When you only need to know which of two or three voices to use. A head-to-head vote, also called an A/B test, plays the same line in two voices and asks "which do you prefer?". It is easy to run and easy to read.

  • Play the pair in random order. Half of the listeners hear voice A first.
  • Offer "no difference". Forcing a choice when there is none adds noise.
  • Tally wins by line group. "Voice A won 11 of 15 amount lines, voice B won 9 of 15 question lines" is a decision you can act on.
  • Ask why in one line. A short note, such as "reads the amount as digits", points to what to fix.

For a deeper method with hidden reference samples, ITU-R BS.1534 is titled "Method for the subjective assessment of intermediate quality level of audio systems" (ITU-R BS.1534, read 16 November 2026). It is often described as the MUSHRA method, which we only name as background, and we did not verify the details of its procedure for this guide. For choosing a voice for phone calls, a MOS or a head-to-head vote is usually enough.

How do you test whether the caller understood?

Play lines with an amount, a date or a number, and ask the listener to write what they heard. This tests the thing a call needs most: did the right information get across? Naturalness is nice, but a pleasant voice that makes callers mishear a date is a failed voice.

  1. Pick 20 lines with an amount, a date or a number in them.
  2. Play each over the phone-style line, once, with no replays.
  3. Ask the listener to write the amount, date or number they heard.
  4. Mark each as right or wrong against the line's text.
  5. Count by voice. "18 of 20 right" for one voice and "14 of 20" for another is a result you can act on.

Run it on your noisiest cases too: a listener in a room with a fan, or a phone on speaker. A voice that sounds fine in a quiet office may fail there.

How do you measure how fast a voice starts?

Time how long it takes from sending the text to hearing the first sound, and run it several times. On a call, the caller hears silence until the first audio arrives, so a slow start sounds like a dropped line. Our guide on voice agent latency shows how this fits into the full reply time.

The time to first audio: the moment the text is sent, the moment the first piece of sound arrives, and the rest of the line still playing out behind it.

Here is a skeleton of the timing code. It is our own sketch, not a vendor API. synthesize_stream stands for whatever call your voice provider documents for streaming audio, and you must replace it:

import statistics
import time

def first_audio_seconds(text):
    start = time.perf_counter()
    for chunk in synthesize_stream(text):  # replace with your provider's streaming call
        if chunk:
            return time.perf_counter() - start
    return None

runs = [first_audio_seconds("Aapki appointment kal 11 baje hai.") for _ in range(20)]
runs = sorted(r for r in runs if r is not None)
print("median", statistics.median(runs), "slowest", runs[-1])

Run it from the same machine and region for every voice. Report the median and the slowest runs, not one run, and say whether the voice streams or returns the whole line at once. A voice that waits for the whole line is not comparable to one that streams.

How should you score the results?

Keep one sheet per language, with one row per voice and one column per thing you measured. The sheet below is a template for the numbers you collect yourself.

  • Voice code: Voice A

    Clear amounts and dates (x of 20)
    your count
    Names right
    your count
    Question tune: natural or flat
    your note
    Head-to-head wins
    your count
    Median time to first audio
    your seconds
  • Voice code: Voice B

    Clear amounts and dates (x of 20)
    your count
    Names right
    your count
    Question tune: natural or flat
    your note
    Head-to-head wins
    your count
    Median time to first audio
    your seconds
  • Voice code: Voice C

    Clear amounts and dates (x of 20)
    your count
    Names right
    your count
    Question tune: natural or flat
    your note
    Head-to-head wins
    your count
    Median time to first audio
    your seconds

Do not add everything into one grade. A voice that is slightly less warm but never mishears an amount may be the right one for lines that carry amounts, and a warmer voice may suit a greeting. Pick a voice per language and per job, then retest when the voice or your lines change.

How Bananaflow fits

Bananaflow is an AI voice agent platform. These parts help with a voice test:

  • A voice for each agent, which you can change without rebuilding the agent, so you can try voices on the same lines.
  • English plus 10 Indian languages, each heard and spoken in its own script. See Languages.
  • Following the caller's language, and switching mid-call when the caller does.
  • Hinglish written down as spoken.
  • Pre-recorded audio mixed with generated speech, and template variables for names and amounts.
  • Testing before real calls: call any agent by voice in your browser, then place a test call from a real phone.
  • Barge-in set per step, so a caller can cut in.

What should you remember about comparing TTS voices?

  • Use your own lines, with extra numbers, amounts, dates, names, mixed speech and questions.
  • Play over a phone-quality line and, in the end, over a real call.
  • Use native-speaker listeners, hidden voice names and shuffled order.
  • Use a MOS for ratings, a head-to-head vote for a simple choice, and a write-down test for what callers need to hear.
  • Time the first sound, several times, from the same place.
  • Pick a voice per language and per job, and never copy a score from another test.

FAQ

Frequently asked questions

  • How do you compare text to speech voices?

    Write the lines your agent will really say, play the same lines in each voice over a phone-quality line, and have native speakers who do not know which voice is which rate them or vote. Then check amounts, dates and names for clarity, and time how fast each voice starts.

  • What is a mean opinion score for TTS?

    A mean opinion score is the average of the ratings listeners give to speech on a fixed scale. ITU-T P.800 sets out how such listening tests are done. A score from another test is not comparable to yours, because the listeners, lines and audio differ, so run your own and compare within it.

  • Should I use a MOS test or an A/B test?

    Use a head-to-head A/B vote when you only need to pick between two or three voices. It is simpler to run and read. Use a MOS when you want a rating per voice that you can track over time. Either way, hide the voice names and shuffle the order.

  • What lines should I use to test an Indian-language voice?

    Use 30 to 40 lines per language that mirror real calls: rupee amounts, dates and times, phone and order numbers, names and places, English words inside the local language, questions, short replies and one long sentence. A native speaker should check each line before listeners hear it.

  • Why test a voice over a phone line?

    A phone call carries much less sound than headphones. The classic phone codec, ITU-T G.711, samples audio 8,000 times a second, so a voice can sound different on a call than in a demo. Play a phone-style copy or, better, a real call to a mobile phone.

  • How many listeners do I need?

    There is no fixed number we can verify for your case. For a first pass, we suggest five or more native speakers per language, each hearing every line in random order, with a few lines repeated to check they are consistent. Add listeners when two voices score close together.

  • How do I measure how fast a TTS voice starts?

    Record the time from sending the text to receiving the first piece of audio, and repeat it 20 or more times from the same machine and region. Report the median and the slowest runs, and say whether the voice streams or returns the whole line at once.

  • Does a high TTS score predict how it sounds on my calls?

    Not by itself. Published scores use their own listeners, lines and audio, and a score in one language does not carry to another. Treat a score as a starting point, then run a test with your own lines, in each language, over a phone line.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Similar articles

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot