Skip to content

Bananaflow MCP is live.

Read the docs

Benchmarking Speech-to-Text on Indian Calls

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published
Cover for "Benchmarking Speech-to-Text on Indian Calls": a white line diagram of a balance on a lit pivot under a pill reading Same rules: a level beam with a shallow pan hanging from each end, a sound wave in the left pan and lines of text in the right, drawn over a soft, out-of-focus water lily in sage mint, its pointed petals blurred into veils, with an aqua haze at one end and a peach glow at the other.

Benchmarking speech to text on Indian calls means running two or more speech recognition systems on the same set of your own phone recordings and comparing their errors under the same rules. The accuracy figures on vendor pages cannot do this for you, because they were measured on other audio. This guide gives the method: how to build a test set, write the correct text, clean it, score it, time it and keep the comparison fair. It gives no scores, because the only score that matters is the one from your calls.

Here is an example we made up. Kavita runs the renewals desk of a health insurance broker in Hyderabad. Two vendors both say they are "95% accurate" in Hindi and Telugu. She pulls 40 recent calls, has them typed out by hand and runs both systems. One vendor does well on the quiet calls and badly on policy numbers. The other is slower, but gets the numbers right. Kavita, the broker and the vendors are made up. The lesson is not: a single headline percentage would have hidden the difference that mattered to her.

What is a speech-to-text benchmark, and why run your own?

A speech-to-text benchmark is a repeatable test: fixed audio, fixed correct text, fixed scoring rules, and a number for each system. You run your own because accuracy depends on the audio, the language mix and the topic, and your calls are not anyone else's.

Researchers say the same. The Vistaar paper argues that diverse benchmarks are needed for Indian languages. It collects 59 benchmarks across language and domain combinations, and evaluates three publicly available and two commercial systems on them (Bhogale et al., Vistaar, read 16 November 2026). The IndicVoices dataset gives a sense of the spread: 7,348 hours from 16,237 speakers in 145 Indian districts and 22 languages, made of read speech (9%), extempore speech (74%) and conversational speech (17%) (IndicVoices, read 16 November 2026). A call centre test set will be much smaller, but it should have the same spirit: many kinds of people and situations.

Our other guides cover what a buyer should look for. Call transcription for Indian languages is the buyer overview, and Hindi speech to text on calls explains why phone audio is hard. For one open model in detail, see how to test Whisper on Hindi phone calls. This post is the general method for any model and any language.

How do you get the right to use the recordings?

Get clear permission before you use any recording in a test, and keep the files private. A caller's voice, name and account details are personal data. A test is a new use of the recording, so do not assume the "this call is recorded" line covers it.

  • Ask your own adviser which consent and notice rules apply to your calls. This is not legal advice.
  • Read the rules. Our guides on the DPDP rules for call centers and on whether call recording is legal in India explain what applies.
  • Share carefully. If a vendor needs audio for a trial, give them only what you have the right to give, under a written agreement, and ask what they keep.
  • Limit access. Store the set in one folder with limited access and delete it when the test ends, unless you need it for the next round.

How do you build a test set for Indian calls?

Pick calls that look like your real traffic, then add the hard cases on purpose. A good set has enough calls in each group you care about, and it is chosen before you run any system. The table is our checklist. The sizes are our rules of thumb, not standards.

  • Cover this: Each language you serve

    Why
    Results do not carry from Hindi to Tamil
    How many, as a start
    30 to 50 calls per language
  • Cover this: Speakers

    Why
    Voices, ages and accents differ
    How many, as a start
    No one speaker in more than 3 calls
  • Cover this: Regions

    Why
    A Hindi call in Lucknow is not one in Pune
    How many, as a start
    Calls from the regions you serve
  • Cover this: Real 8 kHz phone audio

    Why
    A browser or office recording is wider than a phone line
    How many, as a start
    All final test calls from the phone line
  • Cover this: Noise

    Why
    Traffic, a TV, a shop counter, speakerphone
    How many, as a start
    About a quarter of the set
  • Cover this: Mixed speech

    Why
    Hinglish and Tamil with English words
    How many, as a start
    A fair share, as many as your callers use
  • Cover this: Names, numbers, dates

    Why
    These are what the call is for
    How many, as a start
    Calls where each one appears
  • Cover this: Silence and music

    Why
    Hold music and dead air
    How many, as a start
    5 to 10 files with no speech
A test set drawn from many groups: languages, regions, noise levels, mixed speech and silent files, kept apart from the data used to tune a system.

Two more habits help. First, split your calls in two. Use one part to tune settings (a word list, a language choice) and keep the other part sealed for the final score. A system tuned on its own test calls looks better than it is. Second, pick calls before you listen to any system's output. If you remove "bad" calls after seeing the result, you are choosing the answer.

How many calls are enough?

Enough that a clear difference between systems does not vanish when you add ten more calls. There is no single number. We suggest 30 to 50 calls per language to see patterns, and more when two systems are close. We have not measured the right size for your case, and we do not claim a fixed rule.

Do not stop at the average. NIST's SCTK toolkit includes a tool, sc_stats, described as sclite's statistical significance comparison tool (NIST SCTK, read 16 November 2026). A significance test asks whether a gap could be luck. If you do not run one, at least score the two halves of your test set separately. If system A wins on one half and loses on the other, you do not have a result yet.

How do you write the correct text?

Have one person type what was said, have a second person check it, and write the rules down first. The correct text is called the reference. Every score you compute depends on it, so errors in the reference become errors in the score.

  1. Write the guidelines first. One page: which script to use, how to write English words, how to write numbers, and how to mark unclear speech. IndicVoices lists "comprehensive transcription guidelines" among what it shares (IndicVoices, read 16 November 2026), which shows how seriously dataset builders take this step.
  2. Use native speakers. A person who speaks the language and knows the topic. Insurance words, loan words and place names are easy to mis-hear.
  3. Do a second pass. The second person checks the first, line by line.
  4. Measure the two humans against each other. Score person one against person two with the same method you use for systems. The gap is the noise floor of your reference. If two careful people disagree on 6% of the words, a difference of 1% between systems means little.
  5. Keep a spelling list. Roman spellings of Hindi ("accha", "acha") and Tamil words with English letters vary. The list settles them.
  6. Mark what you cannot hear. Do not guess. Remove those spans from scoring, and say so.

How do you clean the text before scoring?

Clean the reference and every system's output with the same rules, then score. Without this, a comma or a different spelling counts as an error that no listener would hear. This table shows the rules to decide. All are choices, so write down yours.

  • Area: Script

    Question to decide
    Native script, Roman, or both?
    Example
    "नमस्ते" or "namaste", not a mix on one side only
  • Area: Hinglish

    Question to decide
    How are English words written inside a Hindi line?
    Example
    "policy" or "पॉलिसी"
  • Area: Numbers

    Question to decide
    Digits or words?
    Example
    "4410" or "चार चार एक शून्य"
  • Area: Amounts

    Question to decide
    How is a rupee amount written?
    Example
    "Rs 12,499" or "बारह हज़ार चार सौ निन्यानवे रुपये"
  • Area: Spelling variants

    Question to decide
    Which spellings count as one word?
    Example
    "accha", "acha", "अच्छा"
  • Area: Fillers and sounds

    Question to decide
    Are "haan", "hmm" and "uh" scored or dropped?
    Example
    Decide once, for all systems
  • Area: Punctuation and case

    Question to decide
    Remove all?
    Example
    Yes, for scoring
  • Area: Unicode form

    Question to decide
    One stored form for each letter
    Example
    Normalise both sides

One trap is worth naming. A cleaner that deletes every Unicode mark can break Hindi, because vowel signs are marks. We show the effect of Whisper's basic normaliser on one Hindi phrase in our Whisper test guide. Run any cleaner on a few of your own sentences and read the output before you trust it.

How do you score word and character error rates?

Count the errors against the reference, then divide by the size of the reference. WER counts substitutions, deletions and insertions in words. CER counts them in characters. Our glossary entry on word error rate has the formula and an example.

Two free tools do the counting. NIST's sclite scores recognition output against a reference and reports alignment statistics (sclite documentation, read 16 November 2026). The open-source jiwer package computes WER and CER, and its README says it works on one or more reference and hypothesis sentences (jiwer, read 16 November 2026). This short script is our own, built on jiwer. We ran it on two sample lines to check it works. It reads a table of calls, cleans both sides, and prints one line per language and group:

import csv
import re
import unicodedata
from collections import defaultdict

from jiwer import cer, wer

def clean(text):
    text = unicodedata.normalize("NFC", text).lower()
    # remove punctuation (P) and symbols (S), keep marks (M): vowel signs live there
    text = "".join(" " if unicodedata.category(c)[0] in "PS" else c for c in text)
    return re.sub(r"\s+", " ", text).strip()

groups = defaultdict(lambda: ([], []))
with open("manifest.csv", encoding="utf-8") as f:  # file, language, bucket, reference
    for row in csv.DictReader(f):
        with open(f"hyp/{row['file']}.txt", encoding="utf-8") as h:
            hyp = clean(h.read())
        refs, hyps = groups[(row["language"], row["bucket"])]
        refs.append(clean(row["reference"]))
        hyps.append(hyp)

for (language, bucket), (refs, hyps) in sorted(groups.items()):
    print(language, bucket, len(refs), "calls",
          "WER", round(wer(refs, hyps), 3), "CER", round(cer(refs, hyps), 3))

Run it once per system, with that system's text files in the hyp folder. Read the output in groups, not as one number: language, noisy or quiet, with or without mixed speech. A system that wins overall can lose in the group you care about most.

When is CER more useful than WER?

When words are long or spelled in more than one way. One wrong letter makes a whole word wrong in WER. In languages whose words run long, such as Tamil or Malayalam, that can turn a near-miss into a full error. CER shows how close the output was. This is our reasoning, not a measured result, so check on your own data. Report both, and say which one you used to decide.

How do you test names, numbers and yes or no answers?

Score each important detail as right or wrong on its own, because a transcript can have a low WER and still get the one number you needed wrong. Pick the details your calls need and count them.

  • Detail: Name

    How to score
    Exact match, or a spelling you accept
    A made-up example
    "Lakshmi Narayanan" against "Lakshmi Narayan"
  • Detail: Phone or policy number

    How to score
    Every digit right, or the whole number fails
    A made-up example
    9 7 4 2 1 against 9 7 4 2 7
  • Detail: PIN code

    How to score
    Six digits, all right
    A made-up example
    500081
  • Detail: Rupee amount

    How to score
    Value right, not just the words
    A made-up example
    Twelve thousand four hundred and ninety-nine
  • Detail: Date

    How to score
    Day and month right
    A made-up example
    "Kal" against "parson"
  • Detail: Yes or no

    How to score
    Meaning right: "haan", "ji", "nahi", "theek hai"
    A made-up example
    Count flipped meanings as serious

Report them as counts: "37 of 50 names right, 31 of 50 numbers right." A manager can act on that. Our guide on how to test a voice agent shows the number, date and amount cases to try at the agent level.

How do you measure delay?

Time two things: how long after the caller stops speaking you get the final text, and, if the system streams, how long before the first partial text appears. On a voice agent, delay adds to every reply the caller waits for. Our guide to voice agent latency shows where speech-to-text delay sits in the full reply time.

  1. Use the same audio and the same place. Run every system from the same machine and region, so network distance does not decide the winner.
  2. Mark the end of speech. Find the moment the caller stops talking in the audio file. This is your zero.
  3. Record the time of the final text. The gap is the delay you measure.
  4. Repeat. Run each file several times and look at the median and the slow ones. One run tells you little.
  5. Say what you measured. A system that waits for a whole file is not comparable to a streaming one. Name the mode.

How do you keep the comparison fair?

Give every system the same audio, the same settings, the same cleaning and the same scoring, and write down everything that could differ. Fairness is a list of small checks, not a feeling.

A fair run: the same test calls go to each system, the same cleaning and scoring are applied, and the settings, versions and dates are recorded beside each result.
  • Same audio. Send the same files, in the same format, to every system.
  • Same settings. If you set the language for one, set it for all. If you use a word list for one, try it on the others.
  • Same cleaning. The reference and each output go through the same rules.
  • Same day. Systems change. Record the model name or version and the date you ran each one.
  • Blind scoring. If people judge errors, hide which system wrote which text.
  • Sealed test set. Nobody tunes on it.
  • Report groups. Show language, noise and mixed speech separately.
  • Say what you did not test. A test on Hindi says nothing about Telugu.

How Bananaflow fits

Bananaflow is an AI voice agent platform. These parts help with a test like this one:

  • Each agent picks its own speech-to-text, and you can change it without rebuilding the agent. You can run the same test calls against different choices.
  • A speech-to-text word list for brand names and jargon.
  • English plus 10 Indian languages, including Hindi, Tamil and Telugu, with switching mid-call. See Languages.
  • Hinglish written down as spoken.
  • A transcript and a recording after each call, and browser testing by voice or text, so you can set a transcript beside your reference.
  • Call data is not used to train models.

What should you remember about benchmarking speech to text?

  • Build the test set from your own 8 kHz calls, with many speakers, regions, noise levels and mixed speech.
  • Get permission to use recordings, and keep them private.
  • Write the reference with guidelines and a second person, and measure how far two humans differ.
  • Clean every text with one set of rules before scoring, and check the cleaner on Hindi, Tamil and Telugu sentences.
  • Score WER and CER, then names, numbers and yes or no on their own.
  • Measure delay, and keep the comparison fair and recorded.
  • Report groups, not one number, and never copy a score from another test.

FAQ

Frequently asked questions

  • How do you benchmark speech to text on Indian calls?

    Run two or more systems on the same set of your own phone recordings, write the correct text by hand, clean every text with the same rules, and count the errors. Score word and character error rates, then score names, numbers and yes or no answers separately, and time the delay.

  • How many calls do I need in a test set?

    There is no fixed number. We suggest 30 to 50 calls per language to see patterns, and more when two systems are close. This is our rule of thumb, not a standard. A significance tool such as NIST's sc_stats can tell you whether a gap could be luck.

  • What is word error rate, and is it enough?

    Word error rate is the number of substitutions, deletions and insertions needed to turn the output into the correct text, divided by the number of words in the correct text. It is not enough alone. A transcript can score well and still get one phone number wrong, so also score names, digits and yes or no answers.

  • Should I use WER or CER for Indian languages?

    Report both. In word error rate, one wrong letter fails the whole word, which can look harsh where words are long or have several spellings. Character error rate shows how close the output was. Decide before the test which one you will use to choose, and clean the text first so spelling variants do not decide the result.

  • Can I use public datasets instead of my own calls?

    Use them as a guide, not a replacement. Vistaar collects 59 benchmarks across Indian languages and domains, and IndicVoices holds 7,348 hours of natural speech in 22 languages. Neither is your callers on your phone line, so your own recordings are the test that counts.

  • How do I measure speech-to-text latency?

    Mark the moment the caller stops speaking in the audio, then record when the final text arrives, and for streaming systems when the first partial text appears. Run every system from the same place, repeat each file several times, and report the median and the slow runs, not one run.

  • How do I keep a comparison between two systems fair?

    Send the same audio and use the same settings, the same cleaning rules and the same scoring. Record the model name or version and the date of each run, keep the test set sealed so nobody tunes on it, and report results by group such as language, noise and mixed speech.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Similar articles

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot