Skip to content

Bananaflow MCP is live.

Read the docs

How to Test Whisper on Hindi Phone Calls

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published
Cover for "How to Test Whisper on Hindi Phone Calls": a white line diagram of one sound wave, under a pill reading Whisper, forking into two lines of text: the top line long and whole under a lit pill reading Clean text, the bottom line broken into short pieces, drawn over a soft, out-of-focus nerine in muted clay peach, its thin, curling petals arching up into veils at both ends, with a soft mint haze at the left end and a soft rose glow at the right.

To test Whisper on Hindi phone calls, you run the open-source model on a small set of your own 8 kHz recordings, write the correct text by hand, clean both texts the same way, and count the errors. Whisper is OpenAI's open-source speech recognition model. Nobody can tell you its Hindi score on your calls, because the score depends on your line, your callers and how you clean the text. This guide shows the steps, and one trap that quietly ruins Hindi scores.

Here is an example we made up. Rohit, a developer at a Pune insurance broker, runs Whisper on a call where a customer says, "haan, mera policy number hai चार चार एक शून्य". The text comes back with the policy number in Roman digits and the rest in Devanagari. His scoring script then deletes every Hindi vowel sign, so the reference and the output both turn into broken letters, and the score means nothing. The broker, the call and Rohit are made up. The scripting mistake is real, and it is easy to make.

What is Whisper, and why test it on Hindi phone calls?

Whisper is a general-purpose speech recognition model from OpenAI, released as open-source code with model weights under the MIT licence (Whisper repository, read 16 November 2026). It can transcribe speech in many languages, translate speech into English and detect the spoken language. A call to Hindi customers is one of the places where you should test it before you trust it.

The reason is simple. The published numbers were measured on other audio. The Whisper paper reports word error rates (WER) on Hindi for read-aloud datasets: on FLEURS the paper lists 102.3% for the tiny model and 21.5% for large-v2 (Table 13), and on Common Voice 9 it lists 108.3% and 21.9% (Table 11) (Radford et al., 2022, tables read 16 November 2026). A WER above 100% is possible when a model adds many extra words.

Two things follow. Model size changes the result a lot, so "we tried Whisper" means nothing without the size. And neither dataset is a phone call: they are people reading text into a microphone. For how a phone line changes things, see Hindi speech to text on calls. For buyers comparing tools, see call transcription for Indian languages.

What do you need before you start?

You need three things: 30 to 50 short recordings of your own calls, a person who can write down exactly what was said, and a laptop that can run Whisper. The count is our rule of thumb. It is enough to see patterns, not enough to publish a number.

  1. Recordings with consent. A recording of a caller's voice is personal data. Use only calls you have the right to use for this purpose, tell the people involved, and keep the files in a place with limited access. We explain the rules in our guide to the DPDP rules for call centers and in is call recording legal in India. This is not legal advice, so ask your own adviser if you are unsure.
  2. Varied calls. Different people and lines, a few from noisy places, plus 5 to 10 files with no speech, such as hold music.
  3. Reference text. For each call, one person writes what was said, in the script you will compare, and a second person checks it.
  4. Software. Install Whisper with pip install -U openai-whisper. It also needs the ffmpeg command-line tool on your path (Whisper setup, read 16 November 2026).
The path one phone recording takes through the test: 8 kHz call audio, a resample to 16 kHz, Whisper with a language setting, and a transcript ready for cleaning.

How do you handle 8 kHz phone audio?

Resample every file to 16 kHz mono once, with the same command, before you run Whisper. A phone call carries narrow audio at 8,000 samples a second, as the G.711 telephone codec defines it (ITU-T G.711, read 16 November 2026), and Whisper's audio code works at a 16,000 samples a second rate.

Resampling does not bring back sound that the phone line removed. It only makes the file the right shape. Whisper's own loader already calls ffmpeg with a 16,000 sample rate and one channel when you give it a file (audio.py, read 16 November 2026), so you can pass your 8 kHz file straight in. We still suggest converting once yourself, so that every model you compare later gets the same file.

If your recordings are from a studio or a browser, make them look like a phone call before you test. This command turns a wide recording into a narrow phone-style one, then back to 16 kHz for Whisper. The G.711 A-law codec name is pcm_alaw and the mu-law name is pcm_mulaw in ffmpeg. We ran this exact command pair on a test tone and the output files had the expected rate and codec:

# wide recording -> phone-style 8 kHz mono (G.711 A-law)
ffmpeg -i wide.wav -ac 1 -ar 8000 -c:a pcm_alaw phone.wav

# phone-style file -> 16 kHz mono for Whisper
ffmpeg -i phone.wav -ac 1 -ar 16000 -c:a pcm_s16le for_whisper.wav

A phone-style copy is only a stand-in for a real call, so use real calls for the final result. For why the band is so narrow, see Hindi speech to text on calls.

How do you run Whisper on a Hindi recording?

Run it from the command line with the model size and language set, and save the output as text. The command-line tool reads one or many files and writes the results to a folder (Whisper README, read 16 November 2026).

whisper for_whisper.wav --model medium --language Hindi --output_format txt --output_dir out_medium

In Python, the README shows the same job in three lines:

import whisper

model = whisper.load_model("medium")
result = model.transcribe("for_whisper.wav", language="hi")
print(result["text"])

The README lists six model sizes, from tiny (39 million parameters) to large (1,550 million), plus a turbo model (model table, read 16 November 2026). Test at least two sizes on the same files, to see what extra time and memory buy.

Should you set the language or let Whisper detect it?

Set the language when you know it, and run an auto-detect pass as a check. The command-line help says the --language option names the language spoken in the audio, and that leaving it empty makes Whisper detect the language (transcribe.py, read 16 November 2026). The README's detection example looks at one 30-second window of audio, so a call that starts in English and moves to Hindi can be labelled by its opening seconds only. Test whether that happens on your own files.

A good test has two runs per file: one with hi set, and one with the option left out. Count how many files auto-detect labels as something other than Hindi or English. The Whisper paper itself says its language identification on FLEURS is not competitive with prior supervised results (Table 5), so do not assume detection is always right.

What about Hinglish, and Roman or Devanagari output?

Whisper writes whatever script it predicts, so one call can produce Devanagari for Hindi words and Roman letters for English words in the same line. That is a normal Hinglish result, not a bug. Decide your target first, then write the reference in that target. Our guide to how Hinglish should be written down sets out the choices, and Hinglish voice agents covers the speech side.

Do not mix targets. If the reference is all Devanagari and Whisper writes "policy number" in Roman letters, the test counts an error that no listener would. Match the reference to the target, or map both sides to one script.

How do you clean the text before scoring Hindi WER?

Write one Hindi-safe cleaner, apply it to the reference and to every output, and never score raw text. WER counts a word as wrong if one character differs. A comma, a danda (the Hindi full stop "।"), a capital letter or a different way of writing a nukta dot all cause errors that no listener would call errors.

Why is Whisper's basic normaliser a trap for Hindi?

Because it replaces marks with spaces, and Hindi vowel signs are marks. The Whisper repository ships a BasicTextNormalizer. In the file we read on 16 November 2026, its remove_symbols step turns any character whose Unicode category starts with M (mark), S (symbol) or P (punctuation) into a space (normalizers/basic.py).

Devanagari vowel signs such as the ि in "पूछिए" are marks. Python's unicodedata says that ा, ि and ी are category Mc, and the dot above a letter (anusvara) and the nukta are Mn. We applied that same rule to the phrase "मेरा नंबर पूछिए" in a short script. It gave back "म र न बर प छ ए": vowel signs gone, words in pieces. A WER computed on that text would be close to meaningless.

We only applied its published rule to one phrase, not inside a model run. Do the same check on a sentence from your own data before you trust any normaliser for Hindi.

What should a Hindi-safe cleaner do?

Keep vowel signs, remove punctuation, and make small choices you can explain. This table lists the rules we suggest. They are our advice, and you may choose differently for your use.

  • Rule: Normalise Unicode to one form (NFC)

    Why
    The same letter can be stored in more than one way
    Watch out for
    Do this first, on both sides
  • Rule: Lowercase Roman letters

    Why
    "Policy" and "policy" are one word
    Watch out for
    Does not touch Devanagari
  • Rule: Remove punctuation and the danda, keep vowel signs

    Why
    Punctuation is not hearing
    Watch out for
    Do not delete Unicode marks. They carry vowels
  • Rule: Decide on the nukta dot

    Why
    ज़ and ज are one sound for many speakers, but two letters on paper
    Watch out for
    Pick fold or keep, and say which
  • Rule: Turn numbers into one form

    Why
    "4410", "चार चार एक शून्य" and "four four one zero" are one number
    Watch out for
    Choose digits or words, and apply to both sides
  • Rule: Fix Hinglish spellings

    Why
    "acha", "accha" and "अच्छा" are one word
    Watch out for
    Build a short list from your own calls
  • Rule: Collapse spaces

    Why
    Extra spaces make extra empty words
    Watch out for
    Do this last

Here is a small cleaner in Python that follows the first three rules and the nukta fold. It is our own code, not a Whisper function:

import re
import unicodedata

def clean(text: str) -> str:
    text = unicodedata.normalize("NFC", text).lower()
    text = text.replace("़", "")  # fold the nukta: ज़ becomes ज
    # remove punctuation (P) and symbols (S), but keep marks (M)
    text = "".join(" " if unicodedata.category(c)[0] in "PS" else c for c in text)
    return re.sub(r"\s+", " ", text).strip()

On "मेरा नंबर, पूछिए। Policy Number!" it returns "मेरा नंबर पूछिए policy number". Add your number and spelling rules on top, and keep them in a file that you version, so the next test uses the same ones.

How do you score the result?

Count word errors and character errors against your reference, then count the things the call needs. Word error rate is the number of substitutions, deletions and insertions needed to turn the output into the reference, divided by the number of words in the reference. NIST's sclite tool does this alignment and is part of the NIST Scoring Toolkit, described as a tool that compares the hypothesised text with the reference text and reports statistics (sclite documentation, read 16 November 2026). Our glossary entry on word error rate has the formula.

For a quick script, the open-source jiwer package computes WER and CER; its README shows from jiwer import wer, then wer(reference, hypothesis) (jiwer, read 16 November 2026). Run clean() on both strings first:

from jiwer import wer, cer

ref = clean(reference_text)
hyp = clean(whisper_text)
print("WER", wer(ref, hyp), "CER", cer(ref, hyp))
From a Whisper transcript to a score: clean both texts the same way, align them, then count word errors, character errors and the fields the call needs.

Report three numbers together: WER for the overall slip, CER (the same count by letter) for small slips inside words, and field accuracy, the one a manager acts on. Our guide on how to measure WER and CER on your own calls explains the field checks, and the general method across models is in benchmarking speech to text on Indian calls.

Why does Whisper write words on silent audio, and how do you test it?

Whisper can write text when nobody is speaking, so you test it with silent recordings and count every word it returns. The paper lists "complete hallucination", where a model outputs a transcript unrelated to the audio, as one of its failure types, along with repeat loops and missed first or last words (Radford et al., 2022, read 16 November 2026).

Phone calls are full of silence, and words from nothing are a risk if the text feeds a summary or an agent's reply.

  1. Collect 5 to 10 silent or music-only files. Make them the same 8 kHz phone-style audio as the rest.
  2. Run them like every other file. Same model, same settings.
  3. Write an empty reference. The right answer is no words.
  4. Count the words you get. Any word counts as an insertion. jiwer 4.0 and later defines this: with an empty reference, every word in the output counts as one insertion (jiwer README, read 16 November 2026).
  5. Check the settings that exist for this. The transcribe() function has no_speech_threshold, compression_ratio_threshold, logprob_threshold and condition_on_previous_text options, and the command-line tool has matching flags (transcribe.py, read 16 November 2026). Change one at a time and rerun the silent files.

The help text for condition_on_previous_text says turning it off may make the text inconsistent across windows, but makes the model less prone to getting stuck in a failure loop. Test that trade-off before you flip it.

How do you test names, numbers and Hinglish words?

Make a list of the details your call must get right, and score each one as right or wrong. Overall WER hides the errors that cost money.

  • Names. Take 20 names from your own customer list. Count how many came back spelled the way you accept. Whisper has an initial_prompt option that gives the model starting text for the first window (transcribe.py, read 16 November 2026). Test it with your list of names, and compare against a run without it. Do not assume it helps.
  • Digits. Score every number as a whole. One wrong digit makes the whole number wrong.
  • Yes or no. "Haan", "ji", "nahi", "theek hai". These are short and easy to lose.
  • English words inside Hindi. "EMI", "policy", "KYC": check the script they come out in.

A scoring sheet for 30 files can be this simple:

  • Run: small, language set

    WER
    your count
    CER
    your count
    Names right
    x of 30
    Numbers right
    x of 30
    Yes or no right
    x of 30
    Words on silence
    x words
  • Run: medium, language set

    WER
    your count
    CER
    your count
    Names right
    x of 30
    Numbers right
    x of 30
    Yes or no right
    x of 30
    Words on silence
    x words
  • Run: medium, auto-detect

    WER
    your count
    CER
    your count
    Names right
    x of 30
    Numbers right
    x of 30
    Yes or no right
    x of 30
    Words on silence
    x words

How should you read the Whisper paper's Hindi numbers?

Read them as proof that model size matters, not as the score for your calls. The Hindi figures quoted above come from read speech, and neither FLEURS nor Common Voice covers Hindi telephone speech. For Indian speech closer to a call, the IndicVoices dataset holds 7,348 hours of natural and spontaneous speech in 22 languages (IndicVoices paper, read 16 November 2026), and the Vistaar benchmark collects 59 Indian-language benchmarks (Bhogale et al., Vistaar, read 16 November 2026). We did not test either on phone audio. They show that testing on recordings like your own is normal practice.

How Bananaflow fits

Bananaflow is an AI voice agent platform. These parts touch the test you just read about:

  • Each agent picks its own speech-to-text, and you can change it without rebuilding the agent. That makes a side-by-side check on your own calls possible.
  • A speech-to-text word list for brand names and jargon.
  • English plus 10 Indian languages, with the agent following the caller's language and switching mid-call. See Languages.
  • Hinglish written down as spoken.
  • A transcript and a recording after each call, and browser testing by voice or text, so you can set a transcript beside your reference. See how to test a voice agent.
  • Call data is not used to train models.

What should you remember about testing Whisper on Hindi calls?

  • Resample once to 16 kHz mono, the same way for every file. Resampling does not restore what the phone line removed.
  • Set the language, and run an auto-detect pass to see how often it guesses wrong.
  • Never score raw text. Use a Hindi-safe cleaner that keeps vowel signs, and apply it to both sides.
  • Test silence on purpose, and count every word Whisper writes.
  • Test two or more model sizes on the same files. The paper's own tables show a very large gap between them.
  • Report WER, CER and the fields the call needs: names, digits and yes or no.
  • The Whisper paper's Hindi numbers are from read speech. Your own calls are the only test that counts.

FAQ

Frequently asked questions

  • Does Whisper support Hindi?

    Yes. Whisper is a multilingual speech recognition model and Hindi is one of the languages in its published results. The Whisper paper lists Hindi word error rates on the FLEURS and Common Voice 9 datasets, both read speech. Those are not phone calls, so test it on your own recordings before you rely on it.

  • How accurate is Whisper on Hindi phone calls?

    No honest single number exists, because it depends on the model size, your line and your callers. The paper reports 21.5% word error rate for large-v2 on Hindi FLEURS and 102.3% for the tiny model, on read speech. Run both sizes on 30 to 50 of your own calls and score them yourself.

  • Which Whisper model size should I test for Hindi?

    Test at least two sizes on the same files. The README lists six sizes from tiny (39 million parameters) to large (1,550 million), plus a turbo model. The paper's Hindi tables show a very large gap between the smallest and largest models, so the size you pick changes the result more than most settings.

  • How do I run Whisper on 8 kHz phone audio?

    Convert each file once to 16 kHz mono with ffmpeg, then run Whisper on the converted file. Whisper's loader also resamples for you, but converting yourself keeps every model you compare on the same input. Resampling does not bring back sound the phone line already removed.

  • Should I set the language to Hindi or use auto-detect?

    Set it to Hindi when you know the language, and run a second pass with auto-detect to see how often the guess is wrong. In the command-line tool, leaving the language option empty makes Whisper detect it. Mixed Hindi and English calls are where detection is most likely to slip.

  • Why is my Hindi word error rate wrong?

    Often because the text was cleaned badly. Whisper's basic normaliser replaces every Unicode mark with a space, and Hindi vowel signs are marks, so words split into pieces. Use a cleaner that removes punctuation but keeps vowel signs, apply it to the reference and the output, and decide your rule for the nukta dot and for numbers.

  • Does Whisper write words when nobody speaks?

    It can. The paper lists complete hallucination, where the output is unrelated to the audio, among its failure types. Test it with 5 to 10 silent or music-only recordings, give them an empty reference, and count every word returned as an insertion.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Similar articles

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot