Skip to content

Bananaflow MCP is live.

Read the docs

Hindi Speech to Text on Calls: What Breaks

Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.Jaseem Abbas

Published
Cover for "Hindi Speech to Text on Calls: What Breaks": a white line diagram of a voice squeezed by the phone line: a tall sound wave runs into a panel with a phone labelled Phone line, under a pill reading A phone call carries a narrow slice, and a low, thin wave comes out of it and runs on to a panel of text lines labelled Speech to text, drawn over a soft, out-of-focus lotus in muted rose, its broad petals blurred into veils, with a pale blush petal glow and a soft sky blue haze at the edges.

Speech to text for Hindi turns spoken Hindi into written words, and on phone calls it makes far more mistakes than on a clean recording. The phone line cuts the sound down, callers mix Hindi and English, names and numbers are hard to guess, and real lines are noisy. This guide is for teams that run Hindi voice agents or analyse Hindi calls. It explains what breaks, what research says, and how to check your own calls.

Here is a call to picture. A microfinance company in Jaipur has its agent phone a borrower to confirm an EMI date. She says, "हाँ जी, मैं सुनीता मीणा बोल रही हूँ, account number नौ-सात-चार... 7-4-2-1" (Haan ji, main Sunita Meena bol rahi hoon, account number nau-saat-chaar... 7-4-2-1). The transcript reads "Sunita Mina", "974 2 1", with one digit gone. The agent asks her to repeat twice. The same speech to text, tried on a clean voice note in the office, had looked perfect. The company, the borrower and the numbers are an example we made up. They are not customer data.

What is Hindi speech to text?

Hindi speech to text is software that listens to spoken Hindi and writes it down. The field is called automatic speech recognition (ASR), and searches for "Hindi transcription" and "Hindi speech recognition" mean the same thing: spoken words in, text out. For calls in other Indian languages, read our guide to call transcription in other Indian languages.

Three words get mixed up, so keep them apart:

  • Transcription writes down what was said, in the language it was said. "कल EMI कट जाएगा" stays Hindi.
  • Transliteration shows the same words in other letters. "Kal EMI kat jayega" is the line above in Roman letters.
  • Translation changes the language. "The EMI will be deducted tomorrow" is English.

The text can come out in Devanagari, in Roman letters, or in a mix. That choice changes how you search and score it, and we cover it under Devanagari or Roman.

Why is Hindi speech recognition worse on phone calls?

Because a phone call carries much less sound than the recordings most speech models learn from, and the sounds it drops are some of the ones that tell words apart. The phone network was built to carry a voice cheaply, not to keep every detail.

The classic phone codec, G.711, passes about 300 to 3,400 Hz and samples the sound 8,000 times a second (G.711 on Wikipedia; ITU-T G.711). This is called narrowband. Wideband audio, sometimes sold as HD voice, covers about 50 to 7,000 Hz (Wikipedia on wideband audio), and ITU-T G.722 is the standard for 7 kHz audio inside the same 64 kbit/s (ITU-T G.722). A human voice reaches from about 100 Hz up to 17 kHz, so a phone call keeps only a part of it.

The sound a phone call carries, about 300 to 3,400 Hz, set against the wider range of a recording made for speech models, showing what the phone line cuts off.

What does that cost? As an illustration, not a measured finding: pairs that differ by a small consonant detail, such as सात (saat, seven) and साठ (saath, sixty), are easy to confuse on a clean line and harder on a phone line, because the detail that separates them is thinner. The speech model has less to go on, and a digit swaps for a neighbour.

Researchers at IIT Madras describe the same problem. In their paper on telephone speech, they note that speech on telephone channels is generally narrowband, sampled at 8 kHz, and that models built for wideband 16 kHz audio do not perform well on it. They add that labelled narrowband data is scarce, which makes telephone speech recognition a real challenge (Sukhadia, Arunkumar and Umesh, "Channel-Aware Pretraining" for telephonic speech recognition). In another study, a model trained on mixed narrowband and wideband speech, and told which was which, improved narrowband results by 13% relative without hurting wideband ones (Mantena et al., "Bandwidth Embeddings for Mixed-bandwidth Speech Recognition"). The lesson is that the phone line is its own problem.

Mobile calls add more on top: compression, dropped packets and echo. Pauses on the line also change when an agent thinks a caller has finished, which we cover in our guide to endpointing and turn detection. The line also adds delay to every reply, which our voice agent latency guide breaks down.

What else breaks Hindi speech recognition on calls?

Four more things, on top of the phone line: Hindi-English mixing, names and places, numbers, and accents and noise. Each one breaks in a way you can check with a short test.

The five things that break Hindi speech to text on calls (narrow phone audio, Hindi-English mixing, names and places, numbers, accents and noise), each with one check you can run.
  • What breaks: Narrow phone audio

    Example
    सात (saat, seven) against साठ (saath, sixty)
    What goes wrong
    Close sounds blur, so a digit or a word swaps for its neighbour
    One check to run
    Play the same line over a real phone call and over a clean recording, and compare the transcripts
  • What breaks: Hindi-English mixing

    Example
    "कल EMI debit होगा न?" (Kal EMI debit hoga na?)
    What goes wrong
    English words come out in Devanagari, or Hindi words come out as the nearest English word
    One check to run
    Say ten mixed sentences on a call and check each English word
  • What breaks: Names and places

    Example
    मीणा (Meena) against Mina, तिरुवनंतपुरम (Thiruvananthapuram)
    What goes wrong
    Rare words are guessed, and one name has several spellings
    One check to run
    Read 20 real names from your customer list and count the exact matches
  • What breaks: Numbers

    Example
    डेढ़ हज़ार (dedh hazaar, 1,500), ढाई सौ (dhai sau, 250), nau-saat-chaar
    What goes wrong
    Hindi and English digits mix in one string, and grouped digits lose a digit
    One check to run
    Read a 10-digit number in groups of three or four and check every digit
  • What breaks: Accents and noise

    Example
    Hindi as a second language, a bus stop, a shop counter
    What goes wrong
    More errors for some speakers, and for loud lines
    One check to run
    Test callers from different regions and a loud line, not only office voices

How does Hindi-English mixing break it?

Callers switch between Hindi and English in the middle of a sentence, and a speech model has to follow the switch. That is code-mixing, and Hinglish is the common form. The three papers on Hinglish speech models, and the habits that cause errors, are in our guide to why Hinglish is hard for a voice agent. We do not repeat them here. What this post adds is that on a phone line, the switch gets harder, because the audio has less detail to separate an English word from a similar Hindi one.

Why do names and places go wrong?

Names are rare and have no single correct spelling. A model that has seen "Meena" often will still hand you "Mina" or "Meenaa", and a place like "तिरुवनंतपुरम" (Thiruvananthapuram), said by a Hindi speaker who is not from Kerala, is a word the model may never have seen.

Research on Hindi points the same way. The LAHAJA benchmark covers 12.5 hours of Hindi from 132 speakers across 83 districts. It found that existing open-source and commercial models performed poorly, and did worse for speakers from the North-East and South India, "especially with content heavy in named entities and specialized terminology" (Javed et al., "LAHAJA"). The recordings were not phone calls, so phone audio would add to the problem, not remove it. For the voice side of names, how the agent says them, see how a voice agent should handle names.

Why are numbers so hard?

Numbers come in more forms than any other kind of speech, and one wrong digit makes the whole number useless. A caller can say "nau saat chaar" in Hindi, "nine seven four" in English, or "नौ सात four" in a mix. Indian speech also has words that are quantities, not digits: "डेढ़" (dedh, one and a half), "ढाई" (dhai, two and a half) and "साढ़े" (saade, plus a half). Callers also read long numbers in chunks, and a pause between chunks can cost a digit if the agent decides the turn is over. That is a timing problem, covered in our guide to endpointing and turn detection.

In the Jaipur example, one digit was lost at a chunk boundary and another was heard as its neighbour. Two small errors made the whole account number wrong.

How do accents and noise change the result?

Hindi is spoken in many ways, and most Hindi speakers use it as one of several languages. In the LAHAJA study, 122 of the 132 speakers were non-native Hindi speakers, speaking it as their second, third or fourth language, from 18 states (Javed et al., "LAHAJA"). A model trained mostly on one kind of Hindi will make more mistakes for the rest.

Noise adds to this. A shop counter, a bus, a phone on speaker. The Gram Vaani challenge used spontaneous telephone speech in regional variations of Hindi with natural background noise, and crowdsourced transcripts of uneven quality (Bhanushali et al., "Gram Vaani ASR Challenge"). It is the closest public picture of what a Hindi phone line looks like.

How accurate is Hindi speech to text on real phone calls?

It depends on your callers, your line and your wording, so no honest single number exists. The best public research number we found is from the Gram Vaani challenge at Interspeech 2022. It released about 1,100 hours of Hindi telephone speech, only 100 of them with labels. Its baseline systems, each trained on those 100 labelled hours, scored a word error rate (WER) of 29.7% with a character error rate (CER) of 15.1% on one, and 32.9% with 19.0% on another, on the evaluation set (Bhanushali et al.).

Read that carefully:

  • These are research baselines from 2022, not products.
  • The CER is lower than the WER. Many errors are small slips inside a word, not a wrong word.
  • The speech was spontaneous, noisy and regional, like your callers.

Why do vendor pages show much lower numbers? Most are measured on clean recordings of people reading text aloud. That says little about someone on a bad line saying "haan, woh... mera number hai...". A low figure on read speech and a high one on phone speech can both be true of the same system.

How do you measure WER and CER on your own calls?

Score the things the call needs, on your own phone audio, with your own callers. Word error rate and character error rate are useful for tracking change. On a live agent, the score that matters is whether the name, the digits and the yes or no came out right.

WER and CER in plain words. WER counts the substitutions, deletions and insertions needed to turn the transcript into the correct text, divided by the number of words in the correct text (Wikipedia on word error rate). It can go above 100% if the system adds many extra words. CER counts the same changes letter by letter.

Why CER helps for Hindi. Hindi has spelling variants that a person sees as the same word. A different vowel sign (ि or ी), a missing dot under a letter (nukta), or "Meena" against "Mina" all count as a whole wrong word in WER. In CER they cost one character. Normalise first: pick one script, turn digits and number words into one form, and lowercase the Roman text. Then the score reflects hearing, not spelling.

Entity accuracy, the score that matches the call. Pick the few details the call must get right, and mark each as right or wrong:

  1. Name. Exact match, or an accepted spelling.
  2. Every digit. One wrong digit means the whole number is wrong.
  3. Yes or no. Including "haan", "ji", "nahi" and "theek hai".
  4. Date and amount. "Kal" and "parson" are one word apart.

Report each as a count: "41 of 50 names right, 33 of 50 numbers right". That tells a manager what to fix, which a single WER does not. If you are weighing Whisper, read how to test Whisper on Hindi phone calls. To compare several options on the same calls, see benchmarking speech-to-text on Indian calls.

Test on the phone line. An office recording, a WhatsApp voice note or a browser test can use wider audio than a phone network. Place real calls from real mobile phones, including a few from noisy places. We set out the full method for recorded calls (30 to 50 calls, hand transcripts) in our guide on how to test accuracy on your own calls, and the test cases for numbers and accents in how to test a voice agent. Here are the two changes for a live agent:

  • Score the agent's answer too. If the transcript says "974 2 1" and the agent reads back "9-7-4-2-1", that is a caught error. If it carries on, it is a failure. Count both.
  • Count repeats. How often did the agent ask the caller to say it again? A high rate is a quick sign that hearing is the problem.

What actually helps Hindi speech recognition on calls?

Design the call so the agent confirms what it can, and keep the hard guesses small. This is advice from how these systems behave, not a promise about any one setup.

  1. Confirm instead of transcribing cold. If your data already holds the caller's name and order number, the agent does not need to hear them from nothing. It can say "क्या मैं सुनीता मीणा जी से बात कर रही हूँ?" (Kya main Sunita Meena ji se baat kar rahi hoon?) and listen for a yes. That turns a hard transcription task into an easy yes or no.
  2. Ask for numbers in groups, and read them back. "Please say the number in groups of three" and "Is that 9-7-4, 2-1?" catch most digit errors on the spot. Our voice agent prompt guide shows how to write these lines and how to collect names and numbers.
  3. Add the words you know to a word list. Research on contextual biasing found that speech models "struggle to recognize rare or unseen named entities", and that giving the model a list of likely names improved named-entity WER by up to 50.34% relative. That test was on an English read-speech set, not Hindi phone calls (Selvakumar et al., "Peeking Into The Future For Contextual Biasing"). The idea carries over, but the size of the gain on your calls is something you must measure.
  4. Choose the speech-to-text option by testing it on your calls. Compare candidates on your own 8 kHz recordings, in your language mix, using the entity scores above. Do not choose on a website number.
  5. Send failed turns somewhere safe. If the agent could not hear a name twice, it should confirm in another way or pass the call to a person, not guess a third time.

How Bananaflow fits

Bananaflow is an AI voice agent platform. These are the parts that touch speech recognition:

  • A speech-to-text word list for brand names and jargon. It helps the agent hear them.
  • English plus 10 Indian languages, with switching mid-call. The agent follows the caller's language from their first clear sentence. See Languages for the list.
  • Each agent picks its own speech-to-text provider and can change it without rebuilding the agent.
  • Template variables and a pre-call data fetch, so the agent can know a caller's name or order number before the call, and confirm instead of transcribing from scratch.
  • A transcript and a recording after each call, and browser testing by voice or text. You can use these to run the entity checks above. The QA step reads the transcript.
  • Call data is not used to train or fine-tune models, ours or our providers'.

Test on your own calls, as above, with the audio, accents and noise your callers bring.

What should you remember about Hindi speech to text on calls?

  • Speech to text for Hindi writes down spoken Hindi. On a phone call it is harder, because the line carries only about 300 to 3,400 Hz.
  • Five things break it: narrow phone audio, Hindi-English mixing, names and places, numbers, and accents and noise.
  • Research on spontaneous Hindi phone speech showed word error rates near 30% for research baselines. Do not read that as a score for any product.
  • Clean-audio numbers do not carry over to phone calls. Measure on your own calls, on a real phone line.
  • Score what the call needs: the name, every digit, the yes or no. Use CER next to WER for Hindi, because spelling variants inflate WER.
  • Confirm what you already know, read numbers back in groups, add known words to a word list, and send failed turns to a safer path.

FAQ

Frequently asked questions

  • What is Hindi transcription?

    Hindi transcription is writing down spoken Hindi, either by a person or by speech recognition software. It is not transliteration, which shows Hindi words in Roman letters, and it is not translation, which turns Hindi into another language. The written result can be in Devanagari, in Roman letters or in a mix, so decide which one your team needs before you compare tools.

  • Is Hindi speech to text accurate on phone calls?

    It is less accurate than on a clean recording, and how much less depends on your callers. Phone audio carries only about 300 to 3,400 Hz, callers mix Hindi and English, and lines are noisy. The best public research on spontaneous Hindi phone speech, the Gram Vaani challenge in 2022, reported word error rates near 30% for its research baselines. Those are not products, so test on your own calls.

  • Why does speech to text get Hindi names wrong?

    Names are rare words, so a speech model has often never seen that exact name, and one name can be spelled several ways, such as Meena and Mina. A 2024 Hindi benchmark found models did worse on content heavy in named entities, and worse for speakers from the North-East and South India. Your own customer data can help: if the agent already knows the name, it can confirm it instead of writing it down cold.

  • What is a good word error rate for Hindi?

    There is no single good number, because it depends on the audio and the speakers. A low figure on clean, read-aloud speech says little about a phone call. For a business, the useful measure is whether the name, the digits and the yes or no were right on your own calls. Measure that first, then use word error rate to track change over time.

  • What is the difference between WER and CER?

    Word error rate (WER) counts wrong, missing and extra words against a hand-written transcript. Character error rate (CER) counts the same mistakes by letter. For Hindi, CER is often the fairer view, because a small spelling difference, such as a different vowel sign or Meena against Mina, makes a whole word wrong in WER but only one character wrong in CER.

  • Can I transcribe Hindi audio for free?

    Yes, free tools exist and they can work well on a clear recording of one speaker. Phone calls are harder, so try any free tool on a few of your own call recordings first and check the names and numbers by hand before you rely on it.

  • How do I convert Hindi speech to English text?

    That is two jobs: transcribe the Hindi, then translate the text. Check the Hindi transcript first, because a mistake in the transcript carries into the translation. If you only need the Hindi words in Roman letters, that is transliteration, which is a third and simpler job.

  • Can a word list help the agent with names?

    Yes. On Bananaflow, a speech-to-text word list for brand names and jargon helps the agent hear them. To control how the voice says a name, respell it in the prompt. Call data is not used to train or fine-tune models, ours or our providers'.

Written by

  • Jaseem Abbas, founder of Bananaflow, in glasses and a dark top against a pale background.

    Jaseem Abbas

    Founder, Bananaflow

    Jaseem founded Bananaflow to build voice AI that understands how people in India really speak, including the switch between languages in the middle of a call.

Similar articles

Bring us one process. We'll pilot it on your real calls, with your team as the fallback.

Plan a pilot