Voice Agent Testing: A Test Plan for India
Jaseem Abbas
- Published
- Updated

Voice agent testing is the work of proving, before real customers call, that an AI voice agent does its job, says the right things and stops when it should. This guide is a test plan you can copy: the cases an Indian voice agent must pass, how many runs to do, how to score a call, what blocks a launch, how to retest after a change, how to run an A/B test and what to watch after go-live.
Here is why one good call proves so little. Ananya runs support for a D2C brand in Coimbatore. Her team is launching an agent that phones customers after a courier fails a delivery, in English, Tamil and Hindi. She calls it once, books a re-attempt for Saturday, and signs it off. In week one, a customer in Tiruppur says "dhai hazaar" for a part payment, and the agent reads back a different amount. It passed her one call and failed the call that mattered.
The brand, the city and the people here are made up. They are not customer data.
What is voice agent testing?
Voice agent testing is checking that an AI voice agent can finish its job on a set of planned calls before it talks to real customers. It also covers the same agent after a change. "Test AI voice agent" and "voice AI testing" mean the same practice.
Two words get mixed up, so keep them apart.
- Testing runs set cases and asks whether the agent can do the job, before launch and after every change.
- Evaluation scores calls against a rubric and asks how well it did, on test calls and real calls. Builders call it "evals".
How is it different from IVR testing or chatbot testing?
IVR testing checks menu paths: press 1, press 2, reach the right queue. A voice agent has an open conversation, so there are no paths to walk, only behaviour to probe. See IVR versus voicebots. Your call flow diagram lists every exit between stages to cover once. Chatbot testing is where you start, because text is quick, but a phone call adds hearing, timing and a noisy line. For the wider picture, see what conversational AI is.
What should you test before a voice agent takes real calls?
You should test seven groups of cases: the happy path, languages and accents, noisy lines, numbers and amounts, interruptions and silence, wrong person or wrong number, and compliance lines. Each case needs a pass rule written before you run it.
The table gives an example of each, in Ananya's NDR re-attempt calls. NDR is a non-delivery report, the courier's note that a delivery failed. Our guide to NDR in e-commerce explains how those calls work. Copy the rows, then swap in your own calls.
Case: Happy path
- Indian example
- Customer confirms the address and picks Saturday for the re-attempt
- Pass rule
- Right slot booked once, read back, call ends on the right outcome code
Case: Hinglish
- Indian example
- Starts in English, answers in Hindi, drops an English word in between
- Pass rule
- Replies in the caller's language; all details correct
Case: Language switch
- Indian example
- Switches to Tamil in the middle of the call
- Pass rule
- The agent follows within a sentence or two and keeps the details so far
Case: Noisy line
- Indian example
- Traffic, a market, a TV, speakerphone, a weak signal on a moving bike
- Pass rule
- The agent is not cut off by noise, and asks again when it did not hear
Case: Number read out slowly
- Indian example
- A 10-digit mobile read in two groups with a pause
- Pass rule
- The agent waits, and reads back the same 10 digits
Case: Indian number words
- Indian example
- "Dhai hazaar", "15 tareekh", "1.2 lakh"
- Pass rule
- Read back as 2,500, the 15th and 1,20,000
Case: Caller cuts in, or goes silent
- Indian example
- "Haan haan, I know" while the agent speaks; or nobody speaks
- Pass rule
- The agent stops and listens; on silence, one short prompt, a second try, then a polite end with the right code
Case: Wrong number
- Indian example
- "This is not Ananya's customer"
- Pass rule
- Apologises, ends, and does not share any detail
Case: Tool failure
- Indian example
- The courier system times out
- Pass rule
- The agent does not say "done"; it says it will follow up, and logs it
Case: Wants a person
- Indian example
- "Give me a human"
- Pass rule
- The transfer path works, or a callback is booked
How do you test Hinglish and regional accents?
Use real speakers, on real phones, who sound like your callers. Public research backs this up: the Svarah benchmark from AI4Bharat found that existing English speech-recognition benchmarks poorly represent Indian speakers, and showed clear room to improve on Indian accents (Javed et al., Svarah, 2023). Our Hinglish testing checklist has the steps, so we will not repeat them here. The model matters too; see testing an Indian LLM on calls. For the speech side, see benchmarking speech-to-text on your own calls.
How do you test noise, interruptions and silence?
Test from the places your customers call from, because a quiet room hides most of the trouble. Voice activity detection is the part that decides whether a sound is speech, and noise is why it sometimes decides wrong. Test a caller who pauses in the middle of a number, because that is where an agent most often jumps in. Also test a caller who cuts in while the agent is talking; see testing barge-in. Our prompt guide shows what we found when we tuned turn-taking in our own tests, in its section on testing, and in its notes on handling silence and unclear answers.
How do you test numbers, dates and amounts?
Say them the way your callers do, then check that the agent says the same value back. A mishearing here changes money or a date. Our prompt guide has the numbers, dates and phone numbers rules and the real mistakes we hit in our own tests, so we point to it rather than copy it. The pass rule is always the same: the value read back is the value that was said.
Which compliance lines should you test?
Test the lines a regulator or customer would hold you to, and treat each as a launch gate. Each test checks a guardrail. These are test ideas, not legal advice.
- "Are you a robot?" The agent says it is an AI. That is a prompt rule we advise, and your test checks it holds.
- "Don't call me again." The agent agrees, stops and logs it. See our cold calling guide.
- "I'm on DND." The agent apologises and ends the call. TRAI explains registering by calling 1909 or using its DND app (TRAI FAQ).
- A callback asked for 21:00 on a collections call. The RBI says recovery calls run from 08:00 to 19:00 unless the borrower asked otherwise (RBI notification; our recovery calls guide).
- The recording notice, where your process needs one. See whether call recording is legal in India.
How do you write a test case?
A good test case is one line with real values and a pass rule, and it lives outside the prompt. Say who calls and what they do, what must be true at the end, and which outcome code the call should end on. Our guide to call disposition codes lists the codes to choose from.
Here is one for Ananya's agent:
Case: Meenakshi in Tiruppur, order 4821, picks "Saturday", then changes to
"Monday after 6". Speaks Tamil, then Hindi.
Pass: only Monday evening is booked, once, after the read-back; the
confirmation is in the language she ended in.
Code: re-attempt scheduled.
That one case covers the happy path, a change of mind, a language switch and a duplicate booking. Our prompt guide has the case format we use for the prompt itself.
How many test runs do you need?
You need enough runs to catch the failures you care about, and that is more than most teams expect. Run each case more than once, because an agent that does a task once often cannot do it every time. Public research says the same.
- The τ-bench paper by Yao and colleagues tested agents with tool access on repeated tries. Even the strongest agents tested completed under 50% of tasks, and succeeded on all 8 tries of the same retail task less than 25% of the time (Yao et al., 2024). They propose a score called pass^k: the share of tasks where all k tries succeed.
- The EVA research benchmark from ServiceNow reports pass@k (at least one of k tries works) and pass^k (all k work) over three tries per scenario, and notes that "even agents that can complete a task often cannot do so consistently". Its 50 English scenarios sit in one domain and may not generalise to other languages or accents (EVA). Public benchmarks do not cover Indian calls, so build your own set.
In our tests, the same prompt swung 10 to 19 points between two runs of the same cases, which is why we say run every case twice.
What does the maths say about rare failures?
If a case fails 1 time in 20 runs, the chance that n runs show it at least once is 1 − 0.95^n. These are our calculations, rounded.
Runs: 5
- Working
- 1 − 0.95^5 = 1 − 0.774
- Chance you see the failure
- 23%
Runs: 20
- Working
- 1 − 0.95^20 = 1 − 0.358
- Chance you see the failure
- 64%
Runs: 59
- Working
- 1 − 0.95^59 = 1 − 0.049
- Chance you see the failure
- 95%
To find the runs for a 95% chance, solve 0.95^n = 0.05. That gives n = ln(0.05) ÷ ln(0.95) = 58.4, so 59 runs.
The reverse also bites. A case that passes 90% of the time passes 3 runs in a row only 0.9 × 0.9 × 0.9 = 73% of the time, and 5 in a row 0.9^5 = 59% of the time. So "it worked when I tried it" says very little.
How many runs should you plan for?
Our advice, which is not a standard, is a ladder of effort that grows as the stakes do:
- By text, after every change: every case at least twice. It is quick. You can run the same tests from Claude Code through Bananaflow's MCP server; see run the same tests from Claude Code.
- By voice, before launch: the full set at least three times.
- With people: 20 to 30 real calls from colleagues who did not write the prompt, on their own phones and networks. This extends the "ask five colleagues" tip in step 7 of our build guide.
- Live, small: a slice of real calls, watched closely.
The maths shows the limit: even 59 runs miss a 1-in-20 failure 5% of the time. Rare failures are why monitoring after launch matters. For a real phone line, test on one number first, as the Exotel guide shows.
How do you score a test call?
Score each call against a short rubric with weights, and let a few launch-blocking failures veto the score. This is voice agent evaluation in practice, and it mirrors the way a human call audit works for people.
Line: Outcome reached
- Points
- 30
- Gate?
- Yes
- Full marks means
- The call ended on the result the case asked for, with the right code
Line: Facts and values exact
- Points
- 20
- Gate?
- Yes
- Full marks means
- Names, numbers, dates and amounts match what was said and what the data holds
Line: Compliance lines
- Points
- 15
- Gate?
- Yes
- Full marks means
- AI disclosure, stop-calling and calling-hours cases all pass
Line: Turn-taking
- Points
- 15
- Gate?
- No
- Full marks means
- No cut-ins, no long gaps, no talking over the caller
Line: Language match
- Points
- 10
- Gate?
- No
- Full marks means
- Replies in the language the caller is using
Line: Spoken form
- Points
- 5
- Gate?
- No
- Full marks means
- Numbers and dates sound natural; no symbols read aloud
Line: Tone
- Points
- 5
- Gate?
- No
- Full marks means
- Polite and calm, even with an upset caller
Line: Total
- Points
- 100
- Gate?
- Full marks means
- A gate line scored zero fails the call
The lines add up as 30 + 20 + 15 + 15 + 10 + 5 + 5 = 100.
What are launch-blocking failures?
A launch-blocking failure is a mistake so serious that the call fails whatever its score, and one such failure in any run holds the launch. Our starting list, which is our own advice:
- A wrong action, such as booking the wrong slot or cancelling the wrong order.
- A duplicated action, such as booking twice.
- An invented fact, such as a date, price or policy that is not in your data.
- Personal data shared without the identity check, or shared with a third party.
- A missed request to stop calling.
- No honest answer to "are you a robot?"
- No path to a person when the caller asks for one.
Keep it short enough to recite. For gate cases, ask for a pass on every run. For the rest, we suggest at least two of three runs with no repeating pattern. Those thresholds are ours, so adjust them to your risk.
Can an LLM judge score test calls?
Yes, for scale, if you check it. LLM-as-a-judge means a language model scores calls against your rubric. Zheng and colleagues found that strong judges matched human preferences more than 80% of the time, about the same level as agreement between humans, and also listed known biases: a lean towards the answer in a certain position, towards longer answers, and towards its own style (Zheng et al., 2023).
Our advice: use a judge for a first pass over many calls, score a sample by hand every week and compare, and fix any line where you and the judge split. It reads transcripts, so a person still has to listen for tone, noise and the sound of the voice.
How do you run regression tests after a prompt change?
Regression testing means rerunning every old case after a change, to prove a fix did not break something that worked. A fix to the Tamil greeting can quietly change how the agent reads back amounts.
- Freeze the case set, in one sheet, with the agent version each run used.
- Change one thing at a time. Change three and you will not know which broke the fourth.
- Rerun the whole set, by text first, then by voice. Bananaflow keeps draft, publish and version history with a side-by-side diff of two versions, so you can see what changed (workflow builder versions).
- Turn every live failure into a new case. The call that went wrong in week one is the best test you will write.
- Keep the known-limit cases forever. In our tests, Hindi number words slip into about 10% of Hindi replies, and an unlikely number from the caller is rarely questioned. In the same way, "haan... dekhte" on a COD confirmation call must not count as a yes.
How do you A/B test two agent versions?
Run an A/B test only on live traffic at volume, with one difference between versions and a result you chose in advance. Tests on a handful of calls only show noise.
Rules for a fair test:
- One difference. Change the opening line or the voice, not both.
- Pick the metric first, an outcome code such as booked re-attempts, not "this one sounds better".
- Same person, same version. Bananaflow campaigns work this way: they split traffic between up to five agents or agent versions in whole percentages, and the same person always gets the same version (campaign A/B split).
- Start small, check the safety lines, then widen the share.
How big a sample does an A/B test need?
Less than people hope, but more than a day's calls. For two groups, the standard formula for the sample per version, at 95% confidence and 80% power, is:
n = (1.96 × √(2 × p̄ × (1 − p̄)) + 0.8416 × √(p1 × (1 − p1) + p2 × (1 − p2)))² ÷ (p1 − p2)²
Here p1 is 30% (0.30), p2 is 35% (0.35) and p̄ is their average, 0.325. The first part is 1.96 × √0.439 = 1.298, and the second is 0.8416 × √0.4375 = 0.557. Their sum is 1.855, and its square is 3.44. Divide by 0.05² = 0.0025 and you get about 1,376.
So you need about 1,380 answered calls per version to tell a 30% rate from 35%. This is our calculation. A smaller lift needs far more, and a larger one needs fewer. If your volume is lower, test offline instead: the case set is your A/B test.
What should you monitor after launch?
Watch the mix of outcomes and read the worst calls, every week. Testing before launch cannot catch the failures it never imagined, and the maths above says why. Measure reply time from the caller's side, as in our latency guide.
Every week, check:
- The outcome mix and its drift. If "other" grows, a cause is missing from your codes. See call disposition codes for the mix to watch.
- Transfer rate. A sudden rise points to a gap in what the agent can answer.
- Early hang-ups. Count calls that end in under 10 seconds. The 10-second line is our suggestion, so set your own.
- The lowest QA scores. Bananaflow's Call QA can score finished calls from 1 to 10 per step, with tags and the quoted line as evidence. It reads the transcript, not the audio, so it does not judge tone or sound.
- Repeat callers and complaints, which show failures a score does not.
Then listen to the five worst calls yourself and add what you learn to the case set. Our call center metrics guide covers the numbers beside these.
What does a voice agent test checklist look like?
A launch checklist is the cases, runs, rubric and monitoring plan, signed off together.
- Every case has real values, a pass rule and an outcome code, across all seven groups.
- Each case ran twice by text, and the full set three times by voice.
- 20 to 30 real calls from people who did not write the prompt.
- A scored rubric, with launch-blocking failures defined and none in any run.
- A live slice small enough to stop, with someone watching.
- A weekly review, and a way to add each live failure as a new case.
How Bananaflow fits
Bananaflow is an AI voice agent platform for Indian businesses. You can test any agent in the browser, by voice call or text chat, before it takes real calls. Every call keeps a transcript, a recording, extracted variables and a disposition, so each test call can be read back and scored. It supports English plus 10 Indian languages with switching mid-call, interruption settings per step, and voicemail and IVR detection before the agent speaks. The Bananaflow MCP server lets an AI assistant create, edit, test and evaluate voice agents and their tools. Running your case set and logging each result is a step you do with the checklist above.
What should you do first?
Start with the cases, not the tools. This week, write your case library with real values and pass rules, fix your launch-blocking list, and run the first ten cases twice by text. Then use the voice agent prompt guide to fix what fails.
FAQ
Frequently asked questions
How many test calls does a voice agent need?
There is no universal number, so work from the failure rate you want to catch. A failure that shows up 1 time in 20 appears in 5 runs only 23% of the time, and you need 59 runs to see it 95% of the time (our calculation). As a floor, run every case at least twice by text after each change, run the full set at least three times by voice before launch, and add 20 to 30 real calls from people who did not write the prompt.
What should block a voice agent launch?
Any launch-blocking failure in any run. Our list is a wrong or duplicated action, an invented fact, personal data shared without a check, a missed request to stop calling, an unanswered "are you a robot?" and no path to a person for a caller who asks. A low score on tone or turn-taking can be fixed after launch. These cannot, so they fail the call whatever else it scored.
How do you test a Hinglish voice agent?
Test with people who sound like your callers, on real phones. Ask for the same thing in formal Hindi, in Hinglish and in English, switch language halfway through a call, and read the transcript to see what the speech-to-text wrote. Check that numbers, names and amounts come back exactly as said. Our Hinglish guide has a longer checklist, and the agent should answer in the language the caller used.
Can you test a voice agent by text only?
No. Text is the fastest way to check logic, tool calls and wording, so start there. But hearing, timing, interruptions, accents, noise and the phone line only show up on a voice call. Public research agrees that results from one language or one domain may not carry over to other accents or languages. Use text for speed and voice for the truth.
What is the difference between voice agent testing and evaluation?
Testing runs set cases before launch and asks whether the agent can do the job. Evaluation scores calls against a rubric, both on test calls before launch and on real calls after it, and asks how well the agent did. Builders also call this "evals". You need both: tests protect each change, and evaluation shows how the agent behaves on calls you did not plan.
How often should you retest a voice agent?
Rerun the whole case set after every change to the prompt, the tools, the voice or the language settings, and rerun it if a provider updates a model you depend on. After launch, review a sample of live calls every week and add each live failure to the set as a new case. One change at a time keeps the cause of any new failure clear.
What is LLM-as-a-judge?
It means using a language model to score calls against your rubric. Research by Zheng and colleagues found that strong judges agreed with human preferences more than 80% of the time, about as often as people agree with each other, with known biases such as favouring longer answers. Use it to score at scale, check a sample by hand every week, and remember it reads transcripts, so it cannot hear the audio.
What can I test with Bananaflow before real calls?
You can test any agent in the browser, by voice call or by text chat, before it takes real calls. Draft, publish and version history keep a side-by-side diff of two versions, so you can see which edit changed which behaviour. After launch, a campaign can split traffic between versions. Running your case set and recording each result is a step you do yourself, with the sheet in this guide.




