Somewhere around call thirty on a Tuesday, a colleague walked past my desk, looked at my headset, and asked whether I'd finally lost it. I was calling our own Aiva Chat line, hanging up, changing one thing (a pause length, a word order, a greeting) and calling again.
I did this, on and off, for most of a week. I do a version of it most weeks. It's the least glamorous part of my job and probably the most useful.
Testing an AI receptionist this way sounds inefficient. We're a small team, automated tests exist, and a designer dialing a phone by hand looks like a strange use of scarce hours. But voice products fail in ways that don't show up anywhere except on a call. A transcript can be perfect while the call itself feels cold, rushed, or slightly off.
If you're about to put an assistant on your own business line, the checklist near the end is the practical part; the rest is what a week of headset time taught us about why it's on that list.
Why testing an AI receptionist by phone matters
As a designer, my instinct is to review things visually. A transcript of a call reads fine almost every time. The words are correct, the information is right, the grammar holds.
A transcript strips out what makes a call feel like talking to a person or to a machine: the gap between when you stop speaking and when the reply starts, the small acknowledgment that tells you you're being heard, the rhythm of how a sentence gets broken up.
None of that shows up if you're reading. All of it shows up if you're on the call, phone against your ear, the way a customer is. I've written about why voice design has to work this way: there's no canvas to review, no mockup to point at, only the call itself.
What the first call hides
The first call always sounds fine. That's the trap. A single cold call is the worst way to evaluate a voice experience, because you're paying full attention and forgiving of anything slightly off.
What call ten reveals
Call ten is where patterns appear. A phrase gets reused too often. A pause runs a beat long after one specific kind of question. A greeting that's technically warm starts to land flat.
What call fifty reveals
By call fifty, you stop hearing the content and start hearing only the shape. That's roughly where a real repeat customer ends up too. If the shape is wrong, fifty calls in is where it starts to grate, even when every answer was correct.
How our call testing works
The process is simple, which is partly why it works.
Step 1: Pick one flow and one variable
A week of calls might focus on rescheduling, or on the greeting, or on how the assistant handles being interrupted. Within that, I change one thing at a time so I know what caused any difference.
Step 2: Call as different people
I call as a rushed caller, a confused one, someone who talks over the assistant, and someone delivering bad news. Each persona surfaces different timing problems.
Step 3: Listen for shape, not words
I keep notes on moments that felt wrong, even when I can't yet say why. "Slightly irritated at 0:40" is a useful note. The reason usually becomes clear a few calls later.
Step 4: Change something small and call again
Most fixes are tiny: a pause, an order of phrases, a shorter confirmation. The test is whether the irritation goes away on the next ten calls.
Isn't this what automated testing is for?
It's a fair question. We do use automated checks for a large part of quality. Whether a fact is correct, whether a booking lands in the calendar, whether an intent gets recognized: tests cover those constantly, without anyone picking up a phone, and that should stay the case.
What automated testing can't do, at least not in a way I'd trust, is tell you whether a correct answer felt considerate. That's closer to asking a script to grade its own bedside manner.
A synthetic caller doesn't get quietly irritated by a confirmation that arrives half a beat too fast after bad news, because it doesn't feel anything. It can confirm the words were right and still miss that the call was slightly unpleasant, which is the part of the product I'm responsible for.
| Transcript review | Automated tests | Live call testing | |
|---|---|---|---|
| Checks facts and accuracy | Yes | Yes, at scale | Yes, slowly |
| Checks bookings and actions | Partly | Yes | Yes |
| Catches timing and pauses | No | Only measurable ones | Yes |
| Catches tone and repetition fatigue | Rarely | No | Yes |
| Covers interruptions and talking over | No | Partly | Yes |
| Cost per check | Low | Very low once built | High, a person's time |
Automated tests win on scale and consistency, and they catch regressions nobody would notice by ear. Transcript review is quick for checking what was said. Live calls are expensive and slow, which is exactly why we use them for the one thing the others can't measure: how the call felt.
Testing languages I don't speak fluently
The part that surprised me most, once we supported more than English and Hindi, is that I couldn't rely on my own ear.
Rhythm doesn't translate
Pacing that feels natural to me in English doesn't automatically transfer. A pause that sounds thoughtful in one language can sound hesitant in another, and politeness markers that make a Japanese or Korean sentence sound respectful don't map cleanly onto Gujarati.
W3C's internationalization work makes a related point about text and layout: conventions that feel neutral in one language are often specific to it. The same holds for the sound of a spoken conversation.
Native listeners make the call
For the languages Aiva Chat speaks where I'm not a confident native listener, I call alongside a teammate who is, and I watch their face more than I trust my own read of the audio. Deciding whether an Arabic greeting feels right isn't a call a non-Arabic speaker gets to make alone.
What the calls taught us
The rescheduling flow that was too fast
A few months ago I was testing a rescheduling flow, back to back, mostly on autopilot. The assistant answered every question perfectly, and I was still faintly irritated. It took a few more calls to work out why: it never paused before confirming a change.
It moved straight into confirmation at the same clipped pace, whether I'd moved a haircut or cancelled something more inconvenient. Correct, fast, and slightly relentless. We added a very short beat before confirmations that change an existing booking. The words didn't change. To a human ear, it became the difference between a system executing a command and something that briefly considered what you asked.
The design flaws that matter most in a voice product tend to hide on a transcript and become obvious around the fortieth call, and finding them is most of my job.
The call that fooled me
Not every finding comes from something going wrong. One useful call was one where nothing felt off at all, and that was the problem. I'd become so used to our own voice that I couldn't tell if it was good or if I'd gone numb to its flaws, the way you stop smelling your own kitchen.
I now mix in calls after a few days away specifically to catch what fatigue lets slide. Some of those catches involved how the assistant handles being talked over, which is easy to miss when you aren't fully present.
Turn-taking is a design decision
How quickly an assistant decides you've finished speaking shapes the whole call. OpenAI's voice activity detection guide describes the trade-off: shorter silence settings respond faster but risk cutting people off. Only listening tells you where that line should sit.
Common mistakes when testing a voice assistant
Judging from one call
The first call is the most forgiving one you'll ever make. Decisions based on it miss repetition problems entirely.
Reading transcripts instead of listening
Transcripts are useful for accuracy and useless for timing. If you only read, you'll ship calls that are correct and uncomfortable.
Testing only as a polite caller
Real callers interrupt, mumble, and change their minds. Testing only as a patient, clear speaker hides the problems that matter most.
Trusting your own ear in every language
A non-native listener can miss what a native speaker hears in a second. Bring someone who speaks the language your customers speak.
Testing while fatigued
After forty calls you stop noticing. Take breaks and come back with fresh ears, or swap testers.
How to test your own AI receptionist like this
You don't need to be a designer to get value from repeated calls. If you run a business on Aiva Chat, a few hours of this before launch pays off.
- Have the assistant call you first. Use Call it now from the setup checklist, as described in the testing guide.
- Call it ten times in a row as different customers before deciding how it sounds.
- Interrupt it on purpose and check it stops and listens.
- Deliver bad news ("I need to cancel") and listen to how the confirmation feels.
- Ask native speakers to test each language your customers use.
- Save the scenarios that matter as test conversations so every publish checks them.
- Come back a day later and listen again with fresh ears.
- Use our launch checklist before moving your main number over.
Why I still do this
I could hand this work to a testing function, and eventually parts of it should live there. But right now, the person deciding what feels right needs to be the person who sat through the calls where it didn't.
A bad pause written up in a ticket reads as trivial; waiting through it, four calls after the last time it happened, wondering if it's you or the phone, is what convinces you to fix it. It's also a job that fits a small team: nobody gets to be purely a manager of the work.
Conclusion
My honest view after all those calls: if you only have time for one kind of testing before launch, make it repeated listening, not more transcript reading. Transcripts tell you the assistant knows your prices. Only your ear tells you whether a caller will enjoy hearing them.
The caveat is that your ear wears out, and it only works in languages you actually speak. Spread your test calls over a few days, bring in someone who wasn't involved in the setup, and hand any non-English testing to a native speaker rather than trusting your own impression.
If you're about to put an AI receptionist on your own line, start with ten calls as ten different customers, then save the ones that matter as test conversations so every future publish checks them. Our guide on how natural AI voice calls sound covers what to listen for.
Ready to hear the result yourself?
Ring the demo line at +1 (970) 972-0974 and try to trip it up: interrupt it, change your mind halfway through a booking, deliver some bad news. Call back a second time and notice what you hear differently. We do this every week, and you're welcome to join in.
When you want to run the same exercise on an assistant that knows your business, start your 14-day free trial, which needs no card, and use Call it now to have it phone you. Or book a demo and we'll listen to a few test calls with you. The voice agent page explains how calls work.
Posts from the people building Aiva Chat in Rajkot — how the product works, what we have learned building it, and guides for getting the most out of an AI receptionist.