---
title: "Why we built our own voice AI stack for Indian callers"
description: "Speech-to-speech models work well for American callers, but not yet for Indian ones. So we built our own cascade of speech-to-text, a frontier LLM and text-to-speech, and tuned what makes a call feel human."
canonical: https://mytribe.in/blog/how-we-made-our-ai-sound-like-a-person
authors: Lavish Thakkar and Ugam Kamat
published: 2026-11-03
last_updated: 2026-11-03
---

# Why we built our own voice AI stack for Indian callers

By Lavish Thakkar and Ugam Kamat · 3 November 2026 · 8 min read

**Summary.**

Speech-to-speech models work well for American callers, but not yet for Indian ones. They sound foreign here, struggle with Hinglish on an 8 kHz phone line, and can't reason like the frontier text models our calls need.

So we run a cascade: speech-to-text (STT), then a large language model (LLM), then text-to-speech (TTS). We built our own in a week instead of renting one, so we can tune what makes a call feel human: endpointing, barge-in, accent and time to first audio.

## Speech-to-speech isn't ready for our calls

The cleanest design for voice AI is a single speech-to-speech model that hears audio and speaks audio, with no text in between. There's less to run and less latency to fight. For American English customer support it's a reasonable choice today.

It doesn't work for us yet, for two reasons. The first is the language. Our callers switch between Hindi and English mid-sentence, over a phone line that carries a fraction of the audio a laptop microphone does, and the speech-to-speech models available today sound foreign to them and mishear them more.

The second reason matters more. Our calls need a lot of reasoning. A customer planning a first birthday or a Ganpati stage brings a loose, subjective brief, and in the middle of the call the agent has to weigh it against a shortlist of providers, their past work and their price lists, call tools, and ask the next useful question. A support call follows a narrow script by comparison. Speech-to-speech models today are a tier below the frontier text models on this kind of reasoning, and we would rather not give that up.

**Two ways to build a voice agent**

- Speech-to-speech: audio in, one model, audio out.
- Cascade: audio in, STT, LLM, TTS, audio out.

Speech-to-speech is a single model from audio in to audio out. A cascade is three models in a row, and every join between them is code you can tune.

So we use a cascade of three models: STT, a frontier LLM, and TTS. Everything that makes a cascade feel natural or robotic happens in the joins between those models.

## Why we built the cascade ourselves

Everyone is building voice AI right now. In our office building in Bellandur, the two teams at the next desks are both building voice bots, one for customer support and one for fraud checks before collecting money. Most teams buy a packaged voice platform and plug it in.

We tried two or three platforms first. One slipped into Punjabi in the middle of a call. One wouldn't let us change its endpointing, the logic that decides when the caller has finished speaking, which is half of what makes a call feel natural. The third gave us a kit to tune every setting ourselves, and at that point it made more sense to write our own. With AI writing most of the routine code, our first version was live in one week.

The bigger reason is flexibility. Every part of a call that makes it feel human is a setting we can change, and every model is a part we can swap. It also matters for where we're going. Customers reach us on WhatsApp and on calls, and we want the two to work as one conversation, with a call able to pick up from a chat and send things back into it. A packaged platform closes that off. We needed a system where every part is ours to open up.

## The stack

**One turn of a call, step by step**

1. **Phone line.** The call comes in over Plivo as 8 kHz audio.
2. **Voice isolation.** Krisp removes background voices, so only the caller is heard.
3. **STT.** Sarvam's saaras:v4 streams the transcript, and its server-side voice activity detection decides when the caller has finished.
4. **LLM.** GPT 6.1 Terra on Amazon Bedrock works out the reply from our data on providers and prices, and calls tools when it needs to.
5. **TTS.** Cartesia's sonic-3.6 speaks the reply.
6. **Our mixer.** It picks the accent for each sentence, adds a faint office background, and sends the audio back down the line.

The dashed boxes are services we rent. How they are joined, and everything between them, is our code.

All of this runs on Pipecat, the open-source voice framework, which we deploy in Mumbai. The agent's prompt and tools live in our agent platform, agent0, outside the voice code, so a prompt change goes live on calls in about a minute without a deploy. Every call is recorded in stereo, caller on one channel and AI on the other, so when something goes wrong we can tell whether the audio was ever there.

**What we learned.** In a cascade, timing gives the bot away more than the voice does. Most of what we've tuned is timing: who speaks first, how fast the first word arrives, how long we wait before deciding the caller has finished, and who is allowed to interrupt.

## Tuning the timing

**Let the caller speak first.** On a phone, the person who picks up talks first. The platforms we tried got this wrong: the bot opened the moment the call connected. On 37% of the calls we checked, the caller was already speaking before the AI had made a sound, sometimes a fifth of a second after connecting. The AI now waits and only opens if the caller stays silent.

**Cut the time to first audio.** The opening line used to be generated by the LLM, which took 2.6 seconds. Now it's sent straight to TTS and plays in about 0.2. When we moved to Cartesia, first audio after a reply dropped to about 0.06 seconds, against anywhere from 0.49 to 5.33 seconds before.

**Shorten endpointing, made safe by voice isolation.** Indian homes are loud, with the TV on, a pressure cooker going or someone else talking in the room. Without help, any of that can count as the caller interrupting, so a bot has to wait longer before it replies. Because Krisp strips out background voices before anything else hears the call, only the caller can barge in, and that let us cut the silence window from the default 500 milliseconds to 300.

**What the timing work changed**

| | Before | Now |
| --- | --- | --- |
| Opening line, time to first audio | 2.6 s | 0.2 s |
| First audio after a reply | 0.49 to 5.33 s | 0.06 s |
| Silence before deciding the caller has finished | 0.5 s | 0.3 s |

37% of the calls we checked had the caller already speaking before the AI made a sound.

## Sounding Indian

Ready-made voices can switch languages, but their English has an American accent. We pick the TTS locale for every sentence from what the AI is about to say, so English comes out the way people here speak it and Hindi written in Latin script is read as Hindi. Prices, dates and phone numbers are normalised the Indian way whatever the language, and a pronunciation dictionary fixes names and words the model gets wrong, starting with "mytribe".

A perfectly clean voice over total silence also sounds fake, so we mix a faint office ambience under the AI's voice.

The voice itself matters as much. The stock voices that labs ship never sound quite as natural as a voice cloned from a real person's recording. We learned this on our Instagram reels, where we record a real voice sample, clone it, and use the clone. Those reels have done well for us, and we're about to do the same on calls. It's one of the reasons we use Cartesia.

Here is what that sounds like on a real call. A customer in Bengaluru planning a proposal at home had just described what they wanted, and the AI plays the brief back:

**Customer:** "...floating candles through strings, as well as Polaroid photos, because my girlfriend is a Harry Potter fan."

**mytribe AI:** "I have got the picture. A large red rose petal heart with a white border in front of the curtains, LED candles around it, red balloons, and Polaroids with a subtle Harry Potter touch."

[Audio: Proposal at home](https://assets.mytribe.in/insecure/raw:1/plain/s3://mytribe-assets-production/site/blog/how-we-made-our-ai-sound-like-a-person/harry-potter-reply-e4c0bb6917.m4a)

Listen for the Indian English accent and the office sound under the AI's voice.

## Where the parts fell short, and how we fixed them

Every part of the cascade has limits on Indian phone calls. We found most of them by listening to calls that went badly. We keep a log of every change, and the first line of it warns that most of the tempting changes have already been tried.

### 1. The AI didn't hear an answer for 15 seconds

- **Example.** A caller said "chaar ghante" (four hours). The AI only heard it 15 seconds later, when the caller repeated it louder.
- **Cause.** Our first TTS setup kept reporting the bot as speaking for about three seconds after its audio had ended, so the caller's answer was treated as an interruption and dropped. The audio sounded perfect, so it took weeks to find.
- **Fix.** We moved TTS to Cartesia on 31 August. The bug went, and first audio dropped to about 0.06 seconds.

### 2. The AI answered a number the caller had already corrected

- **Example.** A caller said "20", then "Haan, 25 honge" (yes, it'll be 25), and the reply to "20" went out first.
- **Cause.** We had added a language-model check on whether the caller had finished their thought. Over a full call it never once said no, and it added latency.
- **Fix.** We removed it the same day. Endpointing stays with the STT model's voice activity detection.

### 3. "South Delhi" became "Saudi Arabia"

- **Example.** STT heard "South Delhi" as "Saudi Arabia". After the caller corrected it, the AI decided they were relocating there.
- **Cause.** STT is measurably worse on 8 kHz phone audio, and place names suffer most.
- **Fix.** Still open at the speech-to-text step. It has got better since we moved to a more capable LLM, which now sees past mistakes like this in the transcript.

### 4. A child's name was misheard twice

- **Example.** A parent had to correct their child's name after the AI misheard it twice.
- **Cause.** Indian names are hard for STT, more so on phone audio.
- **Fix.** The AI reads names back before writing them down, which is how this one got caught on the call.

## Swapping models

Because we own the cascade, each model is a setting. We replaced the TTS model on 31 August and moved the LLM from gpt-6-luna on OpenAI's API to GPT 6.1 Terra on Bedrock on 30 October, and neither change needed a rebuild of anything else. We moved because Terra is a bigger and more capable model, and in our tests it was also faster, starting its reply in about 1 second against 3.1 seconds for gpt-6-luna.

When a speech-to-speech model can handle Hinglish over a phone line and reason as well as a frontier text model, we'll test it against the cascade and switch if it wins.

## Results and limits

When customers message mytribe, the AI offers a call or more chat, and about half of them choose the call. Another 8% ring our number themselves without being asked. Once they're on a call, the AI gets much more out of them:

**Call or chat, side by side**

| | Call | Chat |
| --- | --- | --- |
| Customers who give us their complete requirement | 65% | 36% |
| Time it takes to get those details | under 3 min | 36 min |

Call time is the median call. Chat time is measured from the customer's first message.

Customers talk to it in whatever language they like: 82% of calls are in English, 13% in Hinglish and the rest in Hindi and Kannada. What this means for the business, and for the providers we work with, is in our other article, [Voice AI is a major unlock for the expert services industry](https://mytribe.in/blog/voice-ai-for-expert-services).

The limits are real. STT still mishears names and places on phone audio, and the AI still gets some calls wrong. We find those calls in the recordings and fix them one at a time.

## What's next

The same code already runs a second line, for providers: since late October an expert can ring us about a booking or their profile and talk to an agent that knows their business, with its own voice and its own prompt. Next is voice and chat on the same call. Halfway through a call, the voice agent asks our WhatsApp agent to send the customer a few photos and carries on: "Got them? Tell me which one you like." Because we wrote both agents, that's one request from one to the other.

The other job is making the voice more real. Voice bots are everywhere now, a lot of people have had bad experiences with them, and some distrust has set in. A small share of our customers don't want to talk to any AI at all. More natural voices are how we win them over. Most people have no problem with AI as long as it gets the job done, and we think people will end up preferring a well-built voice AI to a human, because it gets the job done faster.

*Figures come from our own measurements on live calls between August and October 2026, and customer lines are anonymised.*