Problem
Indian SMEs running sales and management operations faced a critical gap: existing voice AI solutions either didn’t support regional languages or introduced latency that frustrated customers. Agents needed to dynamically fetch customer context (previous purchase history, account details, pending issues) mid-call and execute workflows—booking a callback, looking up inventory, transferring to a human agent—without breaking conversation flow.
The challenge wasn’t just inference speed; it was orchestration. A 200ms ASR delay, a 300ms LLM inference, a 150ms synthesis latency—each compounded into a frustrating user experience. We needed sub-second end-to-end performance across 20+ concurrent calls on reasonable hardware.
Data Preparation
We built custom voice datasets for Indian regional languages (Hindi, Telugu, Kannada) by partnering with call centers to collect natural customer interactions. These were annotated for speaker intent, emotion, and turn boundaries. The key insight: turn-detection in Indian languages is different from English—pauses, intonation, and backchannels follow different patterns. We fine-tuned a Silero VAD model on this annotated regional data.
Customer context (name, order history, account balance, pending issues) was normalized from partner CRM APIs into a structured JSON schema, cached in Redis with 5-minute TTL for zero-latency injection into LLM prompts.
Models Evaluated
| Model |
Result |
Note |
| Sarvam Models |
Kept |
Native regional language ASR/TTS, CPU-efficient, <100ms inference |
| Silero VAD |
Kept |
Lightweight voice-activity detection; fine-tuned on regional speech |
| Custom Turn-Detection |
Kept |
Regional pause patterns; trained on partner call data |
| Mistral 7B Instruct |
Kept |
Compact LLM for intent + tool routing; 50–80ms per token |
| Google Cloud TTS |
Cut |
500ms+ latency on concurrency; switched to local synthesis |
| GPT-4 Turbo |
Ruled out |
External API latency + cost incompatible with SMS/voice price point |
Architecture
The pipeline flows through a async orchestrator (Python asyncio):
[Audio Input] -> Sarvam ASR -> Silero VAD (turn-detect)
v
LLM Intent (Mistral) <- [CRM Context from Redis]
v
Tool Router (function-calling) -> CRM Lookup / Booking / Transfer
v
Local Synthesis -> [Audio Output]
Real-time call context is injected via prompt templating: “Customer [NAME] called on [DATE]. Order history: [ORDERS]. Pending issue: [ISSUE]. How do you respond?”
WebSocket servers maintain persistent client connections; a Redis-backed queue routes calls to available inference workers. Call logs (audio transcripts, intent, tool invocation, outcome) are written to PostgreSQL for compliance audits and model monitoring.
Production & Scale
Deployed on AWS EC2 m6i instances (CPU-bound inference workload). Horizontal scaling through an Application Load Balancer; incoming calls distributed via SQS. Inference batching (2–4 sequences per batch) amortizes model load overhead. Connection pooling on Redis and PostgreSQL eliminates per-call overhead.
Monitoring dashboards track latency percentiles (P50, P95, P99), call success rate, and tool invocation accuracy. A/B testing framework allows testing new prompts or models against production without customer impact.
Result: Sub-1s median end-to-end latency, 99.2% uptime, 94% accurate intent classification. Cost per call: ₹0.12 (USD ~0.0015) including infrastructure and AI labor.