Problem
The restaurant industry faced a transformation: customers expected effortless ordering, but traditional phone systems required staff to manually take orders, read them back, and enter into POS. Voice AI promised automation—but integrating with existing restaurant infrastructure (Square POS, payment processing, legacy phone systems) introduced real-world constraints:
- Latency: ASR must complete within 500ms or customers perceive delay
- Accuracy: Misheard menu items or modifications lead to wrong orders (costly)
- Compliance: Payment information requires PCI-DSS handling; customer conversations may contain health data (allergies)
- Fallback: When AI is uncertain, seamlessly transfer to a human agent
Data Preparation
We partnered with restaurant chains to collect 3,000 anonymized call recordings (HIPAA-compliant storage). Each call was annotated with:
- Intent: menu inquiry, place order, modify order, payment, transfer
- Entities: menu items, quantities, dietary restrictions (vegetarian, gluten-free, nut-free, etc.)
- Context: customer ID (if returning), order state
We extracted menu data from Square POS APIs: dish names, prices, allergen info, availability schedules. This data was encoded into few-shot examples for the NLU model: “Customer: ‘I’d like a large pepperoni, extra cheese, no onions.’ -> Items: [Pizza, size: large, toppings: pepperoni+cheese, exclusions: onions]”
The evaluation set included edge cases: accented speech, background noise, interruptions, unclear audio. Ground truth: human-validated orders.
Models Evaluated
| Model |
Result |
Note |
| Whisper (large-v3) |
Kept |
Robust multilingual ASR; 95%+ word accuracy; handles noise/accents |
| Mistral 7B (Intent) |
Kept |
Fast intent classification; few-shot capable |
| Mistral 7B (NER) |
Kept |
Named-entity recognition for menu items; 94% F1-score |
| LiveKit (WebRTC) |
Kept |
Real-time streaming; <100ms latency; SIP integration |
| Square Payments |
Kept |
PCI-DSS compliant; direct POS sync |
| Twilio SIP |
Cut |
Higher latency; LiveKit delivered better p2p performance |
| Fine-tuned ASR |
Cut |
Whisper large-v3 required no fine-tuning (high baseline) |
Architecture
The voice-ordering pipeline:
[Customer speaks] -> LiveKit WebRTC stream
v
Whisper ASR (transcribe)
v
Intent Classifier (Mistral): order / inquiry / modify / transfer?
v
If "order": NER (extract items, quantities, mods)
v
Order State Machine
├─ Validate items exist in menu
├─ Check availability (query POS API)
├─ Confirm dietary restrictions
v
Payment or Transfer
├─ If payment: -> Square API (encrypted)
├─ If transfer: -> SIP trunk to human staff
v
TTS Response -> [Customer hears confirmation]
A finite-state machine manages conversation flow: each user input transitions the order state (empty -> items -> mods -> payment -> confirmed). If the NLU confidence is low, the system escalates to a human.
Database storage is encrypted at the field level (credit card data never stored; only payment token from Square).
Production & Scale
Deployed across 3 restaurant chains (150+ locations). Average call duration: 2m 15s. Order success rate: 89% (rest escalated to human staff).
Key metrics:
- ASR accuracy: 3.2% WER (word error rate)
- Order accuracy: 96% (correct items, mods, quantities)
- Successful orders: 89% (11% require human escalation, mostly edge cases or ambiguous requests)
Reliability: 99.7% uptime; <100ms median ASR latency. Payment processing: zero PCI-DSS violations.
A/B test: Comparing Whisper + Mistral vs a custom fine-tuned ASR model showed no significant improvement from fine-tuning—Whisper large-v3’s pre-trained performance was sufficient.
Learnings:
- Real-world restaurant conversations are messier than training data (overlapping speech, background kitchen noise)
- Dietary restrictions (allergies) require human double-confirmation; automating this fully is risky
- Escalation to human staff is not a failure mode; it’s a feature for complex or ambiguous orders
Result: 150+ restaurants, ~50,000 voice orders processed, zero food-safety incidents attributed to AI misunderstanding.