Voice AI in Practice — The Complete Guide: Real-Time STT/TTS, Speech LLMs, Turn-Taking, and Deepfake Defense (2025)

Season 4 Ep 9 — In Ep 8, audio was one modality among many. Ep 9 focuses on a single product category: voice. Real-time behavior, naturalness, and safety — why holding all three at once is hard, and how to do it.

Prologue — The Year of "AI Without a Screen"
Chapter 1 · Two Architectures
Chapter 2 · Designing the Real-Time Pipeline
Chapter 3 · Designing the Latency Budget
Chapter 4 · Speech LLMs — GPT-4o Realtime, Gemini Live, Moshi
Chapter 5 · Emotion, Prosody, Speed — Expressive Speech
Chapter 6 · Telephony (PSTN), Browser, Mobile
Chapter 7 · Deepfake and Voice-Cloning Threats, and Defenses
Chapter 8 · What Makes Korean-Language Voice Products Different
Chapter 9 · Cost and Operations
Chapter 10 · Three Real-World Cases
Chapter 11 · Twelve UX Principles
Chapter 12 · Ten Antipatterns
Chapter 13 · Checklist — 12 Items Before Launching Voice AI
Chapter 14 · Next Up — Season 4 Ep 10: "LLM Security"

Prologue — The Year of "AI Without a Screen"

OpenAI's GPT-4o demo in May 2024 was the turning point for voice AI. Latency of ~300ms, natural pronunciation, emotional expression, interruption — a single end-to-end model delivered quality that a traditional STT→LLM→TTS pipeline struggles to reach.

As of 2025:

Contact centers: companies where voice AI handles 50%+ of simple inquiries
Education and coaching: the standard UX for English and foreign-language conversation apps
Health and counseling: therapeutic conversation support (medical advice is still off-limits)
Personal assistants: cars, smart homes, wearables

This post pulls together, in one pass, everything you need to know when you build voice AI.

Chapter 1 · Two Architectures

1.1 The Traditional Pipeline (STT → LLM → TTS)

[Mic] → VAD → [Streaming STT] → [LLM] → [Streaming TTS] → [Speaker]

Each stage is independent → flexible, easy to debug
Free choice of models and vendors
Downside: latency accumulates (500ms–1.5s), emotion and prosody are lost

1.2 Speech LLMs (end-to-end)

[Mic] → [Speech LLM: speech → speech] → [Speaker]

Audio tokens go straight in and out
Reproduces emotion, laughter, even sighs
Latency ~300ms
Downside: large models, a limited set of vendors (OpenAI/Google/Moshi and a few others), hard to control in fine detail

1.3 The Hybrid Reality of 2025

Real-time two-way conversation → speech LLM
Non-real-time and batch processing → traditional pipeline (cost and logging advantages)
Enterprise: compliance and audit demands are strong, so the traditional pipeline still dominates
Consumer: speech LLMs are spreading fast

Chapter 2 · Designing the Real-Time Pipeline

2.1 VAD (Voice Activity Detection)

Purpose: send only the segments where someone is speaking to STT, cutting cost and latency
Models: Silero VAD (open), WebRTC VAD, PyAnnote
Parameters: threshold, minimum utterance length, silence-based end detection

2.2 Streaming STT

Emits partial transcripts every 250–500ms
The final transcript arrives when end-of-utterance is detected
Deepgram, AssemblyAI, Whisper (streaming), Google/Azure/AWS, Clova/Kakao

2.3 LLM Processing

You can "start thinking" from the partial transcript (build context in advance)
Generate the response once the final transcript is confirmed
Speculative generation: start the answer from the partial, then correct it once the final arrives

2.4 Streaming TTS

Play back sequentially at sentence boundaries
Natural gaps without "thinking" fillers (uh/hmm)
The core of latency reduction

2.5 Interruption (Barge-in)

The moment the user starts speaking, stop TTS immediately
A response that was only partially played should be folded into the next turn as a "summary/correction"
Making the cut-off feel human rather than robotic is what UX lives or dies on

Chapter 3 · Designing the Latency Budget

3.1 The Target

A figure frequently cited in research and in the field: the average gap between turns in human conversation is ~200ms. If the AI answers within 500ms–1s it feels natural. Past 1.5s it feels awkward.

3.2 An Example Budget (Traditional Pipeline)

VAD end-of-utterance detection: 150ms
STT final result: 150ms
LLM TTFT (first token): 300ms
TTS first audio chunk: 150ms
Network and decoding: 100ms
───────────────────
Total to first audio playback: about 850ms

3.3 Where to Cut

Use a small/fast LLM as the first responder (fillers, acknowledgements)
Streaming: for STT, LLM, and TTS alike
Speech LLMs: the architecture itself is shorter
Locality: deploy in a region close to the user

3.4 Jitter and Stability

p95/p99 matter more than the average
One long turn wrecks the rhythm of every turn that follows → users drop off

Chapter 4 · Speech LLMs — GPT-4o Realtime, Gemini Live, Moshi

4.1 GPT-4o Realtime

Real-time and bidirectional across speech and text
Built on WebSocket/WebRTC
Latency 300–500ms
Considerably more expensive than text, but latency and quality are overwhelming

4.2 Gemini Live

1M context plus real-time multimodal
Can combine screen sharing and camera input
Excellent at interruption and emotional expression

4.3 Moshi (Kyutai, open)

An open-source speech LLM
Full-duplex, low latency
Can run locally — attractive for privacy-oriented products

4.4 Differences

Item	GPT-4o	Gemini Live	Moshi
Price	High	Medium	Self-hosted
Korean	Excellent	Excellent	Limited
Customization	Limited	Limited	High (open)
Modality mix	Speech, text	Speech, video, text	Speech-first
Enterprise	API + logging	Google Cloud	Self-managed

Chapter 5 · Emotion, Prosody, Speed — Expressive Speech

5.1 The Control Axes of TTS

Tone/voice: persona, gender, age
Emotion: neutral, happy, sad, angry, excited, and so on
Speed: 0.8x–1.5x
Stress: emphasis on particular words
Breath and pauses: SSML such as <break time="300ms"/>

5.2 SSML (Speech Synthesis Markup Language)

<speak>
  Hello, <break time="300ms"/>
  today we are going to talk about something <emphasis level="strong">very important</emphasis>.
</speak>

Supported by most vendors, including Google, Microsoft, Naver Clova, and Kakao
ElevenLabs and OpenAI TTS use their own formats (curly-brace tags, natural-language directions)

5.3 The Politics of Emotion

Corporate brand tone: keep excessive emotion out
Services for children: warm tone, slower speed
Medical and counseling: neutral and calm

Do not use the same tone for every voice. Split the profiles to match the service context.

Chapter 6 · Telephony (PSTN), Browser, Mobile

6.1 Telephony Integration

Twilio, Vonage, Plivo, SignalWire: programmable telephony
A SIP/PSTN gateway hands the voice stream to STT/LLM
Korea: local solutions such as LG U+ AI Call, KT AICC, and Naver Cloud AiCall

6.2 The Browser

Two-way audio over WebRTC
Permissions, echo cancellation, and noise suppression are built in
Checking mobile browser compatibility is mandatory

6.3 Mobile Apps

iOS: AVAudioEngine, Speech, AVSpeechSynthesizer
Android: AudioRecord, MediaCodec, TTS API
Native VAD is available

6.4 Automotive and Embedded

Compensating for cabin and wind noise
Safety while driving: short emergency responses
Offline mode is mandatory (connectivity is unstable)

Chapter 7 · Deepfake and Voice-Cloning Threats, and Defenses

7.1 The Threats

Voice phishing: cloning a family member's voice to demand an urgent transfer
Executive impersonation: transfer instructions in the CEO's voice
Identity spoofing: bypassing identity verification at call centers and financial institutions

7.2 Defense Layers

Liveness detection: prepared lines vs. demanding a live response
Voice biometrics: voiceprint plus behavior, in combination
Deepfake detection: synthetic-speech classifiers (accuracy in the 90% range, not perfect)
Callback verification: for sensitive requests, call back on the registered number
Mandatory AI disclosure: laws in some countries and states (notice when the voice is AI)

7.3 Watermarking

Embed an imperceptible watermark in AI-generated speech (for example, Google SynthID for audio)
A detector identifies AI provenance by checking the watermark
Not yet 100% reliable — treat it as one defense layer among several

7.4 Operational Policy

Sensitive transactions are "no AI voice" → hand off to a human
Retain and audit voice logs (within what the law allows)
AI notice when a new call connects: "This call is being handled by an AI agent."

Chapter 8 · What Makes Korean-Language Voice Products Different

8.1 STT

Accuracy on standard Korean is 95%+ with Clova/Kakao
Coverage of regional dialects is still lacking
Custom dictionaries are mandatory for specialist terminology (medicine, law)

8.2 TTS

Naver CLOVA Voice, Kakao i, Supertone: top tier for naturalness
ElevenLabs Korean voices are improving rapidly as well
Controlling Korean-specific intonation and honorific speech matters

8.3 Law and Regulation

Telecommunications Business Act and E-Commerce Act: call-recording notice is mandatory
Personal Information Protection Act: voice data can also contain sensitive information
Finance: AI voice alone cannot authenticate a customer for KYC

8.4 Culture and UX

Consistency between casual and honorific speech
Brand policy on forms of address (-ssi / -nim / gogaek-nim)
Opening a response with something like "Yes, Mr./Ms. So-and-so" raises perceived trust

Chapter 9 · Cost and Operations

9.1 Cost Structure

STT: $0.003–0.02 per minute
TTS: $0.00001–0.00005 per character (depending on voice and quality)
LLM: per token (input/output) — same as text
Speech LLMs: tend to be 2–5x more expensive than a traditional pipeline (early 2025)
Lines (PSTN): an additional per-minute cost

9.2 Scaling

N concurrent calls → a TCP/WebSocket per stream plus an LLM slot
Orchestration (Kubernetes + HPA) plus regional distribution
A warm pool to reduce cold starts

9.3 Observability

Per-call latency (p50/p95), interruption frequency, rate of transfer to a human
Real-time monitors for hallucination and inappropriate statements
Tracking NPS and resolution rate

Chapter 10 · Three Real-World Cases

10.1 First-Line Contact Center Response

Use cases: reservations, address changes, simple FAQs
Complex inquiries get a smooth handoff to a human agent (with a context summary passed along)
Automatic checks on recording and compliance phrasing

10.2 An English Conversation Learning App

Natural conversation via a speech LLM (GPT-4o realtime)
Pronunciation scoring (traditional STT is more accurate here)
Difficulty tuned to the learner's level

10.3 Senior Care and Wellbeing

Companionship, simple reminders, medication prompts
Warm emotional tone, slower speed
Contact a guardian when an emergency (a fall, unusual symptoms) is detected

Chapter 11 · Twelve UX Principles

Latency is everything: past one second it feels awkward
Interruption: stop the instant the user cuts in
Keep it short: three to five sentences per turn
Signal uncertainty: when you don't know, say "I'm not sure about that"
Anti-spoofing: never allow sensitive actions by voice alone
Restrained emotion: brand tone first, no overacting
Fillers: short signals like "mm" and "right" convey "I'm listening"
Repeat requests: when the user says "what?", adjust speed and pronunciation
Closing signal: wrap up naturally with "Is there anything else I can help with?"
Error recovery: if STT looks wrong, ask back with "Did I understand that correctly?"
Block personal data: discourage speaking card numbers or national ID numbers
Accessibility: design separate UX for deaf and elderly users

Chapter 12 · Ten Antipatterns

12.1 Shipping Without Measuring Latency

90% of perceived quality is latency. p95 is the yardstick.

12.2 Overly Long Prompts for the LLM

Real-time behavior breaks. Keep the system prompt short and sharp.

12.3 No Interruption Handling

A robot that keeps pushing while you're talking. It destroys the UX.

12.4 No Consistency in Emotion and Tone

Mixing three voices and confusing the brand.

12.5 Trusting Deepfake Detection Alone

Detectors are imperfect. Combine liveness, callback, and policy.

12.6 Missing Legal Notices

Call-recording and AI-response notices are mandatory.

12.7 Implementing Real Time with Batch TTS

Needs a rewrite to chunk-level streaming.

12.8 Sensitive Transactions by Voice Alone

Identity verification in finance and healthcare needs an additional channel.

12.9 Ignoring Dialects and Elderly Voices

STT accuracy varies widely. Diversify samples and customize.

12.10 No Logs

You lose the evidence for disputes and for quality improvement.

Chapter 13 · Checklist — 12 Items Before Launching Voice AI

Chapter 14 · Next Up — Season 4 Ep 10: "LLM Security"

For voice and for text alike, the biggest threat to a 2025 LLM product is a security incident.

Twelve variants of prompt injection
Jailbreaks and roleplay bypasses
Data exfiltration vectors
Model extraction and parameter leakage
Denial of service (DoS) and billing attacks
Building and automating red teams (PyRIT, Garak)
Guardrail prompts vs. classifier models
Privacy and PII leakage
Supply-chain threats (models, MCP servers)
Standards and regulation (EU AI Act, Korean privacy law)
The incident response playbook

"Security is not a feature. It is a default." And yet through 2024–2025 many LLM products still fail to meet even the defaults.

See you in the next post.

Summary: Voice AI is a three-beat rhythm of real-time behavior, naturalness, and safety. The choice between a traditional pipeline and a speech LLM comes down to your latency, control, and compliance demands, and p95 latency decides 90% of the user experience. Manage emotion, speed, and tone through brand policy, and treat deepfakes and spoofing as something only layered defense handles. Korean-language products push the quality frontier by combining local assets like Clova, Kakao, and Supertone with global speech LLMs. In the era of "AI without a screen," the limit is now the imagination of the person building it.