← AI Architecture Patterns
Foundational

Realtime Voice Agents: Chained Pipelines, Speech-to-Speech Models and Turn-Taking

A voice agent either chains speech recognition, a language model and speech synthesis, or uses one model that hears and speaks. The chain is inspectable and swappable; the single model is faster and keeps tone. Learn the latency budget, the turn-taking controls, the per-minute economics as of November 2026 and the metrics that tell you a call went well.

CONCEPT / AI ARCHITECTURE PATTERNS

TL;DR: There are two ways to build a voice agent. A chained pipeline turns speech into text, runs a language model on the text and turns the reply back into speech. Every stage produces text you can inspect, log and replace, but the delays add up. A speech-to-speech model hears audio and answers in audio, which cuts latency and keeps tone, but gives you fewer places to intervene. Choose by what the call must prove and control, then spend most of your design effort on turn-taking: knowing when the caller has finished, and stopping instantly when they interrupt.

Two architectures, one latency budget

What a caller feels is the silence between the end of their sentence and the first sound of the reply. Both architectures spend part of that silence deciding that the caller has actually finished.

DIAGRAM
Silence before the agent speaks (illustrative) Chained pipeline turn end STT LLM first token TTS 1.15 s Speech-to-speech turn end model first audio 0.8 s 0 s 0.5 s 1.0 s The turn-end wait is in both rows. Shortening it can be the biggest win, and the riskiest: end the turn too early and the agent talks over a caller who only paused.
Read diagram description

Illustrative time to first audio after a caller stops speaking. A chained pipeline spends 300 milliseconds detecting the end of the turn, 200 on final transcription, 450 for the language model's first token and 200 for speech synthesis to start, about 1.15 seconds. A speech-to-speech model spends the same 300 milliseconds on end of turn and about 500 to its first audio, about 0.8 seconds.

The numbers are an illustrative budget, not a benchmark; measure your own stack at p95 on real phone lines. Research systems have pushed speech-to-speech further: the Moshi paper reported about 200 milliseconds in practice against "several seconds" for typical pipelines (author-reported).

What each architecture buys

Chained pipelineSpeech-to-speech
LatencyStages add up; streaming each stage claws some backLower, one model in the loop
Tone and emotionLost when speech becomes textHeard and expressed
InspectionText at every stage, easy to log, filter and auditA transcript is produced alongside, but the model reasons on audio
Swapping partsPick the speech recogniser for your languages and accents, any LLM, any voiceOne vendor's model for all three jobs
Tools and policiesMature tool calling on text modelsSupported, younger
Failure isolationKnow which stage failedHarder to localise

OpenAI's voice-agent guide puts the choice in one line: use the chained path "when each stage needs to be visible or replaceable". That describes regulated calls, such as banking or insurance, where a compliance team needs the text the decision was made on, and multilingual markets where the best recogniser for Hindi, Tamil or code-switched Hinglish may come from a different vendor than the best reasoning model. Speech-to-speech fits conversations where warmth and pace are the product: coaching, companionship, language practice, concierge calls.

Turn-taking is the product

A voice agent can give every right answer and still fail the call on turn-taking.

End of turn. The system must guess when the caller has finished. Wait too long and the line goes dead; cut in too soon and the agent talks over someone who paused to think. Better detectors use the words and the intonation, not only silence.

Barge-in. When the caller speaks over the agent, speech output must stop within a fraction of a second and the agent must remember what it had already said, so it does not repeat itself or lose its place.

Turn eagerness. Platforms expose this as a setting. ElevenLabs documents "turn eagerness" controls for how quickly the assistant responds and separate interruption handling. Treat these as product parameters to test per use case: a booking line wants patience, a quick status check wants speed.

Economics as of November 2026

Voice is priced per audio token or per minute, which makes call length the cost driver. Two published rates, checked on 20 November 2026:

ModelAudio inputAudio output
OpenAI gpt-realtime-2.1$32 per million tokens$64 per million tokens
Google gemini-3.8-live$3 per million tokens, or $0.005 a minute$12 per million tokens, or $0.018 a minute

An illustrative five-minute call in which the caller speaks for three minutes and the agent for two would cost about 3 × $0.005 + 2 × $0.018 = $0.051 on the Gemini per-minute rates. Check how your provider bills the conversation history re-sent on each turn and any text tokens for instructions and tools; long calls grow faster than linearly when history is re-processed. Compare the total with the cost of a human-handled call, then price containment honestly: a call the agent abandons and a human then handles costs both.

What to measure

  • Time to first audio at p95, per architecture and per network condition.
  • Talk-over rate: share of turns where the agent spoke while the caller was speaking.
  • Barge-in recovery: after an interruption, did the agent stop, listen and continue correctly?
  • Task completion and transfer rate to humans, with the reason for each transfer.
  • Repeat-contact rate within seven days, which catches calls that ended politely but solved nothing.

In the interview

Voice design prompts ("design a voice agent for our support line", "our voice agent keeps interrupting people") test whether you know where the time goes and what each architecture gives up. A weak answer picks the newest model. A Hire answer budgets latency. A Strong Hire answer chooses chained or speech-to-speech by audit and language needs, treats end-of-turn detection and barge-in as the core design, prices a call per minute including abandoned calls, and names the metrics that reveal talk-over and silent failure.

Next: latency budgets and human handoffs.

RELATED CONCEPTS
PRACTISE THIS CONCEPT