TL;DR: There are two ways to build a voice agent. A chained pipeline turns speech into text, runs a language model on the text and turns the reply back into speech. Every stage produces text you can inspect, log and replace, but the delays add up. A speech-to-speech model hears audio and answers in audio, which cuts latency and keeps tone, but gives you fewer places to intervene. Choose by what the call must prove and control, then spend most of your design effort on turn-taking: knowing when the caller has finished, and stopping instantly when they interrupt.
Two architectures, one latency budget
What a caller feels is the silence between the end of their sentence and the first sound of the reply. Both architectures spend part of that silence deciding that the caller has actually finished.
Read diagram description
Illustrative time to first audio after a caller stops speaking. A chained pipeline spends 300 milliseconds detecting the end of the turn, 200 on final transcription, 450 for the language model's first token and 200 for speech synthesis to start, about 1.15 seconds. A speech-to-speech model spends the same 300 milliseconds on end of turn and about 500 to its first audio, about 0.8 seconds.
The numbers are an illustrative budget, not a benchmark; measure your own stack at p95 on real phone lines. Research systems have pushed speech-to-speech further: the Moshi paper reported about 200 milliseconds in practice against "several seconds" for typical pipelines (author-reported).
What each architecture buys
| Chained pipeline | Speech-to-speech | |
|---|---|---|
| Latency | Stages add up; streaming each stage claws some back | Lower, one model in the loop |
| Tone and emotion | Lost when speech becomes text | Heard and expressed |
| Inspection | Text at every stage, easy to log, filter and audit | A transcript is produced alongside, but the model reasons on audio |
| Swapping parts | Pick the speech recogniser for your languages and accents, any LLM, any voice | One vendor's model for all three jobs |
| Tools and policies | Mature tool calling on text models | Supported, younger |
| Failure isolation | Know which stage failed | Harder to localise |
OpenAI's voice-agent guide puts the choice in one line: use the chained path "when each stage needs to be visible or replaceable". That describes regulated calls, such as banking or insurance, where a compliance team needs the text the decision was made on, and multilingual markets where the best recogniser for Hindi, Tamil or code-switched Hinglish may come from a different vendor than the best reasoning model. Speech-to-speech fits conversations where warmth and pace are the product: coaching, companionship, language practice, concierge calls.
Turn-taking is the product
A voice agent can give every right answer and still fail the call on turn-taking.
End of turn. The system must guess when the caller has finished. Wait too long and the line goes dead; cut in too soon and the agent talks over someone who paused to think. Better detectors use the words and the intonation, not only silence.
Barge-in. When the caller speaks over the agent, speech output must stop within a fraction of a second and the agent must remember what it had already said, so it does not repeat itself or lose its place.
Turn eagerness. Platforms expose this as a setting. ElevenLabs documents "turn eagerness" controls for how quickly the assistant responds and separate interruption handling. Treat these as product parameters to test per use case: a booking line wants patience, a quick status check wants speed.
Economics as of November 2026
Voice is priced per audio token or per minute, which makes call length the cost driver. Two published rates, checked on 20 November 2026:
| Model | Audio input | Audio output |
|---|---|---|
| OpenAI gpt-realtime-2.1 | $32 per million tokens | $64 per million tokens |
| Google gemini-3.8-live | $3 per million tokens, or $0.005 a minute | $12 per million tokens, or $0.018 a minute |
An illustrative five-minute call in which the caller speaks for three minutes and the agent for two would cost about 3 × $0.005 + 2 × $0.018 = $0.051 on the Gemini per-minute rates. Check how your provider bills the conversation history re-sent on each turn and any text tokens for instructions and tools; long calls grow faster than linearly when history is re-processed. Compare the total with the cost of a human-handled call, then price containment honestly: a call the agent abandons and a human then handles costs both.
What to measure
- Time to first audio at p95, per architecture and per network condition.
- Talk-over rate: share of turns where the agent spoke while the caller was speaking.
- Barge-in recovery: after an interruption, did the agent stop, listen and continue correctly?
- Task completion and transfer rate to humans, with the reason for each transfer.
- Repeat-contact rate within seven days, which catches calls that ended politely but solved nothing.
In the interview
Voice design prompts ("design a voice agent for our support line", "our voice agent keeps interrupting people") test whether you know where the time goes and what each architecture gives up. A weak answer picks the newest model. A Hire answer budgets latency. A Strong Hire answer chooses chained or speech-to-speech by audit and language needs, treats end-of-turn detection and barge-in as the core design, prices a call per minute including abandoned calls, and names the metrics that reveal talk-over and silent failure.
Next: latency budgets and human handoffs.