[BPO Insights] The Last Mile Problem in AI Voice: Why Sub-300ms Latency Is Table Stakes

Last reviewed: February 2026

The Quality Metric Nobody Talks About

Everyone in voice AI talks about conversation design. Natural-sounding voices. Resolution accuracy. Knowledge base depth. Multilingual capability.

Nobody talks about the metric that actually determines whether a caller stays on the line or hangs up: latency.

Latency is the time between when a caller finishes speaking and when the AI begins its response. In a human conversation, this gap is approximately 200 milliseconds. It's not something callers consciously measure. It's something they feel. When the gap is right, the conversation flows. When the gap is too long, something feels broken.

Above 300 milliseconds, callers notice. Not consciously -- they don't think "that response took 400 milliseconds." They think "this feels weird." They start to disengage. The conversation quality degrades not because of what the AI says, but because of when it says it.

Above 500 milliseconds, callers react. "Hello? Are you there?" They repeat themselves. They talk over the AI's response when it finally arrives. The interaction becomes adversarial -- the caller fighting the timing instead of engaging with the content. At this point, it doesn't matter how good your conversation design is or how accurate your resolution logic is. The caller has already decided they're talking to a bad system.

Latency is the invisible quality metric. And it's the hardest one to solve.

The Latency Stack

To understand why sub-300ms latency is so difficult, you need to understand the full latency stack in a voice AI interaction. Every voice AI call passes through four sequential processing layers, and each one contributes latency.

Layer 1: Speech-to-Text (STT) -- 50 to 100ms

The caller's voice has to be converted to text before any processing can happen. This involves capturing the audio stream, running it through a speech recognition model, and producing a text transcript.

Modern streaming STT can process audio in near-real-time, delivering partial transcripts as the caller speaks rather than waiting for the full utterance. The best implementations add 50-80ms of latency. Slower implementations that wait for utterance completion before processing can add 150-300ms on their own -- already consuming half or more of the 300ms budget.

The choice of STT provider and configuration is the first latency decision. Streaming architectures are non-negotiable. Batch processing is a latency death sentence for voice.

Layer 2: LLM Inference -- 100 to 200ms

Once the caller's speech is transcribed, the text goes to a language model that determines the appropriate response. This is where the "thinking" happens -- the AI evaluates the caller's intent, checks knowledge bases, applies business logic, and generates a response.

This is the most variable layer. LLM inference time depends on model size, prompt complexity, context window length, and whether the inference happens on cloud GPUs, edge devices, or specialized inference hardware.

Large frontier models with broad capabilities and long context windows can take 300-800ms for inference alone. That's already game over for the 300ms total budget. Small, optimized models fine-tuned for specific use cases can respond in 80-150ms. The tradeoff is capability: the faster models have narrower knowledge and less reasoning flexibility.

The inference layer is where most voice AI companies make their architectural bet. Do you sacrifice capability for speed, or sacrifice speed for capability? The companies that figure out how to deliver both will separate from the pack.

Layer 3: Text-to-Speech (TTS) -- 50 to 100ms

The AI's text response has to be converted to spoken audio. Modern TTS systems produce natural-sounding speech, but naturalness and speed are often in tension. The highest-quality voice synthesis -- the voices that sound indistinguishable from a human -- typically require more computation than functional-but-robotic alternatives.

Streaming TTS helps here, the same way streaming STT helps in Layer 1. Instead of waiting for the full response text before beginning synthesis, streaming TTS starts generating audio from the first word. The caller hears the AI begin speaking while the rest of the response is still being generated.

But streaming TTS requires coordination with the inference layer. If the LLM generates text in bursts rather than smooth token-by-token streaming, the TTS receives text unevenly and the spoken output has unnatural pauses mid-sentence. Smooth audio requires smooth text generation, which requires inference architectures optimized for consistent token production speed.

Layer 4: Network Round-Trip -- 20 to 50ms

The caller's audio has to travel from the phone network to the processing infrastructure and back. This includes telephony network latency, internet transit, and any load-balancing or routing overhead.

For cloud-based architectures, the round-trip depends heavily on the geographic distance between the caller and the nearest processing node. A caller in Dallas connecting to a processing node in Virginia adds meaningful transit time. A caller in Dallas connecting to a processing node in Dallas adds almost none.

This is where edge computing enters the picture. Every major cloud provider offers edge inference locations, and the voice AI companies investing in edge deployment are shaving 10-30ms off this layer. It doesn't sound like much. But when your total budget is 300ms, 30ms is 10% of the budget.

The Arithmetic Problem

Add the layers together:

STT: 50-100ms LLM Inference: 100-200ms TTS: 50-100ms Network: 20-50ms

Total: 220-450ms

The best-case scenario -- streaming STT, optimized small model, streaming TTS, edge deployment -- barely squeezes under 300ms. And that's the best case. Any single layer running at the high end of its range pushes the total above 300ms.

Consistently delivering sub-300ms latency -- not occasionally, not on average, but on every call, for every turn, across every concurrent session -- is an engineering problem that requires optimization at every layer simultaneously.

You can't solve it by optimizing one layer. A team that achieves 50ms STT but uses a 300ms inference model still fails. A team with 80ms inference but batch TTS still fails. Latency is a system-level challenge that requires coordinated optimization across the entire stack.

The Optimization Approaches That Work

The companies making real progress on latency are pursuing four strategies simultaneously.

1. Edge inference.

Move the LLM inference to the network edge -- physically closer to the caller. Instead of routing audio to a central data center, process it at an edge node within 20ms network distance of the caller. This eliminates the largest network latency contributor and can shave 20-40ms from the total.

The tradeoff: edge nodes have less compute capacity than central data centers. You need smaller, more efficient models that can run on edge hardware. This pushes toward specialized models rather than general-purpose ones.

2. Streaming everything.

Every component in the pipeline should operate in streaming mode. STT should produce partial transcripts as the caller speaks. The LLM should begin inference on partial input when possible. TTS should begin synthesis from the first token, not the last. Each component should be emitting output before the previous component has finished its full output.

True streaming architecture means the AI can begin speaking while the caller is still finishing their sentence -- not interrupting, but pre-computing the likely response and beginning synthesis so that the first audio output is ready the instant the caller stops speaking.

3. Pre-computed responses for common patterns.

In most CX conversations, 30-40% of caller utterances fall into predictable categories. "What's my account balance?" "I need to schedule an appointment." "Can I speak to someone?" For these high-frequency patterns, the AI can pre-compute responses and cache them, bypassing the full inference stack entirely.

When the STT detects a pattern match, the system pulls the pre-computed response and routes directly to TTS, skipping inference entirely. This can reduce latency for matched patterns to 100-150ms -- well under the 300ms threshold.

The engineering challenge is matching accuracy. False positive pattern matches -- the system thinks it recognizes a pattern but misidentifies the intent -- produce wrong answers at low latency. That's worse than a correct answer at high latency. The matching threshold has to be calibrated carefully: high enough to capture the latency benefit, conservative enough to avoid misfire.

4. Optimized model architectures.

The general-purpose LLMs that power chatbots and content generation are not designed for real-time voice. They're designed for accuracy and capability. The next generation of voice-specific models will be architecturally optimized for inference speed -- smaller parameter counts, shallower layers, quantized weights, specialized attention mechanisms that trade broad knowledge for fast response in narrow domains.

These models won't match GPT-class capability. They don't need to. A voice AI handling customer service calls doesn't need to write poetry or solve differential equations. It needs to understand customer intent in a specific domain and generate appropriate responses in under 100ms. Domain-specific, speed-optimized models are the clear direction.

Why Latency Is the Moat

Every other capability in voice AI is converging toward parity. Voice quality? TTS technology is advancing rapidly, and the difference between the top five providers is narrowing. Conversation design? The frameworks and best practices are becoming well-understood. Knowledge base integration? Standard API patterns make this increasingly plug-and-play.

Latency doesn't converge. It diverges.

The companies that invest in edge infrastructure, streaming architecture, pre-computation, and optimized models build compounding advantages. Each optimization is hard to implement, hard to replicate, and hard to reverse-engineer. The 300ms company can't copy the 200ms company by switching a vendor. They have to rebuild their architecture.

And latency advantages compound with scale. At 100 concurrent calls, latency management is straightforward. At 10,000 concurrent calls, maintaining sub-300ms latency requires infrastructure that handles load-dependent latency spikes, geographic distribution of compute, and real-time routing decisions that send each call to the processing node that can deliver the fastest response.

The infrastructure investment required to maintain sub-300ms latency at scale is substantial. It's the kind of investment that creates structural barriers to entry. Not patents. Not brand recognition. Infrastructure depth.

The 2028 Prediction

By 2028, sub-200ms latency will be the industry standard for production voice AI. Not the aspirational target. The baseline expectation.

Three forces drive this prediction:

Inference hardware improvements. Custom silicon designed for LLM inference -- not general-purpose GPUs -- will deliver 3-5x inference speed improvements at lower cost. The hardware roadmap from major chip designers makes this near-certain.

Model architecture evolution. Voice-specific model architectures, purpose-built for real-time inference in narrow domains, will deliver sub-50ms inference for common CX interaction patterns. General-purpose models will still exist for complex reasoning tasks, but they won't be in the real-time voice path.

Edge infrastructure maturation. The edge computing buildout currently underway will put inference-capable compute within 10ms network distance of 90%+ of the U.S. population. The network latency layer effectively disappears.

When all three converge, the total latency budget drops from today's 220-450ms to 100-200ms. At 150ms total latency, voice AI conversations will be indistinguishable from human-to-human timing. At that point, latency stops being a quality differentiator and becomes a hygiene factor -- the minimum requirement to compete.

The companies that reach sub-200ms first will set the quality standard that everyone else is measured against. The companies that are still averaging 400-500ms in 2028 will lose on quality regardless of how good their conversation design is, how accurate their resolution rates are, or how natural their voices sound.

The latency race is the real technical moat in voice AI. Everything else is a feature. Latency is the foundation.

Richard Lin is the CEO and founder of Anyreach, an agentic AI platform for enterprise CX.