Modal 推出 Sticky Sessions：会话感知路由降低语音 agent 延迟

kwindla @kwindla原推发布于 2026.10.22

在 X 上打开原推

Our friends at @modal shipped a new Sticky Sessions feature yesterday that's extremely useful for building very low-latency voice agents and other realtime AI applications. Today, you can build (and serve at scale) voice agents that deliver 500ms voice-to-voice response times. But this is 3-4x faster than most of the voice agents we see in production. To be fast, you need a few things: fast models; good orchestration of turn detection and data streaming between models in your agent pipeline; fast WebRTC or telephony client-server network routing; and optimized placement/hosting/routing for your agent and inference containers. Sticky Sessions makes it easy to route a specific user's requests to the same container every time. If you're an infrastructure person, you might describe this as session-aware serverless routing. The Modal team did a lot of work here to get the details right, so things like load balancing and auto-scaling work nicely in conjunction with Sticky Sessions. The video is a demo clip showing a voice agent running on Modal with a 500ms voice-to-voice latency. Ben Shababo, who built this demo, used: - Modal's Qwen 3 ASR service for transcription - Modal's PhoneLLM Mini service as the agent brain - The brand new @GradiumAI Phonon text-to-speech model for the voice - Pipecat as the agent framework and WebRTC routing layer - Modal Sticky Sessions A few really cool things, here. All the models are running co-located on a mixture of GPUs and CPUs: L4, B200, and the Phonon model is running on CPU. Running Phonon on the CPU means that voice audio starts streaming in <50ms and you don't have to manage a GPU cluster for TTS. There are no extra network hops or network overhead in any of the routing. And the agent container uses Modal memory snapshots so a new agent (orchestration code, Phonon model, and the Smart Turn model) are all ready and responding to new requests in under 2 seconds. If you've ever built out voice agent cluster infrastructure on top of, say, Kubernetes, you know how impressive all of the above attributes are! A link to Ben's repo is in the thread below.