Interaction models: Building natural human-AI dialogue

Anyone who has tried to interrupt an AI voice agent mid-sentence knows what it feels like when a system is not built for human conversation. The rhythm is wrong, the voice is detached from the content, and even when the information is correct, the interaction feels off: not like talking to a knowledgeable person, but like navigating software that happens to use words.

The cause of this unnatural feel is structural: many voice AI systems were built to handle turns, not conversations. They listen, process, and respond to one exchange at a time. That works for simple demos, but it breaks down when the conversation becomes emotional and unpredictable.

Interaction models are AI systems designed to communicate across audio and text in real time, perceiving not just what is said, but when it's said and the kind of emotional response the moment calls for. This article explains what makes an interaction model different, why it matters for businesses deploying voice AI, and how ElevenLabs is building toward it.

Summary

• Most voice AI breaks down in real conversation because it can't handle interruption, silence, or context that carries across turns.
• ElevenLabs’ interaction stack is designed to handle interruptions, pauses, and overlapping speech without losing context or breaking the flow of conversation.
• ElevenLabs is building toward true interaction models with an advanced cascaded architecture, in-house Speech to Text (STT) and Text to Speech (TTS), and a stack tuned for low latency and natural conversation that flows back and forth.

Why most human-AI voice communication doesn’t feel natural

Most voice AI treats a conversation as a series of discrete inputs and outputs: the system waits for a block of speech, reduces it to text, processes that text, and replies. This works for command-and-response interactions, but a real conversation is not a clean sequence of text exchanges. It is a continuous back and forth filled with pauses and interjections. 

Stripping away the flow between turns and the context that carries across them produces three specific failures:

• It interrupts you, or makes you wait: The system can't distinguish a pause for thought from a finished turn, so it either jumps in while you're still speaking or sits in silence after you've stopped. Gather your thoughts mid-sentence, and you get cut off; finish your point, and you wait through a beat of dead air.
• It can't react once it starts talking: The moment the system begins responding, it stops listening. If you interrupt to redirect, it keeps delivering the answer to a question you've moved on from, because it can't register that anything changed.
• It forgets as it goes: Each turn is processed in isolation, so context from five exchanges ago doesn't carry forward. You end up repeating yourself or the system responds like the conversation just started.

The result is a conversation that can be functionally correct and still feel wrong. 

What interaction models do that traditional voice AI cannot

Where traditional voice AI handles one turn at a time, an interaction model runs the full conversation, attending to what is being said and what needs to happen next, all at once. The functional difference is that an interaction model responds to the actual state of the exchange, not just the most recent input.

Interaction models have:

• Real-time response: The system is designed to respond at conversational speed, with end-to-end cycle that can run in under a second depending on configuration.
• Natural handling of interruption, silence, and overlap: It differentiates a pause for thought from a finished turn, so it doesn't cut people off or leave dead air. And when the customer talks over it to redirect, it manages the overlap without losing context or collapsing the conversation.
• Continuity across the full exchange: Rather than resetting context with each turn, the system carries the history of the conversation forward, so what it says at turn 10 reflects everything that has happened since turn one.
• Adaptive delivery: The voice can be programmed to shift (calmer, more direct, more reassuring) depending on the kind of task it is carrying out.
• Parallel task execution: The system retrieves information, runs tool calls, and keeps talking at the same time, rather than going silent while it looks something up.

The test of an interaction model is a call that's going badly. In the recording below, a customer calls about a flight cancellation. They’re tense and need a resolution to their problem quickly.

The agent recognizes the urgency and frustration coming from the customer immediately from the words they are using. It adjusts its tone, handles the interruption without losing its place, and uses its connected tools to offer real solutions to the canceled flight. In the end, the customer gets a resolution without needing a human agent.

Compare this to typical turn-based agents used today. These systems would wait for each pause, answer from a neutral baseline, and miss the frustration the customer is feeling entirely. After a few exchanges that don't address what the caller is actually feeling, the call would likely need to be handed off to a human, leaving the customer more frustrated than when they began.

How ElevenLabs is building toward interaction models

Our conversation pipeline uses an advanced cascaded architecture rather than a fused one, so instead of one model handling everything, each stage is its own specialized component.

The advantage of this approach is that we can optimize each stage of the pipeline independently, swapping in a better model for any component without rebuilding the whole system. And because we build those components in-house, they are co-optimized to pass rich context to each other, not just data. This means the pipeline still behaves like one coherent conversation rather than a chain of separate tools.

Cascaded model structure

Fused model structure

Images: Cascaded vs Fused models

Here's the technology that makes up the pipeline currently:

• Scribe v2 Realtime: Our in-house STT model transcribes speech in around 150 ms across 90+ languages, holding up against background noise, accents, interruptions, and non-verbal events like laughs and pauses. It's built to handle domain-specific vocabulary too, from medical terms to financial jargon.
• Speculative turn-taking: This system decides when to speak, pause, or wait, reading the flow of conversation instead of relying on a hard silence threshold. Improvements to our voice activity detection model help it better filter background speech and short responses, making turn-taking feel more natural.
• Eleven v3 Conversational: Our most expressive TTS model, built for live, back-and-forth dialogue. It carries the emotional temperature of the conversation forward across turns, so the agent's delivery at turn 10 reflects everything that came before it, not just the most recent response.
• Expressive Mode: Built on Eleven v3 Conversational and the turn-taking system, Expressive Mode controls how the agent sounds in the moment - de-escalating when customers are frustrated, reassuring when they’re confused, direct when clarity is what's needed. It also reads expressive tags, so the model can act on cues like [laughs], [whispers], or [sighs] to shape specific moments of delivery.
• Flash v2.5: Our low-latency TTS model supports 32 languages and generates speech in under 75 ms. When latency is the priority, it keeps the response time within the range of natural human rhythm.
• Speech Engine: The connective layer that ties the stack together, linking your server to our ElevenAPI over WebSocket, one connection per conversation. We handle STT and TTS while your server runs the LLM, so technical teams can bring their own model and keep conversation logic on their own infrastructure.

These models are constantly being improved, with new versions being released regularly. Each new release narrows the gap between talking to software and talking to a person, with faster responses, sharper emotional recognition, more languages, and smoother delivery.

Talking while thinking

Not every part of an interaction model has to run in a strict sequence. ElevenAgents can keep working in the background while the conversation continues, rather than defaulting to silence between a question and an answer. 

A few core systems work together to enable simultaneous talking and thinking:

• Parallel tool calls: The agent can query a database, run a tool, or check an order status while it’s still speaking, so retrieval never feels like a dead pause in the conversation. 
• Soft timeout: If an LLM takes longer than expected to generate a response, the agent speaks a brief filler phrase like “Let me think” or “hmmm” instead of leaving an awkward silence. Soft timeout helps maintain a natural conversational flow and reduces the likelihood of interruptions. 
• Interruption ignore terms: Rather than using a fixed silence threshold, the system aims to read the meaning of what’s being said to intuit when a turn has actually finished. Similarly, short affirmations like “okay” or “mm-hmm” can be configured to pass through without triggering a full interruption, meaning the agent doesn’t lose its place every time the caller interjects.

Together, these systems converge to keep the agent from feeling like it’s buffering or loading a response. They form a streamlined conversational engine that fills the gap naturally the way a person would, while simultaneously processing information and gathering thoughts in the background.

How a conversation actually works with ElevenLabs

The components above don't run one after another in a tidy line. They overlap. Here’s how an agent from ElevenLabs would handle a caller who asks, "Has my order shipped yet, or is it still processing?" partway through a support call.

1. The caller speaks: Audio streams to ElevenLabs over the WebSocket connection, and Scribe begins transcribing in real time, around 150 ms behind the voice. 
2. The system reads the flow: While Scribe is still transcribing, speculative turn-taking is already assessing whether the caller has finished or is mid-thought. The trailing "or is it still processing?" reads as a handoff rather than a pause, so it triggers the next step instead of waiting in silence.
3. The LLM assembles context and responds: The transcribed question goes to the LLM alongside the conversation history, the order record retrieved from the business's own systems via RAG, tool outputs from earlier in the call, and the system prompt. It reasons over all of it and generates a response grounded in the caller's actual order, not a generic status message.
4. Eleven v3 synthesizes the reply: The text becomes natural-sounding audio. If Expressive Mode is active, the response carries delivery cues that match the moment, so a routine update sounds easy and unhurried rather than flat. When latency is the priority, Flash handles synthesis in under 75 ms.
5. The reply streams back: Audio starts playing before synthesis is finished, so the caller hears the start of the answer while the rest is still being generated. 
6. The cycle repeats: Each new turn carries the tone, context, and history of the conversation forward.

Across this rapid process, the conversation feels natural. The caller stops adjusting to the machine: they don't slow down, over-enunciate, or wait for a beep. They just talk, the way they would with a person.

Deploy natural-sounding voice agents across your business

Everything described here is in production today, not on a roadmap. From the cascaded architecture to the sub-second pipeline, businesses around the world are already using ElevenAgents to handle real customer conversations at scale.

That includes regulated, high-stakes environments, because our agent architecture supports guardrails, audit logs, and compliance controls. ElevenLabs is certified for SOC 2 Type II, ISO 27001, HIPAA, and PCI DSS Level 1, with Zero Retention Mode and regional data residency for teams that need data to stay inside a specific boundary.

Ready to put an agent to work? You can create an agent and start building in the console, or talk to our sales team about a deployment scoped to your environment.

Interaction models FAQ