Webinar Recap: Building AI Agents That Sound Natural

Most teams want to build an agent that sounds natural.

The gap most team see are agents that complete the task - but it doesn't understand context, urgency, or emotion. And once a caller notices that the trust drops and so does the quality of the interaction.

In our recent Live Workshop: Building AI Agents That Sound Natural, Luigi Sambuy (Forward Deployed Engineering) walked through what separates a robotic-sounding agent from one that feels genuinely human.

Why sounding natural is harder than it looks

Most teams that deploy agents have already solved for accuracy - the agent gets the right answer, books the right table, cancels the right flight. 

What's harder is getting the delivery right.

An agent that doesn't mirror a caller's emotion, doesn't pick up on urgency, and answers every question in the same flat rhythm will always read as artificial, no matter how correct its answers are. Customers don't just want their problem solved — they want to feel heard while it happens.

The four traits of a natural-sounding agent

1. Rhythm & flow - natural pacing, pauses, and speed variation instead of a monotone read
2. Mirroring - responding to a frustrated caller with concern, and to good news with warmth
3. Context - pulling in user history, business systems, and cultural/regional cues so answers aren't generic
4. Identity - a voice that's actually built for the brand, not an off the shelf default

Demo: A natural-sounding agent in action

Scenario: A customer calls Rosette Restaurant to move their reservation. It's their fifth wedding anniversary, and they're running late.

What’s shown:

• The agent opened warmly, acknowledging the anniversary before doing anything else
• It caught a small mismatch between the stated reservation time and what was actually booked, and clarified rather than guessing
• It moved the reservation to 9 PM and proactively offered a window table for the occasion, without being asked
• Throughout the call, the agent used natural pauses, filler words, and a tone that adjusted to the moment - reassuring, warm, unhurried

Why it matters: None of this came from a different model or a more powerful platform. It came from the prompt. The system prompt gave the agent permission to self-correct mid-sentence, trail off naturally while "looking something up," and shift its emotional tone to match the caller - the same techniques available to any team building on the platform today.

Seven levers for building natural agents

1. Emotional tags - available with the v3 conversational model's expressive mode; the single biggest unlock for tone variation
2. Background noise - office ambience, urban sound, or typing noise for a more familiar, human context
3. Tone in the system prompt - this is where most of the difference actually lives
4. Eagerness settings (eager / normal / patient) - matched to how much time the interaction needs
5. Pronunciation dictionary / keywords - for names, brands, and foreign words the agent would otherwise mispronounce. 
6. Output settings - some teams deliberately add phones ringing in the background instead of pristine audio, since customers migrating from a traditional contact center are used to that slightly muffled tone
7. Voice design or cloning - building a fully custom, on-brand voice rather than picking one off the shelf. Voice design lets you prompt for something specific (e.g. "a male in his thirties who sounds like he's coming through a headset"). Voice cloning ranges from an instant clone (about 30 seconds of audio) to a professional clone (more material, higher fidelity). 

Best practices for natural-sounding agents

Write the tone directly into the system prompt. This is where most of the difference between a flat agent and a natural one actually lives. Give the agent explicit permission to self-correct mid-sentence ("the way people actually talk"), to trail off naturally while looking something up, and to let silence land after something serious before responding.

Match emotional tags to the moment. An "excited" tag is wrong for a serious or risky moment. Place tags deliberately for where they'll actually occur in the conversation, and adapt them to the caller's emotional state as the call progresses (e.g. calm and reassuring for an anxious caller vs. quick-but-warm for someone in a rush).

Set eagerness deliberately. Eager works for snappy, high-information exchanges; patient works when someone needs time to retrieve a document or number. Defaulting to eager across the board can make an agent feel pushy rather than natural.

Choose your LLM per node. Use lighter, faster models for simple routing steps and more capable models for nodes doing real decision-making. 

Treat perceived latency as part of "natural." Beyond model choice, speculative turn generation (letting the model start forming a response while the caller is still speaking) and soft-timeout filler words both reduce how long a pause feels, even when the underlying response time hasn't changed.

Localize rather than translate. Pick voices actually trained in the target language, use a pronunciation dictionary, and for single-language use cases, consider writing the prompt itself in that language for a performance boost. Multilingual v2 remains a solid option when snappy latency across languages matters more than expressive range.

Monitor your agent. Use simulation tests to validate an agent without live calls, evaluation criteria for pass/fail post-call scoring, and turn by turn sentiment analysis to catch exactly where in a conversation an agent used the wrong tone - then use that signal to tune the prompt.

Watch the full session

Watch the full webinar here.