Production-Ready Conversational AI at Enterprise Scale: With Scale AI’s Felix Su

Successful enterprise AI isn't just about having the latest technology—it's about having the right architecture and controls built on top of AI primitives. Off-the-shelf LLMs are powerful, but lack deep knowledge about how to follow business logic, brand guidelines, and safety principles. This custom experience can only be achieved by carefully architecting and testing a system of prompts, LLMs, guardrails, tools, and other building blocks to guide generative AI behavior to follow enterprise rules.

The Power of Separation: Why Architecture Matters

When building Conversational Voice AI, developers can either work with multi-modal, voice-to-voice models, or they can stitch together the component parts of ASR (transcription), LLMs, and TTS. For enterprises with specific use-cases, the latter can often be more practical.

"In an enterprise setting, AI often can’t be done with just LLMs” explains Felix Su, Head of Engineering for Scale's GenAI Platform. “A complex system needs to be designed to follow business logic, maintain brand image, and operate within strict guardrails. These behaviors are custom for each enterprise, so it’s more practical to implement these systems on top of LLMs, rather than rely purely on their latent behavior. While multi-modal capabilities have recently been baked into models directly, adoption of these models into critical production use cases has been slower than expected. This is due to lack of control between the generation and the inputs and outputs in their modality. In the case of TIME AI, using STT and TTS allowed us to both control the inputs to text-based LLMs and guard the outputs of our system using custom guardrails. This technique can often be more practical than directly using multi-modal LLMs."

When building the TIME AI experience, Scale indexed TIME’s corpus of articles into knowledge bases for on-demand retrieval, encoded business logic into a series of system prompts, and protected GenAI outputs against hallucinations and breaches in both safety and brand guidelines. On top of this system, they brought the experience to life with a voice from ElevenLabs Conversational AI orchestration platform. 

TIME's Person of the Year Experience: Where Voice Makes the Difference

The recent launch of TIME AI allows readers to engage in natural conversations about TIME's journalism, including their iconic Person of the Year coverage. What sets this implementation apart from countless other chatbots isn't just its knowledge—it's the voice.

"Using the voice really added a nice punchy feel to it," Su reflects. "Without it, it gets lost in the shuffle as just another chat bot. We wanted the agent to feel like a live reading partner, available for off-script conversational deep dives." The implementation combines Scale AI's expertise in AI development with ElevenLabs' Conversational AI platform, creating an experience that feels remarkably human while maintaining strict control over content and brand voice.

Building Enterprise-Ready Systems

For enterprises looking to deploy conversational AI at scale, collaborating with Scale AI and ElevenLabs offers a proven pathway. Scale AI brings deep expertise in building controlled, safe multi-modal AI systems with robust guardrails, while ElevenLabs provides best-in-class voice technology that can be seamlessly integrated into these systems.

While TIME's implementation focuses on journalism, the same architecture can be applied across various enterprise use cases, particularly in customer service and support. Customers are growing tired of basic chatbots as they demand more engaging, human-like interactions. 

Thank you to Felix for contributing to this post. You can find Felix Su on LinkedIn & X, and you can experience the Conversational AI voice experience he built with TIME on their Person of the Year 2024, as well as Person of the Year for 2021, 2022 and 2023.

Developers can get started with Conversational AI using our docs, and businesses looking to deploy trusted Conversational Voice solutions in production can get in touch with the Enterprise solutions team at ElevenLabs and Scale.