OpenAI Realtime API compatibility

ElevenLabs Realtime speaks the same wire protocol as OpenAI’s Realtime API: the same client and server events, over the same WebSocket transport. If you already have an app built with OpenAI’s Realtime SDK, you can point it at ElevenLabs Realtime instead of api.openai.com without switching SDKs.

Available models

How it works

1. Your backend requests a short-lived client secret from ElevenLabs, scoped to a specific session configuration.
2. Your frontend passes that client secret to OpenAI’s Realtime SDK in place of an OpenAI API key.
3. The SDK connects over WebSocket to ElevenLabs Realtime, and the conversation proceeds using the same client and server events as OpenAI’s Realtime API.

Mint a client secret

Request the client secret from your backend, not the browser, so your ElevenLabs API key never reaches the client.

Create an API key in the dashboard here, which you’ll use to securely access the API.

Store the key as a managed secret and pass it to the SDKs either as an environment variable via an .env file, or directly in your app’s configuration depending on your preference.

ELEVENLABS_API_KEY=<your_api_key_here>

import os
from dotenv import load_dotenv
from elevenlabs import (
    CreateClientSecretRequest,
    ExpiresAfter,
    OpenAiAudioConfigInput,
    OpenAiAudioInputConfigInput,
    OpenAiAudioInputFormat,
    OpenAiAudioOutputConfigInput,
    OpenAiAudioOutputFormat,
    OpenAiSessionConfigInput,
    OpenAiTurnDetection,
)
from elevenlabs.client import ElevenLabs
load_dotenv()
elevenlabs = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))
client_secret = elevenlabs.conversational_ai.realtime.create_client_secret(
    request=CreateClientSecretRequest(
        session=OpenAiSessionConfigInput(
            type="realtime",
            model="eleven_realtime_v1_mini",
            instructions="You are a helpful, friendly voice assistant. Keep replies concise.",
            output_modalities=["audio"],
            audio=OpenAiAudioConfigInput(
                input=OpenAiAudioInputConfigInput(
                    format=OpenAiAudioInputFormat(type="audio/pcm", rate=24000),
                    turn_detection=OpenAiTurnDetection(type="semantic_vad", eagerness="medium"),
                ),
                output=OpenAiAudioOutputConfigInput(
                    format=OpenAiAudioOutputFormat(type="audio/pcm"),
                    voice="brian",
                ),
            ),
        ),
        expires_after=ExpiresAfter(anchor="created_at", seconds=600),
    ),
)

The response contains value (the ephemeral credential your frontend will use), its expiry time, and the negotiated session. Return only value to the browser.

Connect from the browser

Pass the client secret to OpenAI’s Realtime SDK (@openai/agents-realtime) as the API key, and point the WebSocket transport at the ElevenLabs endpoint instead of the default OpenAI one.

import { OpenAIRealtimeWebSocket, RealtimeAgent, RealtimeSession } from "@openai/agents-realtime";
const { value: clientSecret } = await fetch("/api/token").then((res) => res.json());
const agent = new RealtimeAgent({
  name: "ElevenLabs agent",
  instructions: "You are a helpful, friendly voice assistant. Keep replies concise.",
});
const transport = new OpenAIRealtimeWebSocket({
  url: "wss://api.elevenlabs.io/v1/convai/realtime",
});
const session = new RealtimeSession(agent, {
  transport,
  model: "eleven_realtime_v1_mini",
  config: {
    outputModalities: ["audio"],
    audio: { output: { voice: "brian" } },
  },
});
await session.connect({ apiKey: clientSecret });

The SDK only wires audio automatically for its WebRTC transport. Over WebSocket, capture and play PCM16 audio at 24kHz yourself, for example with wavtools, the same helper OpenAI uses in its own WebSocket examples.

Limitations

This guide covers connecting directly from a browser. Documentation for connecting from server-side OpenAI Realtime SDKs isn’t published yet.

FAQ