Websocket streaming is a method of sending and receiving data over a single, long-lived connection. This method is useful for real-time applications where you need to stream audio data as it becomes available.

If you want to quickly test out the latency (time to first byte) of a websocket connection to the ElevenLabs text-to-speech API, you can install elevenlabs-latency via npm and follow the instructions here.

​
Requirements

  • An ElevenLabs account with an API key (here’s how to find your API key).
  • Python or Node.js/Typescript installed on your machine

​
Setup

Install dotenv package to manage your environmental variables:

Next, create a .env file in your project directory and fill it with your credentials like so:

.env
ELEVENLABS_API_KEY=your_elevenlabs_api_key_here

Last, create a new file to write the code in. You can name it text-to-speech-websocket.py for Python or text-to-speech-websocket.ts for Typescript.

​
Initiate the websocket connection

Pick a voice from the voice library and a text-to-speech model; Then initiate a websocket connection to the text-to-speech API.

For TypeScript, create a write stream ahead for saving the audio into mp3 which can be passed to the websocket listener.

text-to-speech-websocket.ts (Typescript)
import * as fs from "node:fs";

const outputDir = './output'
try {
  fs.accessSync(outputDir, fs.constants.R_OK | fs.constants.W_OK);
} catch (err) {
  fs.mkdirSync(outputDir)
}
const writeStream = fs.createWriteStream(outputDir + '/test.mp3', { flags: 'a' });

​
Send the input text

Once the websocket connection is open, set up voice settings first. Next, send the text message to the API.

​
Save the audio to file

Read the incoming message from the websocket connection and write the audio chunks to a local file.

​
Run the script

You can run the script by executing the following command in your terminal. An mp3 audio file will be saved in the output directory.

​
Understanding buffering

A key concept to understand when using websockets is buffering. The API only runs model generations when a certain amount of text above a threshold has been sent. This is to optimize the quality of the generated audio by maximising the amount of context available to the model while balancing latency.

For example, if the threshold is set to 120 characters and you send ‘Hello, how are you?’, the audio won’t be generated immediately. This is because the sent message has only 19 characters which is below the threshold. However, if you keep sending text, the API will generate audio once the total text sent since the last generation has at least 120 characters.

In the case that you want force the immediate return of the audio, you can use flush=true to clear out the buffer and force generate any buffered text. This can be useful, for example, when you have reached the end of a document and want to generate audio for the final section.

In addition, closing the websocket will automatically force generate any buffered text.

​
Best practice

  • We suggest using the default setting for chunk_length_schedule in generation_config. Avoid using try_trigger_generation as it is deprecated.
  • When developing a real-time conversational AI application, we advise using flush=true along with the text at the end of conversation turn to ensure timely audio generation.
  • If the default setting doesn’t provide optimal latency for your use case, you can modify the chunk_length_schedule. However, be mindful that reducing latency through this adjustment may come at the expense of quality.

​
Tips

  • The API maintains a internal buffer so that it only runs model generations when a certain amount of text above a threshold has been sent. For short texts with a character length smaller than the value set in chunk_length_schedule, you can use flush=true to clear out the buffer and force generate any buffered text.
  • The websocket connection will automatically close after 20 seconds of inactivity. To keep the connection open, you can send a single space character " ". Please note that this string must include a space, as sending a fully empty string, "", will close the websocket.
  • Send an empty string to close the websocket connection after sending the last text message.
  • You can use alignment to get the word-level timestamps for each word in the text. This can be useful for aligning the audio with the text in a video or for other applications that require precise timing.
Reducing LatencyText To Speech