Twilio custom server

Learn how to integrate a Conversational AI agent with Twilio to create seamless, human-like voice interactions.

Connect your ElevenLabs Conversational AI agent to phone calls and create human-like voice experiences using Twilio’s Voice API.

What You’ll Need

Agent Configuration

Before integrating with Twilio, you’ll need to configure your agent to use the correct audio format supported by Twilio.

1

Configure TTS Output

  1. Navigate to your agent settings
  2. Go to the Voice Section
  3. Select “μ-law 8000 Hz” from the dropdown
2

Set Input Format

  1. Navigate to your agent settings
  2. Go to the Advanced Section
  3. Select “μ-law 8000 Hz” for the input format

Implementation

Looking for a complete example? Check out this Javascript implementation on GitHub.

1

Initialize the Project

First, set up a new Node.js project:

$mkdir conversational-ai-twilio
>cd conversational-ai-twilio
>npm init -y; npm pkg set type="module";
2

Install dependencies

Next, install the required dependencies for the project.

$npm install @fastify/formbody @fastify/websocket dotenv fastify ws
3

Create the project files

Create a .env & index.js file with the following code:

ELEVENLABS_AGENT_ID=<your-agent-id>
4

Run the server

You can now run the server with the following command:

$node index.js

If the server starts successfully, you should see the message [Server] Listening on port 8000 (or the port you specified) in your terminal.

Twilio Setup

1

Create a Public URL

Use ngrok to make your local server accessible:

$ngrok http --url=<your-url-here> 8000
2

Configure Twilio

  1. Go to the Twilio Console
  2. Navigate to Phone Numbers → Manage → Active numbers
  3. Select your phone number
  4. Under “Voice Configuration”, set the webhook for incoming calls to: https://your-ngrok-url.ngrok.app/incoming-call-eleven
  5. Set the HTTP method to POST

Testing

  1. Call your Twilio phone number.
  2. Start speaking - you’ll see the transcripts in the ElevenLabs console.

Troubleshooting

If the WebSocket connection fails:

  • Verify your ngrok URL is correct in Twilio settings
  • Check that your server is running and accessible
  • Ensure your firewall isn’t blocking WebSocket connections

If there’s no audio output:

  • Confirm your ElevenLabs API key is valid
  • Verify the AGENT_ID is correct
  • Check audio format settings match Twilio’s requirements (μ-law 8kHz)

Security Best Practices

Follow these security guidelines for production deployments:

  • Use environment variables for sensitive information - Implement proper authentication for your endpoints - Use HTTPS for all communications - Regularly rotate API keys - Monitor usage to prevent abuse
Built with