Vai alla navigazione

Agenti vocali multipiattaforma con Expo React Native

Crea agenti ElevenLabs per iOS e Android con Expo e l'SDK React Native di ElevenLabs con supporto WebRTC.

Tutorial · Presuppone che tu abbia completato la guida rapida di ElevenAgents e configurato un ambiente di sviluppo Expo.

Introduzione

In questo tutorial imparerai a creare un agente vocale che funziona su iOS e Android usando Expo React Native e l’SDK React Native di ElevenLabs con supporto WebRTC.

Requisiti

  • Un account ElevenLabs con una chiave API.
  • Node.js v18 o versione successiva installato sul tuo computer.

Configurazione

Crea un nuovo progetto Expo

Usando create-expo-app, crea un nuovo progetto Expo vuoto:

npx create-expo-app@latest --template blank-typescript

Installa le dipendenze

Installa l’SDK React Native di ElevenLabs e le relative dipendenze:

npx expo install @elevenlabs/react-native @livekit/react-native @livekit/react-native-webrtc @config-plugins/react-native-webrtc @livekit/react-native-expo-plugin @livekit/react-native-expo-plugin livekit-client

Se riscontri un problema con le dipendenze peer, aggiungi un file .npmrc nella root del progetto con il seguente contenuto: legacy-peer-deps=true.

Abilita i permessi del microfono e aggiungi i plugin Expo

Nel file app.json, aggiungi i seguenti permessi:

app.json
{
"expo": {
"scheme": "elevenlabs",
// ...
"ios": {
"infoPlist": {
"NSMicrophoneUsageDescription": "This app uses the microphone to record audio."
},
"supportsTablet": true,
"bundleIdentifier": "YOUR.BUNDLE.ID"
},
"android": {
"permissions": [
"android.permission.RECORD_AUDIO",
"android.permission.ACCESS_NETWORK_STATE",
"android.permission.CAMERA",
"android.permission.INTERNET",
"android.permission.MODIFY_AUDIO_SETTINGS",
"android.permission.SYSTEM_ALERT_WINDOW",
"android.permission.WAKE_LOCK",
"android.permission.BLUETOOTH"
],
"adaptiveIcon": {
"foregroundImage": "./assets/adaptive-icon.png",
"backgroundColor": "#ffffff"
},
"package": "YOUR.PACKAGE.ID"
},
"plugins": ["@livekit/react-native-expo-plugin", "@config-plugins/react-native-webrtc"]
// ...
}
}

In questo modo React Native potrà richiedere i permessi per il microfono all’avvio della conversazione.

Nota

Per l’emulatore Android, devi abilitare “Virtual microphone uses host audio input” nelle impostazioni del microfono dell’emulatore.

Aggiungi gli agenti ElevenLabs alla tua app

Aggiungi gli agenti ElevenLabs alla tua app inserendo il seguente codice nel file ./App.tsx:

./App.tsx
import { ConversationProvider, useConversation } from "@elevenlabs/react-native";
import type { ConversationStatus, ConversationEvent, Role } from "@elevenlabs/react-native";
import React, { useState } from "react";
import {
View,
Text,
StyleSheet,
TouchableOpacity,
Keyboard,
TouchableWithoutFeedback,
Platform,
} from "react-native";
import { TextInput } from "react-native";
import { getBatteryLevel, changeBrightness, flashScreen } from "./utils/tools";
const ConversationScreen = () => {
const conversation = useConversation({
clientTools: {
getBatteryLevel,
changeBrightness,
flashScreen,
},
onConnect: ({ conversationId }: { conversationId: string }) => {
console.log("✅ Connected to conversation", conversationId);
},
onDisconnect: (details: string) => {
console.log("❌ Disconnected from conversation", details);
},
onError: (message: string, context?: Record<string, unknown>) => {
console.error("❌ Conversation error:", message, context);
},
onMessage: ({ message, source }: { message: ConversationEvent; source: Role }) => {
console.log(`💬 Message from ${source}:`, message);
},
onModeChange: ({ mode }: { mode: "speaking" | "listening" }) => {
console.log(`🔊 Mode: ${mode}`);
},
onStatusChange: ({ status }: { status: ConversationStatus }) => {
console.log(`📡 Status: ${status}`);
},
onCanSendFeedbackChange: ({ canSendFeedback }: { canSendFeedback: boolean }) => {
console.log(`🔊 Can send feedback: ${canSendFeedback}`);
},
});
const [isStarting, setIsStarting] = useState(false);
const [textInput, setTextInput] = useState("");
const handleSubmitText = () => {
if (textInput.trim()) {
conversation.sendUserMessage(textInput.trim());
setTextInput("");
Keyboard.dismiss();
}
};
const startConversation = async () => {
if (isStarting) return;
setIsStarting(true);
try {
await conversation.startSession({
agentId: process.env.EXPO_PUBLIC_AGENT_ID,
dynamicVariables: {
platform: Platform.OS,
},
});
} catch (error) {
console.error("Failed to start conversation:", error);
} finally {
setIsStarting(false);
}
};
const endConversation = async () => {
try {
await conversation.endSession();
} catch (error) {
console.error("Failed to end conversation:", error);
}
};
const getStatusColor = (status: ConversationStatus): string => {
switch (status) {
case "connected":
return "#10B981";
case "connecting":
return "#F59E0B";
case "disconnected":
return "#EF4444";
default:
return "#6B7280";
}
};
const getStatusText = (status: ConversationStatus): string => {
return status[0].toUpperCase() + status.slice(1);
};
const canStart = conversation.status === "disconnected" && !isStarting;
const canEnd = conversation.status === "connected";
return (
<TouchableWithoutFeedback onPress={() => Keyboard.dismiss()}>
<View style={styles.container}>
<Text style={styles.title}>ElevenLabs React Native Example</Text>
<Text style={styles.subtitle}>Remember to set the agentId in the .env file!</Text>
<View style={styles.statusContainer}>
<View
style={[styles.statusDot, { backgroundColor: getStatusColor(conversation.status) }]}
/>
<Text style={styles.statusText}>{getStatusText(conversation.status)}</Text>
</View>
{/* Speaking Indicator */}
{conversation.status === "connected" && (
<View style={styles.speakingContainer}>
<View
style={[
styles.speakingDot,
{
backgroundColor: conversation.isSpeaking ? "#8B5CF6" : "#D1D5DB",
},
]}
/>
<Text
style={[
styles.speakingText,
{ color: conversation.isSpeaking ? "#8B5CF6" : "#9CA3AF" },
]}
>
{conversation.isSpeaking ? "🎤 AI Speaking" : "👂 AI Listening"}
</Text>
</View>
)}
<View style={styles.buttonContainer}>
<TouchableOpacity
style={[styles.button, styles.startButton, !canStart && styles.disabledButton]}
onPress={startConversation}
disabled={!canStart}
>
<Text style={styles.buttonText}>
{isStarting ? "Starting..." : "Start Conversation"}
</Text>
</TouchableOpacity>
<TouchableOpacity
style={[styles.button, styles.endButton, !canEnd && styles.disabledButton]}
onPress={endConversation}
disabled={!canEnd}
>
<Text style={styles.buttonText}>End Conversation</Text>
</TouchableOpacity>
</View>
{/* Feedback Buttons */}
{conversation.status === "connected" && conversation.canSendFeedback && (
<View style={styles.feedbackContainer}>
<Text style={styles.feedbackLabel}>How was that response?</Text>
<View style={styles.feedbackButtons}>
<TouchableOpacity
style={[styles.button, styles.likeButton]}
onPress={() => conversation.sendFeedback(true)}
>
<Text style={styles.buttonText}>👍 Like</Text>
</TouchableOpacity>
<TouchableOpacity
style={[styles.button, styles.dislikeButton]}
onPress={() => conversation.sendFeedback(false)}
>
<Text style={styles.buttonText}>👎 Dislike</Text>
</TouchableOpacity>
</View>
</View>
)}
{/* Text Input and Messaging */}
{conversation.status === "connected" && (
<View style={styles.messagingContainer}>
<Text style={styles.messagingLabel}>Send Text Message</Text>
<TextInput
style={styles.textInput}
value={textInput}
onChangeText={(text) => {
setTextInput(text);
// Prevent agent from interrupting while user is typing
if (text.length > 0) {
conversation.sendUserActivity();
}
}}
placeholder="Type your message or context... (Press Enter to send)"
multiline
onSubmitEditing={handleSubmitText}
returnKeyType="send"
blurOnSubmit={true}
/>
<View style={styles.messageButtons}>
<TouchableOpacity
style={[styles.button, styles.messageButton]}
onPress={handleSubmitText}
disabled={!textInput.trim()}
>
<Text style={styles.buttonText}>💬 Send Message</Text>
</TouchableOpacity>
<TouchableOpacity
style={[styles.button, styles.contextButton]}
onPress={() => {
if (textInput.trim()) {
conversation.sendContextualUpdate(textInput.trim());
setTextInput("");
Keyboard.dismiss();
}
}}
disabled={!textInput.trim()}
>
<Text style={styles.buttonText}>📝 Send Context</Text>
</TouchableOpacity>
</View>
</View>
)}
</View>
</TouchableWithoutFeedback>
);
};
export default function App() {
return (
<ConversationProvider>
<ConversationScreen />
</ConversationProvider>
);
}
const styles = StyleSheet.create({
container: {
flex: 1,
justifyContent: "center",
alignItems: "center",
backgroundColor: "#F3F4F6",
padding: 20,
},
title: {
fontSize: 24,
fontWeight: "bold",
marginBottom: 8,
color: "#1F2937",
},
subtitle: {
fontSize: 16,
color: "#6B7280",
marginBottom: 32,
},
statusContainer: {
flexDirection: "row",
alignItems: "center",
marginBottom: 24,
},
statusDot: {
width: 12,
height: 12,
borderRadius: 6,
marginRight: 8,
},
statusText: {
fontSize: 16,
fontWeight: "500",
color: "#374151",
},
speakingContainer: {
flexDirection: "row",
alignItems: "center",
marginBottom: 24,
},
speakingDot: {
width: 12,
height: 12,
borderRadius: 6,
marginRight: 8,
},
speakingText: {
fontSize: 14,
fontWeight: "500",
},
toolsContainer: {
backgroundColor: "#E5E7EB",
padding: 16,
borderRadius: 8,
marginBottom: 24,
width: "100%",
},
toolsTitle: {
fontSize: 14,
fontWeight: "600",
color: "#374151",
marginBottom: 8,
},
toolItem: {
fontSize: 12,
color: "#6B7280",
fontFamily: "monospace",
marginBottom: 4,
},
buttonContainer: {
width: "100%",
gap: 16,
},
button: {
backgroundColor: "#3B82F6",
paddingVertical: 16,
paddingHorizontal: 32,
borderRadius: 8,
alignItems: "center",
},
startButton: {
backgroundColor: "#10B981",
},
endButton: {
backgroundColor: "#EF4444",
},
disabledButton: {
backgroundColor: "#9CA3AF",
},
buttonText: {
color: "white",
fontSize: 16,
fontWeight: "600",
},
instructions: {
marginTop: 24,
fontSize: 14,
color: "#6B7280",
textAlign: "center",
lineHeight: 20,
},
feedbackContainer: {
marginTop: 24,
alignItems: "center",
},
feedbackLabel: {
fontSize: 16,
fontWeight: "500",
color: "#374151",
marginBottom: 12,
},
feedbackButtons: {
flexDirection: "row",
gap: 16,
},
likeButton: {
backgroundColor: "#10B981",
},
dislikeButton: {
backgroundColor: "#EF4444",
},
messagingContainer: {
marginTop: 24,
width: "100%",
},
messagingLabel: {
fontSize: 16,
fontWeight: "500",
color: "#374151",
marginBottom: 8,
},
textInput: {
backgroundColor: "#FFFFFF",
borderRadius: 8,
padding: 16,
minHeight: 100,
textAlignVertical: "top",
borderWidth: 1,
borderColor: "#D1D5DB",
marginBottom: 16,
},
messageButtons: {
flexDirection: "row",
gap: 16,
},
messageButton: {
backgroundColor: "#3B82F6",
flex: 1,
},
contextButton: {
backgroundColor: "#4F46E5",
flex: 1,
},
activityContainer: {
marginTop: 24,
alignItems: "center",
},
activityLabel: {
fontSize: 14,
color: "#6B7280",
marginBottom: 8,
textAlign: "center",
},
activityButton: {
backgroundColor: "#F59E0B",
},
});

Strumenti client nativi

Una parte importante della creazione di agenti ElevenLabs consiste nel consentire all’agente di accedere alle funzionalità ed eseguirle dinamicamente. Puoi farlo tramite gli strumenti client.

Crea un nuovo file per gli strumenti client: ./utils/tools.ts, quindi aggiungi il seguente codice:

./utils/tools.ts
import * as Battery from "expo-battery";
import * as Brightness from "expo-brightness";
const getBatteryLevel = async () => {
const batteryLevel = await Battery.getBatteryLevelAsync();
console.log("batteryLevel", batteryLevel);
if (batteryLevel === -1) {
return "Error: Device does not support retrieving the battery level.";
}
return batteryLevel;
};
const changeBrightness = ({ brightness }: { brightness: number }) => {
console.log("changeBrightness", brightness);
Brightness.setSystemBrightnessAsync(brightness);
return brightness;
};
const flashScreen = () => {
Brightness.setSystemBrightnessAsync(1);
setTimeout(() => {
Brightness.setSystemBrightnessAsync(0);
}, 200);
return "Successfully flashed the screen.";
};
export { getBatteryLevel, changeBrightness, flashScreen };

Variabili dinamiche

Oltre agli strumenti client, inseriamo anche la piattaforma (web, iOS, Android) come variabile dinamica sia nel primo messaggio sia nel prompt:

./App.tsx
// ...
const startConversation = async () => {
if (isStarting) return;
setIsStarting(true);
try {
await conversation.startSession({
agentId: process.env.EXPO_PUBLIC_AGENT_ID,
dynamicVariables: {
platform: Platform.OS,
},
});
} catch (error) {
console.error("Failed to start conversation:", error);
} finally {
setIsStarting(false);
}
};
// ...

Configurazione dell’agente

1

Accedi a ElevenLabs

Vai su elevenlabs.io e accedi al tuo account.

2

Crea un nuovo agente

Vai a Agents Platform > Agents e crea un nuovo agente dal template vuoto.

3

Imposta il primo messaggio

Imposta il primo messaggio e specifica la variabile dinamica per la piattaforma.

Hi there, woah, so cool that I'm running on {{platform}}. What can I help you with?
4

Imposta il prompt di sistema

Imposta il prompt di sistema. Puoi includere anche qui le variabili dinamiche.

You are a helpful assistant running on {{platform}}. You have access to certain tools that allow you to check the user device battery level and change the display brightness. Use these tools if the user asks about them. Otherwise, just answer the question.
5

Configura gli strumenti client

Configura i seguenti strumenti client:

  • Nome: getBatteryLevel
    • Descrizione: Ottiene il livello della batteria del dispositivo come percentuale decimale.
    • Attendi risposta: true
    • Timeout della risposta (secondi): 3
  • Nome: changeBrightness
    • Descrizione: Modifica la luminosità dello schermo del dispositivo.
    • Attendi risposta: true
    • Timeout della risposta (secondi): 3
    • Parametri:
      • Tipo di dati: number
      • Identificatore: brightness
      • Obbligatorio: true
      • Tipo di valore: LLM Prompt
      • Descrizione: Un numero compreso tra 0 e 1, inclusi, che rappresenta la luminosità desiderata dello schermo.
  • Nome: flashScreen
    • Descrizione: Fa lampeggiare rapidamente lo schermo accendendolo e spegnendolo.
    • Attendi risposta: true
    • Timeout della risposta (secondi): 3

Avvia l’app

Questa app richiede alcune dipendenze native non supportate da Expo Go, quindi devi prima eseguire il prebuild dell’app e poi avviarla su un dispositivo nativo.

  • Terminale 1:
    • Esegui npx expo prebuild --clean
npx expo prebuild --clean
  • Esegui npx expo start --tunnel per avviare il server di sviluppo Expo tramite https.
npx expo start --tunnel
  • Terminale 2:
    • Esegui npx expo run:ios --device per avviare l’app sul tuo dispositivo iOS.
npx expo run:ios --device

Passaggi successivi