Entity detection

Overview

Entity detection is a feature that allows you to detect specific words and phrases in the transcript, providing their exact timestamps. This is useful to detect credit card numbers, names, medical conditions or SSNs which can then be redacted.

When enabled, the model will detect specific entity types in the transcript and provide their exact timestamps.

For example, the following audio:

Outputs the following transcript when we specify "pii" (Personally Identifiable Information) as the entity type:

1
{
2
  "language_code": "eng",
3
  "language_probability": 0.9869821071624756,
4
  "text": "My name is Jill. My date of birth is the 12th of July 1987, and my credit card number is 4242-4242-4242-4242.",
5
  "words": [
6
    { "text": "My", "start": 0.099, "end": 0.259, "type": "word", "logprob": 0.0 },
7
    { "text": " ", "start": 0.259, "end": 0.299, "type": "spacing", "logprob": 0.0 },
8
    { "text": "name", "start": 0.299, "end": 0.42, "type": "word", "logprob": 0.0 },
9
    ...
10
  ],
11
  "transcription_id": "Y2ZX8AxHUzTPCIualYiE",
12
  "entities": [
13
    {
14
      "text": "Jill",
15
      "entity_type": "name",
16
      "start_char": 11,
17
      "end_char": 15
18
    },
19
    {
20
      "text": "12th of July 1987",
21
      "entity_type": "dob",
22
      "start_char": 41,
23
      "end_char": 58
24
    },
25
    {
26
      "text": "4242-4242-4242-4242",
27
      "entity_type": "credit_card",
28
      "start_char": 89,
29
      "end_char": 108
30
    }
31
  ]
32
}

The result shows the PII entities detected in the transcript, with their exact timestamps.

Integrating entity detection

Entity detection is integrated into the Speech to Text API by passing the entity_detection parameter to the convert method.

1
import os
2
from dotenv import load_dotenv
3
from io import BytesIO
4
import requests
5
from elevenlabs.client import ElevenLabs
6
7
load_dotenv()
8
9
elevenlabs = ElevenLabs(
10
    api_key=os.getenv("ELEVENLABS_API_KEY"),
11
)
12
13
audio_url = (
14
    "https://storage.googleapis.com/eleven-public-cdn/documentation_assets/audio/stt-entity-detection-pii.mp3"
15
)
16
response = requests.get(audio_url)
17
audio_data = BytesIO(response.content)
18
19
transcription = elevenlabs.speech_to_text.convert(
20
    file=audio_data,
21
    model_id="scribe_v2", # Model to use
22
    # Entity types to detect, accepts a list of specific entity types or categories.
23
    # To detect all entity types, use the "all" category.
24
    entity_detection=["pii"],
25
)
26
27
print("Entities detected:", transcription.entities)

Entity types

The following entity types are supported for detection. You can detect entire groups using pii, phi, or pci, or specify individual entity types by their label. To detect all entity types, use the all category.