Entity detection

Overview

Entity detection is a feature that allows you to detect specific words and phrases in the transcript, providing their exact timestamps. This is useful to detect credit card numbers, names, medical conditions or SSNs which can then be redacted.

When enabled, the model will detect specific entity types in the transcript and provide their exact timestamps.

For example, the following audio:

Outputs the following transcript when we specify "pii" (Personally Identifiable Information) as the entity type:

1
{
2
  "language_code": "eng",
3
  "language_probability": 0.9869821071624756,
4
  "text": "My name is Jill. My date of birth is the 12th of July 1987, and my credit card number is 4242-4242-4242-4242.",
5
  "words": [
6
    { "text": "My", "start": 0.099, "end": 0.259, "type": "word", "logprob": 0.0 },
7
    { "text": " ", "start": 0.259, "end": 0.299, "type": "spacing", "logprob": 0.0 },
8
    { "text": "name", "start": 0.299, "end": 0.42, "type": "word", "logprob": 0.0 },
9
    ...
10
  ],
11
  "transcription_id": "Y2ZX8AxHUzTPCIualYiE",
12
  "entities": [
13
    {
14
      "text": "Jill",
15
      "entity_type": "name",
16
      "start_char": 11,
17
      "end_char": 15
18
    },
19
    {
20
      "text": "12th of July 1987",
21
      "entity_type": "dob",
22
      "start_char": 41,
23
      "end_char": 58
24
    },
25
    {
26
      "text": "4242-4242-4242-4242",
27
      "entity_type": "credit_card",
28
      "start_char": 89,
29
      "end_char": 108
30
    }
31
  ]
32
}

The result shows the PII entities detected in the transcript, with their exact timestamps.

Integrating entity detection

Entity detection is integrated into the Speech to Text API by passing the entity_detection parameter to the convert method.

1
import os
2
from dotenv import load_dotenv
3
from io import BytesIO
4
import requests
5
from elevenlabs.client import ElevenLabs
6
7
load_dotenv()
8
9
elevenlabs = ElevenLabs(
10
    api_key=os.getenv("ELEVENLABS_API_KEY"),
11
)
12
13
audio_url = (
14
    "https://storage.googleapis.com/eleven-public-cdn/documentation_assets/audio/stt-entity-detection-pii.mp3"
15
)
16
response = requests.get(audio_url)
17
audio_data = BytesIO(response.content)
18
19
transcription = elevenlabs.speech_to_text.convert(
20
    file=audio_data,
21
    model_id="scribe_v2", # Model to use
22
    # Entity types to detect, accepts a list of specific entity types or categories.
23
    # To detect all entity types, use the "all" category.
24
    entity_detection=["pii"],
25
)
26
27
print("Entities detected:", transcription.entities)

Entity types

The following entity types are supported for detection. You can detect entire groups using the category keywords pii, phi, pci, other, or offensive_language, or specify individual entity types by their label. To detect all entity types, use the all category.

Entity redaction

In addition to detecting entities, you can redact them from the transcript text with the entity_redaction parameter. It accepts the same format as entity_detection: all, a category keyword, or a list of specific entity types. The redacted types must be a subset of the detected types.

When redaction is enabled, matched entity text is replaced with a marker in both the transcript text and the word-level output, and the entities field is not returned in the response.

1
transcription = elevenlabs.speech_to_text.convert(
2
    file=audio_file,
3
    model_id="scribe_v2",
4
    entity_detection=["pii"],
5
    entity_redaction=["pii"],
6
    entity_redaction_mode="enumerated_entity_type",
7
)
8
9
print(transcription.text)
10
# "Hi {NAME_0}, your card ending in {CREDIT_CARD_0} was declined."

The entity_redaction_mode parameter controls the marker format:

Next steps