Audio deepfake detection: How it works & how to verify ElevenLabs audio

Audio deepfake detection is the process of analyzing voice audio to determine whether speech is authentic. AI voice synthesizers that generate deepfakes are more and more accessible. It’s become important both in public life and in business to quickly identify whether the speech you’re hearing is genuine or has been manipulated by a bad actor.

In this article, we’ll explore how audio deepfake detection works, the different types of detection, their strengths and weaknesses, and how you can integrate deepfake detection into large-scale fraud detection stacks.

Summary

• Deepfake synthesized speech is becoming harder to detect by ear alone. It poses a significant risk in both public life and business.
• Multiple types of deepfake detection software exist. Conventional tools use statistical analysis to assess the probability that an audio sample is AI-generated.
• New watermarking technology allows you to get a definitive yes or no answer as to whether a voice sample was synthesized.
• Watermarking is highly accurate, but detectors only work on a provider’s own audio. For example, ElevenLabs’ Audio Detector can identify watermarks generated by its own products with high accuracy but not those from other companies.
• Watermarking is an important resource for organizations that need large-scale deepfake detection, but it should be only one of many tools in their deepfake-detection stack.

What is audio deepfake detection?

Audio deepfake detection is the process of determining whether voice audio was generated or manipulated by artificial intelligence.

A deepfake is any form of media (image, video, or audio, most commonly) created by a generative AI application for the purpose of misrepresenting what someone says or does. Audio deepfakes have many malicious use cases. They can be used for fraud, to attempt to access personal accounts, as political propaganda, or to secretly impersonate a spouse or family member in domestic disputes.

Based on current trends, the victims of audio deepfake scams are most likely to be:

• Politicians & other public figures: Individuals with extensive voice samples available for training online might have their voice deepfaked by opponents or third parties to mislead the public.
• Family members: A bad actor may attempt to use found voice recordings to scam elderly family members with deepfaked requests for money.
• Corporate executives: Faked interviews, presentations, and earnings calls might be used to harm a company’s brand.
• Customer service teams: Scammers might use collected voice samples to scam customer service teams into granting private account access.

Unfortunately, humans are not reliable detectors of deepfakes. In controlled research settings, people’s ability to identify them is no better than chance, and that research is from back in 2024. Generative media has grown significantly more sophisticated since then.

Education and legislation will go a long way towards inoculating society against the influence of deepfakes, but reliable detection methods are still necessary.

How does audio deepfake detection work?

Conventional deepfake detection uses statistical analysis to measure the probability that media was generated by an AI. A detection tool uses machine learning techniques to recognize the underlying patterns in different types of synthesized speech.

The first step to doing this is building a library of audio content to compare suspected deepfakes against. This should be a diverse set of data, meaning it needs to include genuine audio samples of a variety of types and many known deepfake samples to help make positive detections.

At runtime, it follows a three-stage process: preprocessing, feature extraction, and classification.

Audio preprocessing

When you give a trained detection tool an audio sample, it converts it into spectrographic data, or a mathematical representation of frequency changes over time.

Feature extraction

Next, the detection tool extracts features from the sample data that are like fingerprints for synthesized speech. For example, AI speech is sometimes too perfect and lacks the tonal and timing irregularities found in natural speech. Comparing the speaker’s pitch and tone to its library will also help the tool identify other red flags.

Classification

Because conventional deepfake detection occurs through statistical analysis, it can’t give you a definitive “yes” or “no” answer to the question of whether a sample is a deepfake. Instead, it generates a confidence score, or probability, that the sample is a deepfake.

Because it measures similarities between a sample and an existing audio library, conventional tools are limited by the known deepfake samples they have available to them. Anytime someone releases a new voice-cloning architecture, you need to retrain your detection tool on speech synthesized in that new system to accurately detect it.

What’s the difference between watermarking and statistical detection?

Watermarking is an entirely different approach to speech classification. A machine learning classifier makes a probabilistic guess whether a sample is authentic based on what it knows from its training data about natural speech, like what real emotional weight and emphasis look like in spectrographic data, and what types of tonal and timing changes are natural.

Watermarking is deterministic. A watermark is a mathematical signature embedded within audio data by the synthesizer at levels a human listener can’t hear. A watermark detector verifies, yes or no, whether it finds that hidden signature.

A detector operates in three straightforward stages:

1. Extraction: The tool extracts spectrographic data and isolates the frequencies where it expects to find a watermark.
2. Key matching: It attempts to unlock the signature with a key.
3. Verification: If the signal decodes correctly, the tool confirms that the AI voice generator in question generated the audio.

ElevenLabs’ watermark detection process

ElevenLabs’ watermarking tool is Audio Detector. It scans audio samples and verifies whether the content was generated by an ElevenLabs product. If it doesn’t find a watermark, it then runs the statistical AI Speech Classifier to assess whether the audio is AI-generated from another source.

Because of how watermarking works, ElevenLabs Audio Detector only detects ElevenLabs-generated audio. It can’t currently identify watermarks of other AI voice providers.

And conversely, only ElevenLabs’ Audio Detector can detect ElevenLabs watermarks.

Also note that content generated by any third-party synthesizers available through ElevenLabs is generally not watermarked.

Other options for detecting ElevenLabs-generated audio

The ElevenCreative AI Speech Classifier is available free and without sign-in. It will analyze the first minute of an audio sample and provide a confidence score for how likely the audio was generated by an ElevenLabs model.

Note that it cannot reliably classify Eleven v3 audio. It is most effective in classifying audio generated by older ElevenLabs TTS models. Its accuracy will also drop when analyzing heavily compressed or edited speech content.

How can you detect synthetic voices at scale?

Some business teams need the ability to detect deepfakes at scale in near-real-time. Take the example of a national bank’s customer service team. Its representatives are going to be interacting with thousands of customers a day. Team members must regularly share and update private account data.

That team’s leadership cares a great deal about how fast they can both identify a fraudulent caller and respond to legitimate customers. They need a voice detection stack that can identify audio from all major voice-generation engines live during customer calls.

Building a full voice-detection stack

ElevenLabs’ Audio Detector plays an important role here, but it’s important to note again that it only detects ElevenLabs-watermarked audio. It’s not a complete voice-detection system on its own. Organizations that want full voice-detection capabilities need to assemble several different types of tooling:

• Audio watermarking detectors: For all major voice synthesis providers.
• Speech classifiers: One or more classification engines that can score audio data.
• Telephony and network telemetry: Analytics for call metadata, like phone number, caller ID, and call routing.
• Behavioral biometrics: Software to analyze speaker intent to score for risk of fraud.

Get started with ElevenLabs deepfake detection

ElevenLabs is committed to making a positive contribution to public life, both through meaningful voice generation services and by providing the public with the tools they need to identify deepfakes and other malicious uses of AI-generated audio.

Do you have an audio clip you want to test for deepfakes? Sign up to try the ElevenLabs Audio Detector or get started with the AI Speech Classifier today.

FAQ