Speaker Diarization Benchmarks: Plaud Transcribe and Other Top Models

AI products that capture conversations require models that don’t just transcribe (going from speech to text), but also diarize speakers (attributing who said what). LLMs can recover from an errant word or two, but mislabeled speakers can confuse your agents from understanding conversation context.

Speaker diarization is one of the most important features for transcription models and APIs. Most of the most popular services offer a field in their API to include these speaker labels.

curl -X POST ai/transcriptions \
  -d '{
    "params": {
      "diarization": {
        "enabled": true
      }
    }
  }'

{
  "segments": [
    {
      "start": 0,
      "end": 4.2,
      "text": "Hey how is everyone",
      "speaker": "Speaker 1"
    },
    {
      "start": 5,
      "end": 10.2,
      "text": "Pretty good pretty good",
      "speaker": "Speaker 2"
    }
  ]
}

Transcription services from Plaud, AWS, and Assembly all offer this type of transcription to feed AI-driven products. But what makes each of them different? In this post, we’ll cover:

1. How speaker diarization is measured and benchmarked
2. How different models from Plaud, AWS, Assembly, etc. compare
3. How you can compare diarization models for yourself

Measuring Speaker Diarization

Core Metrics

There are 3 metrics we use at Plaud for our speaker diarization benchmarks:

1. Diarization Error Rate (DER): the proportion of total audio time that is missed, falsely detected as speech, or attributed to the wrong speaker. This is the industry standard; however, it’s confounded by Voice Activity Detection (VAD) errors and overlap errors.
2. Confusion Time Rate (CTR): the proportion of speech time that is wrongly attributed to a speaker with VAD & overlap errors EXCLUDED. This is measuring the error rate of strictly speech time.
3. Word Confusion Rate (WCR): the proportion of incorrectly attributed words to a speaker.

The first two metrics, Diarization Error Rate and Confusion Time Rate, are measurements of incorrectly attributed time, while the third metric is a measurement of text. For the purposes of our benchmarks, we will be presenting Confusion Time Rate (cleaned for VAD & overlap errors that can confound the metric) and Word Confusion Rate to have one measurement for time and text.

When you’re building agents, Word Confusion Rate may be the most important metric. The transcription text is ultimately what is fed to AI systems and systems of record. So the lower the Word Confusion Rate, the more accurate speaker labels are when given as context into AI systems. Imagine if you were building a healthtech platform, recording doctor to patient interactions. Would you want 10% of your transcriptions to be incorrectly attributed to the doctor vs patient?

Test Sets

The data sets for measuring speaker diarization need to be audio files with multiple speakers. Like any benchmark, more data and diversity will lead to more accurate representations of the real world.

Open-source datasets like Mozilla’s Common Voice and Google’s FLEURS are great datasets for transcription, but for diarization, the Plaud team opted to leverage real-world data contributed from Plaud users. These included interviews, meetings, lectures and calls. This gave us a test set that is more representative of real-world conversations rather than scripted readings.

For each recording, we manually annotated the transcript with the correct speakers. We then calculate Confusion Time Rate and Word Confusion Rate for each of these audio files against each of the following transcription models:

1. Plaud-Transcribe-1.0
2. Azure Fast-Transcribe
3. Byteplus ASR-2.0
4. Deepgram Nova3
5. AssemblyAI Universal
6. ElevenLabs Scribe V2
7. AWS Transcribe

Example

Here is a 26-second stretch of a three-speaker conversation from the English test set. Switch between models to see the same audio diarized by each one, against the human-annotated reference.

Diarization Benchmark Results

When it comes to our core speaker diarization metrics measured against our test sets, Plaud-Transcribe-1.0 is the best performing model on our test set across Word Confusion Rate and Confusion Time Rate. We'll look at results across languages below, but Plaud-Transcribe-1.0 is the only transcription model that ranks consitently high scross languages.

English Dataset

Among the top diarization models for English, Azure Fast-Transcribe and ElevenLabs Scribe V2 also see Word Confusion Rates below 2%, joinging Plaud-Transcribe-1.0.

Word Confusion Rate is generally more important for transcripts and context for agents, but confusion time rate is also a good measurement if you're labeling audio files or using timestamp data.

Plaud-Transcribe-1.0 performed similarly well in the other languages we tested against (Chinese and Japanese). Models like Byteplus ASR-2.0 jump when diarizing Chinese, while models like AWS Trancribe have lower variation across languages.

Chinese Dataset

Japanese Dataset

Implications & Further Exploration

Considering the results above, we want to acknowledge that Plaud’s models are trained off real-world conversation data (meetings, lectures, interviews, etc.), the same type of data that’s in our test set. This gives Plaud a distinct advantage in this test set, as we’re testing the same multi-speaker, noisy, real-world data that we built Plaud-Transcribe-1.0 distinctly for.

While we’d like to extrapolate and claim that Plaud’s transcription models have the most accurate speaker diarization in the market, we know that your conversation environment may be unique and unlike our test set.

You can try these transcription models for yourself and test your audio data in our Transcription Playground. No signup required. Just record or upload audio and compare the diarization for different transcription services.

Wrapping Up

Speaker diarization performance is one of the top aspects you should look at when evaluating transcription services, especially if you’re building AI products.

If you’re looking for one of the best performing models, trained on real-world offline data, Plaud Embedded gives you access to all of our transcription models, as well as dedicated Plaud devices that you can integrate with your AI product. Our customers have found that having a wearable or small recording device improved customer adoption, user experience, and had the best recording quality for real-world, in-person conversations.

Sign up for a Plaud Developer account to explore how you can start integrating your AI product with Plaud. The Plaud team wants to help you build the best AI solutions for your users, everything from best-in-class recording hardware to real-world-trained transcription software.