What Is Speaker Diarization? Who-Said-What, Explained

The short answer: Speaker diarization is the process of working out who is speaking, and when in a recording. It does not identify people by name — it separates voices and labels them consistently, so a conversation becomes a transcript you can read like a script.

Speaker diarization takes an audio recording and works out who is speaking, and when. The name comes from “diary”: the algorithm produces a timeline of the conversation. Speaker A from 0:00 to 0:12, Speaker B from 0:12 to 0:31, Speaker A again after that. Combined with speech-to-text, it turns a recording into what people actually want — a transcript where every line is attributed.

If you have seen the word written as diarisation, that is simply the British spelling. There is no difference in meaning, and both spellings refer to the same task.

Why plain transcription isn’t enough

A transcript without diarization is one long block of text. Every word might be correct, and the result can still be unusable:

so where did we land on the launch date marketing wants the 14th but engineering needs one more week then let’s lock the 21st and tell the team today

Who committed to the 21st? Was it the person with the authority to? You were in the meeting, so you might remember today. In two weeks, you won’t. Diarization is what makes the same words legible:

Speaker A: So where did we land on the launch date? Speaker B: Marketing wants the 14th, but engineering needs one more week. Speaker A: Then let’s lock the 21st and tell the team today.

Same audio, same words — but now it’s a record of a decision, with the decision-maker visible.

How diarization works, briefly

Modern systems typically do three things:

1. Detect speech. Find the parts of the recording where anyone is talking at all, discarding silence and noise.
2. Fingerprint the voice. Short slices of speech are converted into voice embeddings — numerical fingerprints that capture how a voice sounds (pitch, timbre, resonance) rather than what it says.
3. Cluster. Slices with similar fingerprints get grouped as the same speaker. The system doesn’t know the speakers’ names — it knows there are, say, three distinct voices, and it labels them consistently throughout.

Newer approaches fold these steps into a single neural model that transcribes and attributes at the same time, which handles fast back-and-forth exchanges better than pipeline systems.

Diarization, identification, separation, segmentation

These four terms get used interchangeably and mean genuinely different things. The distinction matters, because it determines what a tool can and cannot do for you.

Diarization separates voices without knowing whose they are. It can tell you there are three speakers and keep them consistently labelled, but it cannot tell you their names. No setup, no prior knowledge of the people in the room.

Speaker identification matches a voice to a specific known person. That requires enrolling a sample of that person’s voice beforehand, so the system has something to compare against. This is the technology behind “Hey Siri” recognising its owner. It is a different job with a different prerequisite.

Speaker separation — sometimes called source separation — splits overlapping voices into separate audio streams. It changes the audio itself, pulling two people talking at once into two cleaner tracks. Diarization only labels; separation actually unmixes.

Speaker segmentation is the narrower step of finding the boundaries where the speaker changes, without deciding which speaker each stretch belongs to. It is usually one stage inside a diarization pipeline rather than something you use on its own.

In practice, most transcription products you can buy do diarization. Very few do identification, because it requires you to enrol every participant first — which nobody wants to do before a meeting.

How accuracy is measured

The standard score is diarization error rate, or DER: the proportion of audio time that is attributed incorrectly. It sums three separate failures — speech given to the wrong speaker, speech the system missed altogether, and silence or noise mistaken for speech. Lower is better.

Two things about DER are worth knowing before you compare any vendor’s claims.

First, it is measured in time, not words. A transcript can have a very good word accuracy and a poor DER at the same time, because getting the words right and getting the attribution right are separate problems.

Second, DER is extremely sensitive to the recording. Published research figures generally come from curated datasets, and the same system will score far worse on a real meeting with four people, a hard table and a coffee machine. Treat any single accuracy number quoted without its conditions as marketing rather than measurement.

Why it’s genuinely hard

• Overlapping speech. People interrupt and talk over each other; overlapping voices blur both fingerprints.
• Similar voices. Two speakers with close pitch and accent produce embeddings that sit near each other.
• Short utterances. “Yes.” gives the model almost nothing to fingerprint.
• Distance and echo. A speaker far from the microphone sounds different from the same speaker up close.
• Unknown speaker count. The system usually has to work out how many people are present as well as who is who, and an over- or under-estimate cascades into every label.

This is why recording conditions matter: a phone placed centrally in a room gives a diarization model dramatically more to work with than one buried in a bag. The practical side of this — where to put the microphone, and what to expect from a real conversation — is in how to transcribe audio with multiple speakers.

Why the labels say “Speaker A” and not names

This surprises people, and it follows directly from what diarization is. The model has clustered voices by acoustic similarity; nothing in that process reveals identity. Names would require either an enrolled voice sample for each person or a human saying who is who.

In practice the fix is trivial: the speakers appear in a consistent order, so you rename Speaker A once and the whole transcript follows. What matters is that the separation is right — that is the part software cannot ask you to do by hand. The conventions for naming them, and how to fix labels that came back wrong, are covered in how to label speakers in a transcript.

What good diarization makes possible

• Meeting minutes with owners. Decisions and action items attach to people, not to “the room”. (Why this matters for meetings →)
• Quotable interviews. Journalists and researchers can pull attributed quotes without re-listening. (Interviews →)
• Searchable lectures. The lecturer’s thread separates cleanly from questions in the room. (Lectures →)

Do you need to set anything up?

With modern tools, no. In Record and Transcribe, diarization runs automatically on every memo as part of transcription — there are no voice profiles to train and nothing to configure. You record the conversation on iPhone or Apple Watch; the transcript comes back with every line labelled. (How our app identifies speakers →)

The honest limit is the one above: no product removes the effect of bad audio. If the recording is a phone in a bag across a noisy room, no amount of modelling recovers who said what, and anything claiming otherwise is overselling. Give it a centrally placed microphone and ordinary turn-taking, and the result is a recording you can read like a script — with everyone’s words as their own.