Speaker Diarization Accuracy in Multi-Participant Meetings

Speaker diarization is the "who spoke when" problem: take a recording of several people talking and cut it into segments labeled by speaker. It sounds like a solved problem by now. It isn't, and the reason has less to do with any single vendor's engineering than with math that gets worse every time another person joins the call, or two people start talking at once.

Diarization is not transcription. Automatic speech recognition answers "what was said." Diarization answers "who said it, and when." Meeting tools need both, but the two are solved by different mechanisms, and diarization is the harder one to get right as speaker count and overlap grow. Researchers score it with Diarization Error Rate, or DER: add up speaker confusion time, false alarm time, and missed detection time, then divide by the total duration of the recording. That's a different number from word accuracy, and it's the one that matters when a wrong speaker label ends up attached to a decision or a commitment in a meeting summary. Get the speaker wrong on an action item, and the mistake doesn't stay contained. It flows into a CRM record, into a summary someone reads three weeks later, into what a team believes actually happened in the room.

Set expectations accordingly: live, real-time diarization runs behind what a system can do when it processes the same audio after the fact, and conditions (speaker count, mic setup, how much people talk over each other) swing the result more than any vendor's marketing page suggests. Three things decide any headline accuracy number: how many speakers were in the room, what the audio setup looked like, and how much of the conversation involved people talking over each other.

How multi-speaker conditions compound the error rate

The ICASSP 2022 M2MeT Challenge is a useful anchor. The winning system posted a DER of 2.98%, against an official baseline of 15.60%. A gap that size, over five times, is the distance between a naive system and one built with serious engineering effort behind it. It shows the ceiling is reachable. It also shows almost nothing gets there by default.

The MISP 2025 Challenge, hosted at Interspeech 2025, had its top audio-visual system reach a DER of 8.09%, a 7.43% improvement over baseline. That's with video added as a second modality, giving the system lip movement and visual cues on top of audio, and the best system in the world still couldn't drop below single digits. Video assistance narrows the gap. It doesn't close it.

Neither number reflects a real meeting. Both challenges run on controlled corpora, recorded and structured for research. The M2MeT AliMeeting dataset capped sessions at 2 to 4 speakers, and more speakers means more opportunities for the system to make attribution errors. More speakers means more embedding collisions, where the system's internal representations of different voices start to look alike, and more boundary errors, where it can't cleanly tell where one person stopped and another started.

The MISP-Meeting dataset pushes further: sessions of 4 to 8 participants, an overlap ratio of 56.95% across the corpus. Well over half the audio involves more than one person talking at once. That's closer to what an executive meeting or a busy stand-up actually sounds like than most lab conditions get. Treat the research numbers as a floor. Production environments, with worse mics, more participants, and less discipline about taking turns, routinely land below it.

Overlapping speech: why it is the dominant source of diarization error

Most diarization systems assume one speaker per frame: one voice attributed to each moment in time. Real conversation breaks that assumption immediately. When two people talk at once, the system has three options, none of them good. Pick one speaker and ignore the other, which is a confusion error. Flag the segment as ambiguous and drop it, which is a missed detection. Or misassign the whole stretch of audio to the wrong person.

Some systems dodge this by suppressing overlapped audio outright, which keeps the speaker labels clean but silently deletes the words spoken during that overlap. Suppressing overlapped audio outright is a separate failure: it keeps the speaker labels clean but silently deletes the words spoken during that overlap. It won't appear as a DER penalty, since DER measures speaker attribution rather than word content, but it corrupts the meeting record just as badly. Someone reading the transcript later has no way of knowing a chunk of the conversation is simply gone.

Short speaker turns make the problem worse. Rapid back-and-forth, the kind that happens in a real argument over a decision, produces very short segments for the system to work with, and shorter segments mean less acoustic signal per turn. The clustering step, which groups segments by likely speaker, has less to go on and makes more mistakes. A 56.95% overlap ratio, as in the MISP-Meeting corpus, is closer to how people actually talk in unstructured conversation than most controlled lab conditions reflect. And the moments most likely to involve overlap, interruptions, quick agreement, someone jumping in to correct a number, are often exactly the moments carrying the most decision-relevant content. The failure hits hardest precisely where the stakes are highest.

Audio quality and environment as multipliers on every other error source

Background noise hurts diarization two ways at once. It masks the acoustic features the system uses to tell voices apart, and it introduces stray sounds, a chair creak, an appliance hum, someone typing, that the system can mistake for an additional speaker. Research published in Expert Systems flagged low speech-to-noise ratio, alongside short audio duration, as one of the most damaging conditions for diarization performance.

Far-field microphones, the kind built into conference room arrays sitting several feet from whoever is talking, weaken the signal and add reverberation. The AliMeeting corpus, used in the M2MeT challenge, covered microphone-to-speaker distances from 0.3 to 5.0 meters to match how a real conference room sounds to a mic mounted on a table or wall. There's a domain mismatch problem too: models trained heavily on broadcast news or phone call audio don't transfer cleanly to conversational meeting speech, because the pacing, register, and background conditions differ enough to matter.

The practical hierarchy is not complicated. A headset or close mic beats a room array, every time. A single decent microphone near the people talking beats a laptop mic at the far end of a long table. None of this stacks neatly, either. Poor audio plus a large speaker count plus a high overlap ratio doesn't just add three problems together; each one makes the others harder to recover from. Noisy audio makes overlap detection less reliable, and a crowded room makes noise harder to filter, since there's more simultaneous signal competing for the same channel.

Why overlap remains an unsolved problem for current systems

The main architectural answer to overlap is Overlapped Speech Detection, or OSD: find the overlapping segments first, isolate them, then run the clustering step that assigns speaker labels. pyannote.audio 3.0, released in 2023, added an OSD module that isolates overlapped segments before the diarizer runs. That's a real, practical improvement for systems built on that pipeline.

The frontier research is headed somewhere more ambitious: unified, end-to-end Speech-LLM systems that treat multi-speaker transcription as a structured generation problem instead of a two-stage pipeline. The DM-ASR paper (arXiv, 2026) names several systems working this direction, including SpeakerLM, JEDIS-LLM, MOSS Transcribe Diarize, TagSpeech, and VIBEVOICE-ASR. These systems use long-range conversational context better than a cascaded pipeline can, and they decode more flexibly instead of being locked into rigid, stage-by-stage assumptions.

That same paper found timing modeling is still weakly represented in the literature relative to speaker identity. Speaker identity gets captured well. Precise timing lags behind it, and that gap matters specifically for meeting tools, because retrieval, action item timestamping, and any real evaluation of diarization quality depend on knowing not just who spoke but exactly when. A system that nails identity while blurring timing only solves half of what meeting software actually needs.

DM-ASR itself, from Wuhan University and Tencent (arXiv, April 2026), reframes multi-speaker ASR as multi-turn dialogue generation, separating speaker-and-timing structure from the words themselves, and adds word-level timestamp prediction as a capability competing unified systems lack.

Cascaded pipelines (diarize, then transcribe) stay modular and practical, but they propagate errors: get the diarization step wrong and the transcription inherits the mistake. Unified end-to-end systems are improving fast, but they're data-hungry and still maturing, and neither approach is close to finished. An 8.09% DER on a controlled research corpus, the best result at MISP 2025, is not the number a team gets on a noisy hybrid call with a bad conference phone sitting in the middle of the table.

What transcription accuracy numbers from AI meeting tool vendors reflect

Leading ASR models post strong word accuracy on clean audio, and those numbers drop once crosstalk, accents, and domain-specific jargon enter a real multi-speaker room. Hands-on comparisons across the major tools tend to land in similar transcription accuracy ranges for English. The brand a team picks matters less to raw transcription quality than the audio conditions that team actually records in.

Diarization is where products actually separate from each other. AssemblyAI's own published comparison shows its flagship model posting a diarization cpWER of 30.17, ahead of Deepgram Nova-3 EN at 37.92 and ElevenLabs Scribe v2 at 35.26. cpWER, concatenated minimum-permutation word error rate, penalizes speaker assignment mistakes directly, which makes it a far more honest stand-in for meeting note quality than a plain word-accuracy figure. A transcript can get every single word right and still be useless if half the sentences are attributed to the wrong person.

The meeting notetaker market has consolidated onto a small number of underlying speech-to-text engines, and two competing apps might be running the exact same transcription backend under the hood without either one advertising it. Differentiation increasingly happens above the transcription layer, in what the product does with the output: structured decisions, action items with named owners, fields formatted to drop straight into a CRM. That's where the real quality gap sits now, not in raw accuracy.

Vendor-published accuracy numbers are usually measured on clean, single-speaker audio, and that detail rarely gets disclosed clearly. A real multi-participant meeting, with overlap and background noise, produces a lower number that almost never makes it onto a marketing page. Treat a vendor's stated accuracy as a ceiling, not a floor. The floor gets set by everything above it: speaker count, overlap ratio, mic quality.

Features to look for in an AI meeting tool to handle these failure modes

Ask directly whether the tool uses overlapped speech detection to preserve both speakers' words, or whether it quietly suppresses the overlapped region instead. The answer says a lot about how mature the underlying architecture actually is.

Check how accuracy holds up as participant count grows. Tools built mainly on 2-speaker or telephony-style training data tend to degrade as a room gets crowded, and that degradation becomes visible only after it has already cost someone a wrong attribution.

Audio input flexibility affects whether the tool captures usable speech accurately, particularly with close-mic setups like headsets and desktop audio versus far-field room mics. Does the tool work well with close-mic setups like headsets and desktop audio, not only far-field room mics? Bot-based tools that join a call as a visible participant only get whatever audio the conferencing platform hands them, inheriting that platform's microphone limits. Botless tools that capture system audio directly on a device can get cleaner input in remote settings, though each approach carries its own tradeoffs depending on the meeting.

A diarization mistake that slips through shouldn't cascade into a wrongly attributed action item or a bad CRM field. Look for tools that treat speaker-attributed action items as a structured output with a name properly attached.

Language and accent coverage deserves scrutiny too. Models trained heavily on one language or accent cluster tend to degrade outside it, and for multilingual or globally distributed teams, that's a baseline requirement, not a nice-to-have.

Temporal grounding is the one most buyers skip evaluating. Can the tool timestamp not just who spoke but when a specific decision or commitment happened? The timing gap found in the DM-ASR research is visible in real products, and tools that surface searchable, time-linked moments from past meetings are ahead of ones that only spit out a flat summary.

No tool on the market eliminates diarization error, and the benchmark numbers make that plain without any need for interpretation. Aim lower, and more realistically: a tool whose failure modes are predictable, whose structured output can be trusted enough to act on, and whose audio capture strategy actually matches the kind of meetings a given team runs.

Practical steps to reduce diarization errors before and during meetings

Audio setup is the single highest-leverage fix available, full stop. A headset or a dedicated USB mic placed near each speaker produces far cleaner voice embeddings than a room mic or a laptop sitting at the end of a table, and this is where the most error reduction happens per dollar spent.

Cut ambient noise before it reaches the recording. Close the door, mute participants who aren't speaking, and turn on whatever noise suppression the operating system or conferencing platform offers before the signal ever hits the diarization engine.

Manage speaker count on purpose. Splitting a large group session into smaller breakouts doesn't just make the meeting itself more productive, it directly lowers DER, since the research consistently shows accuracy dropping once speaker count climbs past four.

Set a light facilitation norm: one person talking at a time, interruptions kept to a minimum. That alone cuts the overlap ratio, the single biggest driver of diarization error. The MISP corpus's 56.95% overlap figure reflects conditions far removed from orderly turn-taking, not an unavoidable fact of how meetings work.

Fix mistakes early rather than late. Most tools let someone edit speaker labels after the meeting ends, and correcting a misattributed speaker before notes get pushed into a CRM or shared with a team stops that error from becoming part of the organization's permanent record.

For in-person meetings, mic placement relative to the room shapes how accurately the system can separate speakers. The AliMeeting corpus, used in the M2MeT challenge, covered far-field conditions from 0.3 to 5.0 meters for exactly this reason: a device sitting in the middle of a large room faces a different problem entirely from a close-mic setup. Match the hardware to the room.

Speaker count, overlap, and audio quality are the same three variables researchers use to benchmark these systems. Understanding how they interact gives a team real leverage, both in choosing which tool to trust and in setting up the room to get the most out of whatever tool they've already chosen.