AirCaps Diarization-v1: who spoke when, in real-world audio

Diarization depends on what the reference labels call speech. Original meeting annotations often include short pauses within an utterance. Word-aligned labels tend to mark those pauses as silence. A model that spans a pause can be correct under the first convention and produce a false alarm under the second; a model that removes it can instead accumulate missed speech against the original labels.

Our evaluation follows original dataset annotations, including the AMI and AliMeeting convention used by DiariZen’s recipes. NVIDIA uses word-aligned labels for AMI, AliMeeting and NOTSOFAR. VoiceArena also uses alignment and human review, with silence boundaries above 200 ms. This makes comparisons across those evaluations sensitive to label choices.

For example, Nemotron 3 Diarization scores 27.66% DER on our AMI references, including 24.92% missed speech. That gap should not be read entirely as a difference in speaker recognition. DIHARD III uses the same reference release as NVIDIA’s evaluation, and its labels are tight: most short pauses are marked as silence, much as in word-aligned labels. That convention favors Nemotron, which leads there, 12.72% versus 14.41% for AirCaps Diarization-v1. About half of that gap is false alarm in pauses shorter than one second, where AirCaps Diarization-v1 keeps the speaker active; most of the rest is speaker confusion. AirCaps Diarization-v1 has not been evaluated on VoiceArena.

Our official NOTSOFAR references also differ from the CHiME-8 release used by DiariZen’s authors, which omits filler-only utterances. Their published 16.7% and our measured 19.31% therefore describe different references.

Each dataset’s DER is total missed speech, false alarm and speaker-confusion time divided by total reference speaker-time. Overlap is counted, and the collar is zero: no boundary tolerance is removed from scoring. The primary average gives each of the seven datasets equal weight rather than weighting by recording count or duration.

The primary evaluation contains 538 recordings. CALLHOME uses 20 English Part 2 conversations; this differs from NVIDIA’s multilingual evaluation and its 0.25-second collar. VoxConverse and MSDWild add 722 recordings and are shown separately. Our runs use single-channel audio without reference speech boundaries or known speaker counts.

We ran AirCaps Diarization-v1, DiariZen, Nemotron and the four ASR providers. We also ran 3D-Speaker on NOTSOFAR-1, RAMC, DIHARD III, CALLHOME and MSDWild, and both pyannote pipelines on NOTSOFAR-1. Other baseline cells are authors’ published results. Averages containing those cells may combine different dataset splits or scoring details.

Scribe, AssemblyAI, Meta and Grok predictions use one fixed post-processing rule: merge overlapping segments from the same speaker, then bridge gaps up to 1.0 second only when no other predicted speaker is active in the gap. These are post-processed scores rather than native provider scores. The rule brings word-timed outputs closer to the original annotations.

Meta processed all 1,260 recordings whole, without splitting. Nemotron 3 Diarization uses the offline configuration with a 30.4-second input buffer, 16 kHz audio, BF16 and batch size 8. Its buffer settings are speaker cache 264, FIFO 40, chunk 340, right context 40 and cache update period 300, in 80 ms frames. 3D-Speaker uses its standard pipeline with overlap enabled; Community-1 uses its regular overlapping-speaker output.

Nemotron’s VoxConverse score is excluded because NVIDIA discloses training on VoxConverse’s test set in its model card. AssemblyAI is excluded because its target-model coverage is incomplete on both additional datasets; responses from fallback models are not included.

Grok’s MSDWild score is excluded because 11 recordings returned empty outputs. Its one remaining empty DIHARD response is counted as no speech, so all reference speech in that recording contributes to missed speech. Its DIHARD score and primary average include that penalty.