Bluejay / Labs
Open navigation

Working paper / 2026

MISTT

Multi-condition Impairment Speech-to-Text Bench

Abstract

Multi-condition Impairment Speech-to-Text Bench (MISTT) measures the speech-to-text models that voice agents depend on. Every model hears the same 2,999 frozen clips of voice-agent calls, doctor–patient consultations and five-language parliamentary speech, plus 400 degraded copies of 100 of them: phone lines, low-bitrate codecs, packet loss, background noise and an intelligible background talker. Streaming models are timed the way an agent waits on them, from the forced end of speech to the final transcript. Accuracy carries bootstrap confidence intervals, and the degraded copies are paired with their clean originals, so robustness is measured on the same words.

Leaderboard ↗Read the white paper

v0 · October 2026

Clinical consultations

PriMock57

babylonhealth/primock57 · Babylon Health

Clips in MISTT

1,000

Hours

1.47

Reference words

15,376

Consultations

57

Overview

PriMock57 is a corpus of 57 mock primary-care consultations recorded by Babylon Health and published at ACL 2022. Seven clinicians consulted 57 colleagues acting as patients, who worked from case cards; the conversation itself is unscripted role-play.

Clinicians spoke from laptops and patients from mobile phones in an office environment with background noise, over telemedicine software, so the source audio is already compressed. Doctor and patient were recorded on separate channels and transcribed manually by transcribers experienced in clinical conversation.

Why it was chosen

  • Cost of errorClinical speech is where a single misrecognized word, a drug name or a dose, changes the outcome. It tests the vocabulary that general benchmarks rarely contain.
  • Conversational registerTurns are short, interactive and disfluent, with hesitations, repairs and backchannels, the opposite of read speech.
  • Human referencesManual transcription by domain-experienced transcribers provides a reliable reference.
  • LicenceCC BY 4.0 permits publishing results and audio samples with attribution.

What it provides

In MISTT, primock57 contributes half of the English WER index and half of the paired robustness subset. Every evaluated model has a higher error rate here than on pipecat, and the most accurate models differ between the two, so it measures something pipecat does not.

Speech
two-party clinical dialogue, unscripted role-play
Speakers
7 clinicians (5 British English, 2 Indian accents) and 57 patients (47% British English, 32% European, 21% other)
Channel
laptop (clinician) and mobile phone (patient) over telemedicine software; Opus-compressed
Duration
8 h 38 min of consultations, 3 min 48 s to 14 min 18 s each
References
manual, segmented per utterance, Praat TextGrid
Example clip · primock57-907d391ea7f5 · 7.1 s

“Wheezing and coughing, OK. And you've mentioned you're using the blue inhaler. So, how often are you using the blue inhaler?”

The same clip under four channel perturbations, with model transcripts, is in the methodology.

Known limitations

  • Consultations are role-played, not real patient encounters.
  • 57 consultations is a small number of independent units, so MISTT resamples by consultation and its intervals are wider.
  • Audio was compressed by the recording software, so the clean condition is not studio quality.

Format

Distributed on GitHub (audio via Git LFS) as separate doctor and patient WAV files per consultation, with Praat TextGrid transcripts.

Native format

Audio
WAV, 16-bit, 16 kHz; one file per channel: dayN_consultationNN_{doctor,patient}.wav
Transcripts
Praat TextGrid, one interval tier per channel (xmin, xmax, text)
Tags
<UNSURE>…</UNSURE>, <UNIN/> (unintelligible), <INAUDIBLE_SPEECH/>
Splits
none; also includes clinician notes and evaluation scripts

As sampled into MISTT

Selection
1,000 single-speaker utterances across all 57 consultations, 2–15 s, ≥ 3 words
Tags
utterances with any self-closing tag dropped; <UNSURE> markup stripped, words kept
Processing
16 kHz mono PCM16, RMS −20 dBFS, Silero VAD speech bounds
Independence
intervals resample whole consultations

Manifest row (clips-v0.jsonl)

{
  "clip_id": "primock57-00031c6cf8ae",
  "dataset": "primock57",
  "source_id": "[email protected]",
  "conversation_id": "day1_consultation02",
  "speaker": "patient",
  "language": "en",
  "duration_s": 8.382,
  "reference": "Um I mean, at that time yes. That's, those symptoms, like these symptoms appear like, when the symptoms appeared again, I tried those and it didn't work.",
  "speech_start_ms": 352,
  "speech_end_ms": 8382,
  "license": "CC-BY-4.0"
}

Results on this dataset

81 models scored; median WER 6.91%. The ten lowest are listed. Intervals and the full ranking are on the leaderboard.

  1. 1Amazon TranscribeAmazon · batch4.77%
  2. 2Pulse ProSmallest.ai · batch4.93%
  3. 3Gemini 3.1 Pro (high)Google · batch5.10%
  4. 4Gemini 3.1 Pro (low)Google · batch5.20%
  5. 5Amazon Transcribe StreamingAmazon · streaming5.22%
  6. 6Nemotron 3.5 Transcribe 30B-A4BNVIDIA · batch5.23%
  7. 7Zoom Scribe EN ProZoom · batch5.31%
  8. 8Scribe v2ElevenLabs · batch5.34%
  9. 9Solaria-3Gladia · batch5.57%
  10. 10PulseSmallest.ai · batch5.72%

Source and attribution

GitHub
github.com/babylonhealth/primock57
Paper (ACL 2022)
aclanthology.org/2022.acl-short.65/
arXiv
arxiv.org/abs/2204.00333
Pinned revision
cd2ac707
Licence
CC BY 4.0
Attribution
PriMock57 by Babylon Health (Papadopoulos Korfiatis et al., 2022), CC BY 4.0.

Citation

@inproceedings{papadopoulos-korfiatis-etal-2022-primock57,
  title     = {PriMock57: A Dataset Of Primary Care Mock Consultations},
  author    = {Papadopoulos Korfiatis, Alex and Moramarco, Francesco and Sarac, Radmila and Savkov, Aleksandar},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  pages     = {588--598},
  year      = {2022},
  address   = {Dublin, Ireland},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2022.acl-short.65}
}