Working paper / 2026
Multi-condition Impairment Speech-to-Text Bench
Abstract
Multi-condition Impairment Speech-to-Text Bench (MISTT) measures the speech-to-text models that voice agents depend on. Every model hears the same 2,999 frozen clips of voice-agent calls, doctor–patient consultations and five-language parliamentary speech, plus 400 degraded copies of 100 of them: phone lines, low-bitrate codecs, packet loss, background noise and an intelligible background talker. Streaming models are timed the way an agent waits on them, from the forced end of speech to the final transcript. Accuracy carries bootstrap confidence intervals, and the degraded copies are paired with their clean originals, so robustness is measured on the same words.
v0 · October 2026
NVIDIA · Batch · #29 of 45 batch by English WER · nvidia-parakeet-tdt-0-6b-v2-batch
English WER
5.16%
95% CI 4.82% to 5.50%
E2E p50
235 ms
p95 404 ms · RTFx 29.0×
$ / 1k audio min
$0
free trial tier
Δ Severe
+5.8 pts
4.01% → 9.86% on 100 clips
| English index | 5.16% | 4.82% – 5.50% |
| pipecat (voice-agent turns) | 3.41% | 3.07% – 3.76% |
| primock57 (doctor–patient) | 6.91% | 6.34% – 7.44% |
| Multilingual (de/fr/es/it pooled) | — | |
| VoxPopuli English | 6.26% | 5.24% – 7.23% |
English substitutions 2.61%, deletions 1.58%, insertions 0.97% · raw (un-normalized) English WER 18.72% · hallucination 0.32% of clips · empty 0.05% · VoxPopuli CER 3.77%
| p50 | p90 | p95 | p99 | max | |
|---|---|---|---|---|---|
| End-to-end | 235 ms | 344 ms | 404 ms | 612 ms | 1.71 s |
Speed factor (RTFx) 29.0×
| Mix | WER | Δ pts (95% CI) | Worse | Rank |
|---|---|---|---|---|
| Clean | 4.01% | — | — | #59 |
| Light | 3.77% | −0.2 (−1.0 to +0.6) | 13/100 | #43 |
| Moderate | 8.89% | +4.9 (+3.0 to +7.3) | 46/100 | #46 |
| Severe | 9.86% | +5.8 (+3.7 to +8.4) | 46/100 | #44 |
| Crosstalk | 24.72% | +20.7 (+15.6 to +26.0) | 70/100 | #66 |
Bleed 5.50% of background-talker words · ranks are out of 81 on the subset.
