Working paper / 2026
Multi-condition Impairment Speech-to-Text Bench
Abstract
Multi-condition Impairment Speech-to-Text Bench (MISTT) measures the speech-to-text models that voice agents depend on. Every model hears the same 2,999 frozen clips of voice-agent calls, doctor–patient consultations and five-language parliamentary speech, plus 400 degraded copies of 100 of them: phone lines, low-bitrate codecs, packet loss, background noise and an intelligible background talker. Streaming models are timed the way an agent waits on them, from the forced end of speech to the final transcript. Accuracy carries bootstrap confidence intervals, and the degraded copies are paired with their clean originals, so robustness is measured on the same words.
v0 · October 2026
AssemblyAI · Streaming · #7 of 49 streaming by English WER · assemblyai-stt-3-7-realtime-agent-stream
English WER
3.97%
95% CI 3.63% to 4.31%
TTFS p50 / p95
101 ms
p95 227 ms
$ / 1k audio min
$7.40
list $0.0075/min
Δ Severe
+2.6 pts
2.36% → 4.97% on 100 clips
| English index | 3.97% | 3.63% – 4.31% |
| pipecat (voice-agent turns) | 1.69% | 1.49% – 1.92% |
| primock57 (doctor–patient) | 6.24% | 5.57% – 6.95% |
| Multilingual (de/fr/es/it pooled) | 12.07% | 11.00% – 13.18% |
| VoxPopuli English | 6.62% | 5.67% – 7.56% |
| VoxPopuli German | 10.32% | 8.69% – 12.01% |
| VoxPopuli French | 11.00% | 9.06% – 13.08% |
| VoxPopuli Spanish | 7.78% | 6.22% – 9.51% |
| VoxPopuli Italian | 18.78% | 16.18% – 21.52% |
English substitutions 1.49%, deletions 1.88%, insertions 0.60% · raw (un-normalized) English WER 17.81% · hallucination 2.54% of clips · empty 0.03% · VoxPopuli CER 8.36%
| p50 | p90 | p95 | p99 | max | |
|---|---|---|---|---|---|
| TTFS | 101 ms | 189 ms | 227 ms | 315 ms | 816 ms |
| TTFT | 213 ms | 410 ms | 427 ms | 664 ms | 5.96 s |
| TTFB | 464 ms | 1.36 s | 1.76 s | 3.17 s | 6.46 s |
| Connection setup | 57 ms | 67 ms | 77 ms | 174 ms | 334 ms |
English WER@500 ms 3.97% (final 3.97%) · finals before t0 0.0% · fallback finals 0.0% · partial words later rewritten 15.4%
| Mix | WER | Δ pts (95% CI) | Worse | Rank |
|---|---|---|---|---|
| Clean | 2.36% | — | — | #19 |
| Light | 2.48% | +0.1 (−0.3 to +0.7) | 7/100 | #14 |
| Moderate | 4.17% | +1.8 (+1.0 to +2.9) | 25/100 | #11 |
| Severe | 4.97% | +2.6 (+1.4 to +3.9) | 30/100 | #7 |
| Crosstalk | 5.05% | +2.7 (+1.4 to +4.0) | 31/100 | #3 |
Bleed 0.47% of background-talker words · ranks are out of 109 on the subset.