Working paper / 2026

MISTT

Multi-condition Impairment Speech-to-Text Bench

Abstract

Multi-condition Impairment Speech-to-Text Bench (MISTT) measures the speech-to-text models that voice agents depend on. Every model hears the same 2,999 frozen clips of voice-agent calls, doctor–patient consultations and five-language parliamentary speech, plus 400 degraded copies of 100 of them: phone lines, low-bitrate codecs, packet loss, background noise and an intelligible background talker. Streaming models are timed the way an agent waits on them, from the forced end of speech to the final transcript. Accuracy carries bootstrap confidence intervals, and the degraded copies are paired with their clean originals, so robustness is measured on the same words.

Leaderboard ↗Read the white paper

v0 · October 2026

Leaderboard

81 speech-to-text models, 45 batch and 36 streaming, each run once over the same 2,999 clean clips and 400 degraded copies. The headline is English WER: an equal-weight mix of voice-agent caller turns and doctor–patient turns, fixed before any result was seen. Filter to batch or streaming to compare like with like: they answer different questions, how accurate and how fast once the caller stops talking. Models whose interval overlaps the leader's are shaded, and we don't rank inside that band.

Mode

Weights

Applies to every chart and table below

Bluejay MISTT Bench: English WER

batchstreaming95% CI · lower is better

bluejay-labs

Scroll for more →+20 models evaluated, not shown

Frontiers

Two metrics at a time

The dashed staircase joins the models nothing else beats on both axes. Only frontier models are labelled; hover any dot for its numbers.

Streaming latency versus accuracy

streamingTTFS p50 · English WER

Cost versus accuracy

batchstreaming$/1k min · English WER

Clean versus Severe

batchstreamingsame 100 clips · WER

Latency versus accuracy

6 of 36 models sit on the frontier: Universal Streaming English (5 ms, 8.7%), Ink-2 (semantic endpoint) (71 ms, 4.1%), Ink Preview (72 ms, 4.0%), Ink-2 (73 ms, 4.0%), U3.5 Pro Streaming (max accuracy) (173 ms, 3.9%), Amazon Transcribe Streaming (555 ms, 3.8%). Streaming only: batch latency measures something else. The fastest point is co-located with our us-west-2 worker, so part of its lead is network distance.

Cost versus accuracy

2 of 81 models sit on the frontier: Nemotron 3.5 Transcribe 30B-A4B ($0.00, 3.5%), Pulse Pro ($4.00, 3.4%). A free trial endpoint sits at the left edge; token-priced models are estimates from measured token usage.

Clean versus Severe

2 of 80 models sit on the frontier: Scribe v2 (1.7%, 5.4%), Zoom Scribe EN Pro (1.9%, 3.9%). Points far above the others lost more accuracy to the phone line, packet loss and 10 dB noise than their clean score suggests.

Table

Every metric, one row per model

Click a column to sort. Hover a header for its definition, a cell for its interval.

Batch and streaming · 81 models

Vendor

#
1	Pulse Pro Smallest.ai · batchEN	3.38%±0.28	1.82%	4.93%	—	358 ms	$4.00	+5.1	+3.7	0.7%
2	Amazon Transcribe Amazon · batch	3.43%±0.30	2.09%	4.77%	10.56%	7.99 s	$6.35	+2.8	+9.9	3.3%
3	Nemotron 3.5 Transcribe 30B-A4B NVIDIA · batch	3.54%±0.32	1.85%	5.23%	11.79%	2.74 s	$0^*	+5.8	+10.4	1.9%
4	Scribe v2 ElevenLabs · batch	3.60%±0.35	1.86%	5.34%	12.17%	599 ms	$3.67	+3.7	+60.6	24.3%
5	Zoom Scribe EN Pro Zoom · batchEN	3.63%±0.32	1.95%	5.31%	—	1.09 s	$3.33	+2.0	+4.4	1.7%
6	Solaria-3 Gladia · batch	3.73%±0.32	1.90%	5.57%	13.16%	2.59 s	$2.99	+7.9	+7.3	1.3%
7	Universal-3.5 Pro AssemblyAI · batch	3.78%±0.31	1.78%	5.77%	11.88%	2.24 s	$3.50	+2.8	+7.7	2.2%
8	Gemini 3.1 Pro (high) Google · batch	3.79%±0.33	2.48%	5.10%	13.60%	4.90 s	$35.8^~	+3.0	+18.3	6.0%
9	Gemini 3.1 Pro (low) Google · batch	3.81%±0.36	2.41%	5.20%	13.45%	3.61 s	$31.0^~	+3.9	+13.1	5.1%
10	Amazon Transcribe Streaming Amazon · streaming†	3.81%±0.33	2.40%	5.22%	12.39%	555 ms^†	$11.4	+6.4	+4.2	2.2%
11	Step-Audio 3 ASR Max StepFun · batchen/fr/es	3.81%±0.34	1.71%	5.92%	10.73%	941 ms	$6.67	+6.3	+9.8	1.7%
12	Gemini 3.1 Flash-Lite (minimal) Google · batch	3.89%±0.38	1.52%	6.27%	13.37%	1.18 s	$1.06^~	+6.7	runaway	1.8%
13	U3.5 Pro Streaming (max accuracy) AssemblyAI · streaming	3.93%±0.35	1.86%	5.99%	12.33%	173 ms	$7.46	+3.6	+3.5	1.1%
14	Saaras v4 Sarvam · streamingEN	3.94%±0.31	1.74%	6.15%	—	344 ms	$6.52	+5.0	+3.9	0.8%
15	Qwen3 ASR Flash Alibaba · batch	3.98%±0.33	1.71%	6.25%	10.74%	913 ms	$2.22	+5.2	+8.3	1.8%
16	Ink-2 Cartesia · streamingen/fr/es	4.02%±0.34	2.12%	5.93%	9.79%	73 ms	$7.04	+5.7	+7.3	3.4%
17	Ink Preview Cartesia · streamingen/fr/es	4.03%±0.35	2.11%	5.94%	9.78%	72 ms	$7.04	+5.8	+7.3	3.4%
18	Gemini 3.5 Transcribe Google · batch	4.03%±0.36	1.49%	6.58%	12.18%	739 ms	$5.22^~	+6.3	+7.2	0.6%
19	Universal-3 Pro AssemblyAI · batch	4.04%±0.31	1.73%	6.34%	12.18%	2.22 s	$3.50	+3.6	+11.7	3.4%
20	Qwen3 ASR Flash Realtime Alibaba · streaming	4.05%±0.30	1.81%	6.28%	16.56%	363 ms	$6.17	+7.0	+7.5	1.1%

1–20 of 81

1 / 5

WER is pooled within a dataset (total errors ÷ total reference words) after Whisper's English normalizer plus two spelling equivalences (ok/okay, alright/all right) applied to every model. Intervals are 1,000-sample bootstraps, resampled by consultation for primock57. See methodology.
† Amazon Transcribe Streaming and Gladia Solaria-1 Live have no flush message. Their TTFS is the first natural final within 2 s of speech end (Artificial Analysis's fallback), so it is not directly comparable with forced-endpoint models.
EN marks English-only models (no multilingual score). Cartesia Ink and StepFun Step-Audio support English, French and Spanish of our five languages and are scored on those.
$/1k min is the effective cost of this run at each vendor's pay-as-you-go rate, including billing increments, minimums and the 1 s silence tail every stream receives. ~ marks token-priced models (Gemini, Qwen Omni, GPT-4o Transcribe), estimated from measured token usage. * marks NVIDIA's free, rate-limited hosted endpoints, shown at $0.
Δ Severe and Δ Crosstalk are paired: the same 100 clips clean and degraded, in percentage points. Bleed is the share of the background talker's words that appear in the transcript. Batch models hear the talker's 2 s tail after the caller stops; streams end 1 s after speech end, so batch and streaming crosstalk numbers are not strictly comparable. "Runaway" marks Gemini Flash-Lite rows where repetition loops on a few clips push the pooled WER past 100%. See methodology.
All requests ran from one AWS us-west-2 host. Vendors hosted in or near us-west-2 get a network advantage; AssemblyAI Universal Streaming English's ~5 ms TTFS is real but co-located.
Not yet ranked: 20 further configurations that did not complete every clip of both sets. They are listed on the models page with the reason.