MISTT Bench: Speech-to-text for voice agents under channel impairments

A voice agent is only as good as the words its speech-to-text model hands it. A wrong medication name, a dropped "not", or a background voice transcribed as the caller's request becomes a wrong action a moment later. Choosing an STT model is also a three-way trade between accuracy, latency and price, made harder by billing rules and by audio that is rarely clean: callers are on phone lines, on mobile codecs, in kitchens and cars, with other people talking nearby.

MISTT, the sibling of MIVAS, measures exactly that. The name stands for Multi-condition Impairment Speech-to-Text. We always say channel impairments, never "impaired speech", which in speech research means speech disorders.

On the audio a voice agent actually receives, which speech-to-text models are accurate, fast and affordable, and how much of that survives a bad line?

Four public benchmarks shaped the design. Each gets something right, and each leaves a gap that MISTT is built to fill.

• Artificial Analysis has the right protocol: one full pass per model version, matched by API id, with streaming timed from the end of speech. Its audio includes voice-agent speech, but it doesn't vary the channel, so it can't say what a phone line or a noisy room does to a model.
• The Hugging Face Open ASR Leaderboard covers the widest range of English datasets and several languages, scored with WER and batch throughput on fixed hardware, mostly for open models. It now includes URGENT24, which scores noisy and clean copies of the same speech; it has no streaming track.
• Speech Robust Bench is the closest in spirit: clean audio paired with distorted copies, including a controlled competing talker, scored as the change in WER. It tests open checkpoints rather than deployed APIs, and its distortions approximate a phone band by resampling but include no codecs or packet loss.
• CHiME supplies real far-field, multi-speaker audio, but as a research challenge for submitted systems, not a ranking of products.

Our lane is Artificial Analysis's protocol, plus conversational and medical audio, plus paired degradations that include an intelligible background talker, run against the commercial APIs voice agents actually call, with an interval on every number.

1. 01Frozen manifest. Every clip pinned by source revision and SHA-256, speech start and end precomputed once with Silero VAD. Every model hears identical bytes.
2. 02Run once per version. Rerun only on a new model release or a dataset or normalizer bump.
3. 03Frozen weights. English index = 50% pipecat + 50% primock57; overall = 40/40/20 with VoxPopuli. Fixed in the spec before the first run.
4. 04Intervals, not podiums. 95% intervals (the conventional level, not tuned) from a 1,000-sample bootstrap on every WER, primock57 resampled by consultation. No order is claimed inside overlapping intervals; overlap is cautious, so a shaded model may still be behind the leader.
5. 05Agent-shaped latency. Real-time streaming, a forced endpoint at the shared VAD speech end, and a clock that starts only when the last frame has actually been sent.
6. 06No composite score. Composites hide trade-offs and make #1 depend on weights. We show Pareto frontiers instead: accuracy against latency, and against price.
7. 07Paired robustness. 100 clips × 4 mixes, each compared with itself clean. Reported as a separate probe, never folded into the main rank.
8. 08Replayable scoring. Every raw provider message is logged with a core time stamp; metrics are computed offline from the logs, so fixes never need new provider calls.

pipecat is the closest public match to voice-agent traffic. Its references are Gemini-generated and human-reviewed according to the source, so we treat agreement against the reference as a warning sign (section 10). primock57 has human transcripts of mock GP consultations: conversational, dense with medications, doses and dates, and recorded over telemedicine software with patients on mobile phones, so the audio is already Opus-compressed. We took 1,000 single-speaker turns across all 57 consultations, dropped any utterance with a self-closing tag such as <INAUDIBLE_SPEECH/> (it appears 814 times) and stripped <UNSURE> tags. VoxPopuli adds German, French, Spanish and Italian (200 clips each, plus 200 English) read by HTTP range requests from a pinned revision.

Selection rules for every clip: 2–15 s, one speaker, at least three words, deduplicated, 16 kHz mono PCM16, loudness-normalized to −20 dBFS RMS, seeded and deterministic. Silero VAD (pinned ONNX, 0.5 on / 0.35 off, runs under 250 ms dropped, +100 ms end pad) gives speech start and end. One VoxPopuli clip was excluded because VAD found no speech (its source text was misaligned).

Rejected: real telephony recordings (we chose to simulate impairments so the channel can be separated from the speaker), SLURP audio and Speech-MASSIVE (non-commercial), Switchboard and Fisher (LDC licence), and synthetic TTS speech, which inflates scores. Scale: the spec started at about 2,900 clips; a 300-clip set was rejected because its ±0.7-point interval can't separate the top models. The median English interval in v0 is ±0.37 points.

WER is (S + D + I) ÷ reference words after normalization, pooled within each dataset. Both sides go through Whisper's EnglishTextNormalizer (English) or BasicTextNormalizer (other languages). After the first full scoring we found that primock57's transcripts write "OK" and "alright" where models write "Okay" and "All right". That cost every model about 2.5 points on primock57 and inflated Amazon's lead, so we added two English equivalences (ok → okay, alright → all right) for every model. We also record substitution, deletion and insertion rates, CER on VoxPopuli, raw un-normalized WER, a hallucination rate (three or more consecutive inserted words) and an empty-transcript rate.

Streaming. Audio goes out in real time as 20 ms frames on an absolute-deadline pacer (50 ms for AssemblyAI, which rejects smaller chunks; 80 ms for Deepgram Flux). At the VAD speech end the harness sends the vendor's finalize message, then one second of silence. t0 is the moment the last speech frame and the finalize have actually been sent, so harness slip never counts against a model. TTFS (t0 → final) is the headline; TTFT is speech onset → first non-empty partial; TTFB is audio start → first server message; WER@300/500 ms is the WER of whatever text exists that long after t0. Models with no flush message (Amazon Transcribe Streaming, Gladia Solaria-1 Live) use AA's fallback, the first natural final within 2 s, and are flagged. When a provider's own endpointing finalizes before t0, TTFS is scored as 0 and flagged rather than dropped (Deepgram Flux English does this on 9% of clips).

Pacing gate. A stream counts only if no inter-frame gap exceeds 40 ms and drift stays under 2%. Failures are harness faults, resent, never charged to the model. Batch models are timed end to end (request sent → response received) and by speed factor (RTFx).

Each vendor adapter only translates its wire protocol into normalized events; a model is a YAML file (vendor, parameters, input format, languages, concurrency cap, pricing). The shared core renders audio, paces, finalizes, time-stamps, retries, logs and scores. A final commits a segment and a partial is the current segment; adapters for providers that resend cumulative text diff it themselves. Concurrency adapts per model (start at 2, double on clean streaks, halve with a 30 s cooldown on rate limits, stop on auth or quota errors). Every model writes identical results.csv columns and a summary.json with every metric present, null-shaped where it doesn't apply.

Per-vendor decisions are recorded in each YAML. The ones that touch results: Cartesia's final chunks are joined as one running segment; Inworld's socket is closed after the forced-endpoint acknowledgement (its documented shutdown transcribed the silence tail as "Oh."); Together's Nemotron stream gets 320 ms of silence after t0 to flush its lookahead, which does not flatter TTFS; NVIDIA Riva and Google Chirp finalize by half-closing the gRPC stream; Speechmatics Melia-1 requires language: multi; Voxtral Small only works through chat completions with the shared transcription prompt; Alibaba's fun-asr alias is English only because it translated German; AssemblyAI Universal Streaming English uses format_turns for punctuated output; Sarvam is sent as en-IN.

The target was Artificial Analysis's STT roster: 106 host-model entries (61 batch, 45 streaming), matched by exact API id. We built 101 configs across 33 vendors, including three that aren't on AA's list: Deepgram Flux Multilingual, AssemblyAI U3.6 Pro streaming in its default balanced mode, and Alibaba's international fun-asr-realtime alias. A model is ranked only if every clip of both sets finished (scored, refused by the provider, or outside its languages): 81 did.

AA entries we could not run at all: Amazon Nova 2 Pro (gated preview), the Fun-ASR 2026-02-28 snapshot (its China region needs real-name verification), NVIDIA Nemotron 80/560/1120 ms (only the 160 ms build is hosted) and Microsoft MAI-Transcribe (not offered in our Azure region).

The clean run r-20261020-221120 ran on one EC2 c7i.8xlarge (32 vCPU) in AWS us-west-2, one OS process per model, all models at once, reachable only through AWS Session Manager. We first verified that 43 concurrent streaming processes × 12 streams held a worst frame gap of 30 ms, under the 40 ms gate. 68 models finished in about an hour, 82 within about 2.5 hours, and the rest were limited by vendors. The robustness run r-20261022-025101 used the same host for the 80 complete models. NVIDIA Nemotron 3.5 Transcribe was added afterwards from a laptop against NVIDIA's single-H100 endpoint; it is batch-only, so host placement does not affect its scores.

Region caveat. Vendors hosted in or near us-west-2 get a network advantage. AssemblyAI Universal Streaming English's 5 ms TTFS is real but co-located. AA's numbers have the same effect; we state it rather than correct for it.

A failed clip is either the harness's fault (resent until it succeeds) or the model's (kept and counted). Together AI's "no available server" responses (1,984 clips) and Zoom and StepFun 5xx errors were transient and resent. Real failures stayed in: Qwen Omni's content filter refused 6 clips each for Flash and Plus, Gemini 3.1 Pro at high thinking stalled on 26, and Rev AI rejected some clips as too short. Unsupported languages are not failures; those clips are out of that model's scope.

Two scoring bugs were fixed before publishing: the log reader split lines on a Unicode line separator (U+2028) inside Gemini output, so it now splits on newline only; and a half-written last line from a live attempt is skipped. Some outputs looked like bugs and turned out to be real model behaviour, kept as scored: GPT-4o Transcribe writes some accented English as foreign-sounding words and dates as digits; Voxtral Small answers some English clips in Portuguese; Gemini 2.5 Flash-Lite loops and emits subtitle time stamps on VoxPopuli (its German WER is 375%).

Bluejay MISTT Bench: English WER · top 25 of 81

batchstreaming95% CI · lower is better

bluejay-labs

Figure 1 Shaded rows overlap the interval of the lowest point estimate (Pulse Pro, 3.38%). Median across all 81 models: 4.77%.

Accuracy is a tie at the top. Pulse Pro has the lowest batch English WER (3.38%) and Amazon Transcribe Streaming the lowest streaming (3.81%), but 11 batch and 15 streaming models sit inside those leaders' intervals. Inside the band, the choice should be made on latency, price, languages and robustness. At the other end, GPT-4o Transcribe, Universal Streaming English, Solaria-1 Live have the highest English WER; for GPT-4o Transcribe (14.35%) the behaviour in section 9 explains most of it.

Doctor–patient audio is harder for everyone. Median WER is 2.40% on pipecat and 6.91% on primock57; every model is worse on primock57, by at least 2.3 points. The leaders differ: Gemini 3.5 Transcribe, Gemini 3.1 Flash-Lite (minimal), Gemini 3.5 Transcribe Live lead pipecat, while Amazon Transcribe, Pulse Pro, Gemini 3.1 Pro (high) lead primock57. A benchmark built only on assistant-style turns would pick different winners from one built on clinical conversation, which is why the English index weighs them equally.

Figure 2 14 vendors serve one model family in both modes. Streaming is worse in 14; the intervals separate for Gladia, Mistral, Smallest.ai.

Streaming costs accuracy. Seeing the audio only up to the forced endpoint, with no right context, costs a median +0.36 points of English WER. For most vendors the gap is inside the noise; for 3 it is not.

Figure 3 Pareto frontiers (dashed). Streaming frontier: Universal Streaming English (5 ms, 8.74%), Ink-2 (semantic endpoint) (71 ms, 4.08%), Ink Preview (72 ms, 4.03%), Ink-2 (73 ms, 4.02%), U3.5 Pro Streaming (max accuracy) (173 ms, 3.93%), Amazon Transcribe Streaming (555 ms, 3.81%). Cost frontier: Nemotron 3.5 Transcribe 30B-A4B ($0), Pulse Pro ($4.00).

Latency. Median streaming TTFS is 179 ms; the fastest are Universal Streaming English (5 ms), Ink-2 (semantic endpoint) (71 ms), Ink Preview (semantic endpoint) (71 ms), Ink Preview (72 ms), Ink-2 (73 ms). The slowest include Chirp 3 Streaming (731 ms), GPT Live Transcribe (609 ms), GPT Realtime Whisper (563 ms), Amazon Transcribe Streaming (555 ms), two of which use the fallback. WER@500 ms is close to final WER for most models, because their finals land well within half a second. The exception is Chirp 3 Streaming: 54.00% English WER at 500 ms against 5.10% final, because its final arrives at 731 ms and it produces no early partials (TTFT p50 5.47 s). An agent that acts at 500 ms would act on nothing. For batch, median end-to-end time is 1.16 s; the fastest are Whisper Large v3 Turbo (Groq) (194 ms), Parakeet TDT 0.6B v3 (Together) (202 ms), Pulse (219 ms), Parakeet TDT 0.6B v2 (235 ms).

Price and billing. Effective cost runs from $0 (NVIDIA's free trial endpoints) and Step-Audio 2.5 ASR at $0.37 and Velma-2 Batch English at $0.42 to Gemini 3.1 Pro (high) at $35.8, Gemini 3.1 Pro (low) at $31.0, Azure Speech Realtime at $19.1 per 1,000 audio minutes. Billing rules move the bill on short turns (median clip 7.29 s): per-second rounding adds a median 6% for batch models that use it, the one-second streaming tail about 8%, and Groq's 10-second minimum makes Whisper Large v3 Turbo 36% dearer than its list rate.

Multilingual. 70 models support at least one non-English language. Among those covering all five, the lowest pooled German/French/Spanish/Italian WER belongs to Qwen3.5 Omni Plus (10.12%), Amazon Transcribe (10.56%), Qwen3.5 Omni Flash (10.67%). Median WER by language: en 7.2%, de 11.5%, fr 11.6%, es 8.3%, it 19.5%. VoxPopuli references spell numbers out and the non-English normalizer doesn't convert digits, so models that format numbers are penalized there.

The clean scores may flatter: all three datasets are public. The robustness probe uses new audio and pairs each degraded clip with its clean original, so a ΔWER is what the channel costs a model on the same words.

Base clips. The pool was every English pipecat and primock57 clip of 5–15 s (1,085). Difficulty is the median clean WER over the 80 complete models, which replaced the planned Whisper + Parakeet pre-scoring: 80 models are a steadier judge than two. We removed 156 reference-error suspects (at least 70% of models within one word of the most common hypothesis, and that hypothesis differing from the reference), kept the easy and hard terciles, and asked Gemini 3.1 Pro (temperature 0, JSON schema) about each of the 618 candidates: is the reference exact, is there one speaker, is every word intelligible, is there an entity. Hard clips mostly failed on wrong references. The final 100 are 25 per dataset × difficulty cell, entity clips drawn first (84 of 100 have one), at most three per primock57 consultation; 20 more passed clips became the calibration dev set.

Mixes. Noise is MUSAN's free-sound background set (CC BY 4.0), after Gemini heard voices in 13 of the 34 files and we dropped them (21 remain), so speech only enters through Crosstalk. The talker is one AMI distant-microphone speaker, cut from single-speaker stretches using the manual word timings, running for the whole clip and 2 s past the caller. Noise and talker are added at the microphone, before the codec; loss is a Gilbert–Elliott burst model on 20 ms frames with repeat-last-frame concealment; output is renormalized to −20 dBFS; the speech-end anchor always comes from the clean clip.

Calibration. On the 20-clip dev set with four batch models, Severe at 10 dB SNR lifted strong models to about 3× their clean WER (the target); 5 dB gave 5–6× and stopped discriminating. Crosstalk WER is mostly talker words, so its SIR was tuned on the caller's substitutions and deletions instead: 10 dB was the closest to target that keeps the talker clearly audible (the spec's 5 dB was too loud to be fair).

Fairness. Gemini then listened to all 400 degraded clips. A clip is fair if at most one trivial filler word is masked, and for Crosstalk the caller must stay dominant. Round 1 flagged 47; after five rounds of swapping base clips (9), reseeding (35) and making six clips 3 dB milder, none were left. Gemini's "background speech" flag also fired on Light clips, so it is recorded but doesn't fail a clip.

Figure 4 One base clip in every condition, with four models' transcripts. Listen to all of them on the methodology.

Figure 5 Kendall τ between the clean and mix rankings: Light 0.703, Moderate 0.633, Severe 0.588, Crosstalk 0.374.

Results. Median paired ΔWER is +0.5 (Light), +4.9 (Moderate), +6.1 (Severe), +11.7 (Crosstalk) points. A clean phone line barely matters; codec damage, loss and noise cost about five to six points; a background talker costs about twelve. The rank order degrades in step: only 4 of the clean top 10 remain top 10 on Severe, and 4 on Crosstalk.

On Severe the smallest losses belong to Zoom Scribe EN Pro (+2.0), U3.6 Pro Streaming (min latency) (+2.6), U3.6 Pro Streaming (max accuracy) (+2.7), Qwen3.5 Omni Plus (+2.7), Universal-3 RT Pro (+2.7); AssemblyAI's U3.6 and RT Pro streaming models recur near the top of every mix. Velma-2 Batch English climbs from #54 clean to #20 on Severe. The biggest Severe fallers are Gemini 2.5 Flash-Lite (#48 → #81), Solaria-3 (#15 → #48), Gemini 3.5 Transcribe Live (#9 → #40), Step-Audio 2.5 ASR Streaming (#40 → #70).

Background speech is its own failure mode. Scribe v2 is #1 on the clean subset and #79 on Crosstalk: it transcribes 24% of the talker's words. Median bleed is 3.4% for batch and 3.2% for streaming. The lowest bleed belongs to GPT-4o Transcribe (0.25%), Zoom Scribe Live (0.42%), Universal-3 RT Pro (0.51%), but low bleed is not automatically good: GPT-4o Transcribe also drops caller words under noise. Batch models hear the talker's 2 s tail after the caller stops and streams end 1 s after speech end, so batch and streaming bleed are not strictly comparable. A model that writes down an audible voice is hearing correctly; it isn't separating speakers, which is what a voice agent needs.

Runaways. Gemini 2.5 Flash-Lite, Gemini 3.1 Flash-Lite (minimal) fall into repetition loops on a few degraded clips, pushing pooled WER past 100%. Their rows are flagged rather than charted.

Table 5 Caveats specific to the probe: Gemini selected the clips and is also ranked; reseeding until a clip passes leans slightly toward easier realizations; 100 clips give wider intervals than the main set; batch models hear the talker tail and streams don't.

• All three base datasets are public and may be in training data; clean scores can flatter. The degraded copies are new audio and the more trustworthy signal.
• pipecat references were generated by Gemini and human-reviewed. Suspect references were removed from the robustness subset, but the main set's references are uncorrected in v0; the planned disagreement-triggered review is still to do.
• Entity accuracy was designed and the subset is entity-tagged, but the per-type scorer isn't built, so v0 reports no entity numbers.
• primock57 is 57 consultations; its intervals are wide because they are clustered, which is correct.
• Degradations are simulated, not carrier recordings. One talker at one level, and the talker is meeting speech.
• The robustness subset is 100 clips. Gemini was used as the listener instead of human listening on every subset reference.
• One worker region (us-west-2) favours co-located vendors. Trial-tier vendors are rate-limited. Prices were checked on 19 October 2026 and token-priced models are estimates.
• Native endpointing, diarization, single-factor impairment attribution and more crosstalk variants are deferred to v1.1.

At each vendor's pay-as-you-go rate, the clean run of the 81 ranked models comes to about $212; including the configs that didn't finish, the clean run was about $260 of estimated API spend. The robustness run was about $34, Gemini's listening checks about $25, and the EC2 host about $1.40 an hour. Gemini 3.1 Pro at high thinking sometimes thought for more than 60k tokens on an ambiguous clip (over five minutes, about $0.75 each); those calls were retried at low thinking after a timeout.

Every number on the MISTT pages and in this paper is generated from the run's per-model summary.json and results.csv files and the robustness report by scripts/ingest_mistt.py in the Bluejay Labs repository. The harness, model configs, manifests (clips-v0.jsonl, robust-v0.jsonl) and the robustness build log live in the mistt-bench repository. The clean audio comes from the three public datasets at the pinned revisions above; the degraded audio is rebuilt deterministically from clean SHA + mix + seed. Sources: pipecat, primock57, VoxPopuli, MUSAN, AMI, Artificial Analysis methodology.

@misc{savalia2026mistt,
  title  = {MISTT Bench: Speech-to-text for voice agents under channel impairments},
  author = {Savalia, Yash},
  year   = {2026},
  month  = {oct},
  note   = {Bluejay Labs. v0, clean run r-20261020-221120, robustness run r-20261022-025101},
  url    = {https://research.getbluejay.ai/whitepapers/mistt-bench}
}