MISTT

The benchmark proceeds in four stages (Figure 1). Clips are drawn from three public corpora, segmented into single-speaker turns, normalized and pinned by content hash. A stratified English subset is rendered under four channel perturbations, each paired with its clean original. Every clip is sent to every model through its production endpoint: batch models receive the complete clip, and streaming models receive real-time audio frames followed by a forced end-of-turn signal. Responses are logged verbatim and scored offline under a single normalization procedure.

Version 0 covers 81 models (45 batch, 36 streaming) from 25 vendors, 2,999 clean clips (7.06 h, 65,106 reference words) and 400 perturbed clips, for 275,319 scored transcriptions.

Each evaluated model receives five criteria:

• Accuracy: English and multilingual WER.
• Latency: TTFS (streaming), end-to-end (batch).
• Robustness: paired ΔWER per perturbation.
• Speaker isolation: bleed of a competing talker.
• Uncertainty: 95% bootstrap interval on every WER.

Figure 2 illustrates why the profile matters. The model shown ranks 25th of 81 on clean English WER and among the top four under every perturbation. Across all models, only 4 of the ten most accurate on the clean paired subset remain in the top ten under the Severe condition, and the rank correlation with the clean ordering (Kendall τ) falls from 0.703 under Light to 0.374 under Crosstalk. Clean-audio accuracy is therefore an incomplete predictor of accuracy in deployment.

Absolute error rates on MISTT are higher than on clean-audio benchmarks by construction. The median model reaches 3.21% WER on the clean paired subset and between 2.5 and 4.5 times that under Moderate, Severe and Crosstalk (Figure 5). These higher figures are the intended measurement, not a harsher scoring rule.

Artificial Analysis [15] is the closest methodological antecedent: it evaluates commercial APIs once per model version, includes scripted voice-agent speech, and times streaming systems from a VAD-detected end of speech. MISTT adopts these conventions. The Open ASR Leaderboard [16] covers the widest range of English corpora and languages, primarily for open models, and now includes paired noisy and clean copies of one test set. Speech Robust Bench [17] is the most complete robustness study, with more than one hundred paired perturbations including a controlled competing talker, evaluated on open checkpoints. CHiME-8 [18] provides real far-field, overlapping conversational speech for submitted research systems.

Three properties in Table 1 are, among these benchmarks, unique to MISTT: telephony codecs applied as a controlled condition, simulated packet loss, and a measurement of how much of a competing talker's speech a model transcribes (bleed). MISTT is also the only benchmark in the comparison that combines paired perturbations, streaming latency and confidence intervals in a single evaluation of commercial APIs.

Three corpora satisfy these requirements (Table 2). pipecat contributes requests addressed to a voice assistant, the closest public analogue to voice-agent traffic. primock57 contributes clinical consultations, a domain where single-word errors (a drug name, a dose) carry the highest cost, and supplies human references. VoxPopuli contributes the same parliamentary setting in five languages, so that differences between languages are not confounded by differences between corpora. Each corpus is described in full, with format and attribution, in the dataset directory.

Figure 3 separates two axes of coverage. Linguistic content is supplied by the source corpora; the acoustic channel is supplied by the perturbations in Section 04.

The two English corpora differ in kind, not only in vocabulary (Figure 4). pipecat turns are long, single-shot requests spread across everyday domains, with about a third containing a number, date, place or name. primock57 turns are shorter and more interactive, more than two in five end in a question, and most concern symptoms or intake details. Every model in the benchmark has a higher error rate on primock57 than on pipecat, and the most accurate models on each corpus differ, which is the reason the English index weights them equally.

Each preprocessing step (Table 3) removes a variable that would otherwise be attributed to the model. Loudness normalization prevents recording level from being scored as recognition error; a shared, pinned voice-activity detector fixes the end of speech from which every streaming latency is measured; content hashing guarantees that every model, including models evaluated in later reruns, receives byte-identical input. At approximately 1,000 clips per corpus the median English WER interval is ±0.37 points.

Corpora considered and excluded: SLURP and Speech-MASSIVE (non-commercial licences), Switchboard and Fisher (LDC licence terms), real carrier recordings (which confound channel with speaker and cannot be paired with a clean original), and synthetic text-to-speech audio, which is cleaner than natural speech and inflates accuracy.

4.1 Design of the four conditions

Light models an ordinary telephone call: a 300–3,400 Hz band-pass and G.711 μ-law coding at 8 kHz [1] [2]. G.711 is the codec of the public switched telephone network, is mandatory in WebRTC [3], and remains a default codec for SIP trunking, for example on Twilio Elastic SIP Trunking [6].

Moderate models a weak VoIP or mobile connection: Opus at 8 kbit/s narrowband (the lower end of the recommended range for narrowband speech [4] [5]), 2% packet loss and additive non-speech noise at 15 dB SNR. Severe models a poor connection: the G.711 line, 5% loss and noise at 10 dB SNR. Loss is bursty, following a Gilbert–Elliott model (mean burst of three 20 ms frames) of the kind used for burst metrics in RTCP XR [9], with repeat-last-frame concealment. For reference, the ITU-T Y.1541 objective for VoIP is a loss ratio of at most 10⁻³ [7], and RFC 6958 associates loss rates above about 5% with noticeable degradation [8]. The 10 and 15 dB noise levels fall within the 9–21 dB range used in 3GPP codec testing for street and babble noise [10]; measured everyday listening environments span roughly 2–14 dB [11].

Noise and the competing talker are added before the codec, as they reach a real handset. Strengths were calibrated on a separate 20-clip development set with four batch models: Severe at 10 dB SNR raised strong models to about three times their clean WER; 5 dB produced five to six times and ceased to discriminate between models.

Figure 5 shows the effect across all models on identical speech. The clean bar is the quantity most benchmarks report. The direction of the effect matches published results on natural recordings, for example Whisper large-v2 at 2.7% WER on LibriSpeech test-clean against 17.6% on CallHome telephone conversations [14], a gap that also reflects the difference in speaking style.

The interactive figure presents two of the 100 base clips under all five conditions, with the transcripts of four models aligned word by word against the reference. On clean audio the four models are nearly indistinguishable on both clips; the perturbations separate them.

4.2 Competing talker

The Crosstalk condition adds one intelligible speaker from the AMI Meeting Corpus (distant microphone, CC BY 4.0) at 10 dB signal-to-interference ratio, then the telephone line. The talker continues for two seconds after the target speaker stops. Background speech from other people or from media devices is a documented source of recognition error in device-directed speech [12]; the 10 dB level is milder than the 0–5 dB mixtures common in source-separation research [13].

For a voice agent, a competing talker produces two distinct failures. The first is degraded recognition of the caller, measured as ΔWER. The second is transcription of the background speaker, which the agent then passes to its language model as though the caller had said it. MISTT measures the second failure directly as bleed: the fraction of the talker's reference words, taken from AMI's manual word timings, that appear in the hypothesis. Bleed is not folded into WER, since a model that transcribes an audible voice has recognized speech correctly and failed only to attribute it.

Crosstalk is the most disruptive condition (median ΔWER +11.7 points) and the one that most reorders the field (τ = 0.374). Median bleed is 3.4% for batch models and 3.2% for streaming models. Because batch models receive the talker's two-second tail and streams close one second after the end of speech, the two modes are not strictly comparable on this measure.

4.3 Subset selection and fairness review

The 100 base clips are stratified 25 per corpus and difficulty tercile (easy and hard by median clean WER across models), with at most three clips per primock57 consultation and 84 clips containing an entity. Clips whose pipecat reference disagreed with the consensus of at least 70% of models were excluded as probable reference errors.

A perturbed clip is a fair test only if a careful listener could still recover every reference word. Gemini 3.1 Pro (temperature 0, structured output) served as that listener at three points: it screened the MUSAN noise files, removing 13 of 34 that contained audible speech so that speech enters only through the Crosstalk condition; it verified each candidate base clip; and it reviewed all 400 perturbed clips against their references.

Two caveats apply. Gemini both assisted selection and appears among the evaluated systems. Reseeding until a clip passes biases the subset slightly toward easier realizations of each condition.

Audio is transmitted as 20 ms frames on absolute deadlines (50 ms for AssemblyAI and 80 ms for Deepgram Flux, which reject smaller frames). At the VAD end of speech the harness sends the provider's finalize or flush message, then one second of silence, and closes the stream. The reference time t0 is the moment the final speech frame and the finalize message have been transmitted, so harness delay is never attributed to the model.

• TTFS (time to final segment): t0 to the provider's final transcript for the turn. The primary latency measure.
• TTFT (time to first token): speech onset to the first non-empty partial transcript.
• WER@300 ms, WER@500 ms: WER of the text held N ms after t0, final or not, for agents that begin responding before finalization.

Two providers expose no finalize message (Amazon Transcribe Streaming, Gladia Solaria-1 Live); for these, TTFS is the first natural final within 2 s of the end of speech, following Artificial Analysis, and is flagged wherever reported. Where a provider's own endpointing finalizes before t0, as Deepgram Flux does on 9% of clips, TTFS is recorded as zero and flagged rather than discarded.

Batch models receive the full clip in one request and are timed end to end, from request to response, alongside the real-time factor. For short turns end-to-end latency is the more informative measure: a 7 s clip processed at 20× real time still incurs 350 ms.

Execution. Each provider is accessed through a thin adapter that translates its protocol into four normalized events (partial, final, finalize acknowledgement, error); audio rendering, pacing, timing, retries and logging are shared. All raw messages are logged with a harness timestamp and every metric is computed by replaying the logs. The clean run (r-20261020-221120) executed all models concurrently on one AWS c7i.8xlarge in us-west-2; providers hosted near that region benefit from shorter network paths (AssemblyAI Universal Streaming English reports a median TTFS of 5 ms while co-located).

Failure attribution. Every failed request is assigned before scoring to the harness (retried) or to the model (retained and scored). Transient provider capacity errors, including 1,984 "no available server" responses from Together AI, were retried. Model-side refusals were retained: Qwen Omni's content filter refused 6 clips and Gemini 3.1 Pro at high reasoning effort stalled on 26. Clips in languages a model does not support (18 models) are excluded from that model's scope.

WER is (substitutions + deletions + insertions) divided by the number of reference words, pooled within each corpus, so longer clips carry proportionally more weight. Before alignment, references and hypotheses pass through Whisper's EnglishTextNormalizer for English and BasicTextNormalizer for German, French, Spanish and Italian [14], which remove casing and punctuation and standardize spellings and number formats. Figure 10 shows the scale of the effect: the median raw English WER across models is about four times the normalized figure, and in the example the entire raw error is formatting.

One adjustment was made after the first full scoring and is applied to every model: the equivalences ok → okay and alright → all right. primock57 references use the former spellings and nearly all models produce the latter; without the rule every model lost about 2.5 points on primock57.

Semantic WER, in which a language model judges whether a hypothesis preserves meaning, was not adopted. Its scores depend on the judge, change when the judge changes, and cannot be reproduced exactly. Errors that alter meaning without altering many words (fifteen for fifty) are better addressed by entity-level scoring, listed under future work.

Reported measures. English WER is the equal-weight mean of pipecat and primock57 WER (Figure 11), fixed in the design specification before any model was run. Multilingual WER pools the four non-English VoxPopuli languages; character error rate is also recorded on VoxPopuli. Auxiliary measures: substitution, deletion and insertion rates; hallucination rate (runs of three or more inserted words); empty-transcript rate; and raw, unnormalized WER.

Intervals are computed from 1,000 bootstrap resamples. pipecat and VoxPopuli are resampled by clip; primock57 is resampled by consultation, because turns within a consultation share speaker, microphone and topic and are not independent. The English index interval combines the two corpus bootstraps.

The leaderboard shades every model whose interval overlaps that of the lowest-WER model. Overlap is a conservative criterion: two overlapping 95% intervals do not establish equivalence, and a shaded model may still be marginally less accurate. The 95% level is the conventional one and was not tuned to the results.

• Public source audio. All three corpora are public and may appear in training data, which can favour clean scores. The perturbed audio is newly generated.
• Reference quality. pipecat references were generated by Gemini and reviewed by humans, according to the source. Where at least 70% of models agree with each other and not with the reference, the reference is treated as suspect; such clips were removed from the paired subset but not yet from the main set.
• Simulated channel. Codecs, loss and noise are simulated, and concealment is repeat-last-frame rather than Opus's native concealment, which is likely harsher. The competing talker is a single meeting speaker at one level.
• Scope of the paired subset. 100 clips, English only; its intervals are wider than those of the main set. Latency is not yet measured under perturbation.
• Single region and default settings. All requests originated in us-west-2. Models ran on provider defaults, apart from Table 4 and a shared transcription prompt for LLM-based models, without custom vocabularies.

Future versions are planned to add real carrier recordings, speakerphone and far-field capture, spelled sequences such as confirmation codes, e-mail addresses and street addresses, languages beyond Western Europe, multi-turn calls, entity-level accuracy for numbers, dates and names, and latency measured on perturbed audio.