Working paper / 2026

MISTT

Multi-condition Impairment Speech-to-Text Bench

Abstract

Multi-condition Impairment Speech-to-Text Bench (MISTT) measures the speech-to-text models that voice agents depend on. Every model hears the same 2,999 frozen clips of voice-agent calls, doctor–patient consultations and five-language parliamentary speech, plus 400 degraded copies of 100 of them: phone lines, low-bitrate codecs, packet loss, background noise and an intelligible background talker. Streaming models are timed the way an agent waits on them, from the forced end of speech to the final transcript. Accuracy carries bootstrap confidence intervals, and the degraded copies are paired with their clean originals, so robustness is measured on the same words.

Leaderboard ↗Read the white paper

v0 · October 2026

Back to datasets

Multilingual speech

VoxPopuli

facebook/voxpopuli · Meta AI (Facebook Research)

Clips in MISTT

999

Hours

2.92

Reference words

26,527

Source paragraphs

704

Overview

VoxPopuli is a large multilingual corpus of European Parliament event recordings from 2009 to 2020, released by Meta AI at ACL 2021. Its transcribed portion covers 1,791 hours and 4,295 speakers across 16 languages.

Sessions were segmented into speaker paragraphs by diarization and into utterances of at most 20 seconds by forced alignment to the Parliament's official plenary transcripts. Segments with poor alignment were filtered out.

Why it was chosen

One setting, many languagesEvery language comes from the same institution, room and recording chain, so differences between languages are not differences between corpora.
CoverageFour widely spoken European languages with 200 clips each, enough for a separate estimate per language.
LicenceThe data is distributed under CC0, with acknowledgement of the European Parliament requested.

What it provides

In MISTT, VoxPopuli supplies the multilingual WER: 200 clips in each of German, French, Spanish and Italian, pooled, plus 200 English clips reported separately. Character error rate is also recorded. Models are scored only on languages they support.

Speech: parliamentary speeches and debate
Speakers: European Parliament participants, many of them EU officials
Languages in MISTT: English, German, French, Spanish, Italian (200 clips each; 199 Spanish)
References: official plenary transcripts, aligned to audio

Known limitations

Parliamentary register and vocabulary differ from telephone conversation.
References spell numbers out, and the non-English normalizer does not convert digits, which penalizes models that write numerals.
The source notes gender imbalance and that segments can be mistranscribed where alignment failed. One Spanish clip was excluded because its text did not match its audio.

Format

Distributed on Hugging Face as Parquet, one configuration per language, with train, validation and test splits.

Native format

Container: Parquet; configs per language plus multilang and en_accented
Fields: audio_id, language, audio (16 kHz), raw_text, normalized_text, gender, speaker_id, is_gold_transcript, accent
Splits: train / validation / test

As sampled into MISTT

Selection: 200 clips per language from the pinned revision, single speaker, 2–15 s, ≥ 3 words
Access: read with HTTP range requests from the pinned Parquet files
Reference: normalized_text; raw_text kept in the manifest
Processing: 16 kHz mono PCM16, RMS −20 dBFS, Silero VAD speech bounds

Manifest row (clips-v0.jsonl)

{
  "clip_id": "voxpopuli-05546a934945",
  "dataset": "voxpopuli",
  "source_id": "20170614-0900-PLENARY-16-de_20170614-16:50:48_1",
  "language": "de",
  "speaker": "96871",
  "duration_s": 6.578,
  "reference": "es gibt jetzt im zusammenhang mit der verstärkten zusammenarbeit einen ersten gang von einigen mitgliedstaaten.",
  "extra": {
    "raw_text": "Es gibt jetzt im Zusammenhang mit der verstärkten Zusammenarbeit einen ersten Gang von einigen Mitgliedstaaten."
  },
  "license": "CC0-1.0"
}

Results on this dataset

Multilingual WER pools German, French, Spanish and Italian. 70 of 81 models support at least one of the four; the median is 12.64%.

English

7.2%

median of 81 models

German

11.5%

median of 63 models

French

11.6%

median of 70 models

Spanish

8.3%

median of 70 models

Italian

19.5%

median of 63 models

1Ink PreviewCartesia · streaming9.78%
2Ink-2Cartesia · streaming9.79%
3Ink-2 (semantic endpoint)Cartesia · streaming9.81%
4Step-Audio 2.5 ASR StreamingStepFun · streaming9.90%
5Ink Preview (semantic endpoint)Cartesia · streaming9.92%
6Qwen3.5 Omni PlusAlibaba · batch10.12%
7Amazon TranscribeAmazon · batch10.56%
8Qwen3.5 Omni FlashAlibaba · batch10.67%
9Step-Audio 3 ASR MaxStepFun · batch10.73%
10Qwen3 ASR FlashAlibaba · batch10.74%

Source and attribution

Hugging Face: huggingface.co/datasets/facebook/voxpopuli
GitHub: github.com/facebookresearch/voxpopuli
Paper (ACL 2021): aclanthology.org/2021.acl-long.80
Pinned revision: 42f01879
Licence: CC0 1.0
Attribution: VoxPopuli by Meta AI (Wang et al., 2021). Audio and transcripts from the European Parliament; acknowledgement of the European Parliament is requested.

Citation

@inproceedings{wang-etal-2021-voxpopuli,
  title     = {{V}ox{P}opuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation},
  author    = {Wang, Changhan and Riviere, Morgane and Lee, Ann and Wu, Anne and Talnikar, Chaitanya and Haziza, Daniel and Williamson, Mary and Pino, Juan and Dupoux, Emmanuel},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  month     = aug,
  year      = {2021},
  address   = {Online},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2021.acl-long.80},
  pages     = {993--1003}
}