MISTT

The Pipecat STT benchmark set is a collection of 1,000 English utterances published by the Pipecat project for evaluating speech-to-text services. Each clip is a single turn of the kind a user speaks to a voice assistant: a request, a question or an instruction, often cut off mid-sentence as a real turn would be.

The audio is drawn from Pipecat's smart-turn dataset, restricted to English and to recordings of real speakers (synthetic speech excluded). Reference transcripts were generated with Gemini and reviewed by humans, according to the project's documentation.

• RepresentativenessIt is the only public corpus built from turns addressed to a voice assistant, which is the traffic a voice agent's STT model receives.
• Breadth of requestsRequests span everyday domains: home and errands, accounts and tech support, shopping, banking, travel, food and general questions.
• LicenceCC BY 4.0 permits publishing results and audio samples with attribution.

In MISTT, pipecat contributes half of the English WER index and half of the paired robustness subset. Its turns are longer and less interactive than primock57's: median 29 words, about 8% phrased as questions, and roughly a third containing a number, date, place or name.

Speech

single-speaker requests to an assistant, spontaneous

Domains

home and daily life, tech support, shopping, banking, food, work, travel, media, general knowledge

Clip length

1–16 s in the source (mean 9.6 s); 2–15 s after MISTT filtering

Speakers

not stated by the source

References

Gemini-generated, human-reviewed (per the project README)

Known limitations

• Speaker demographics, accents and recording conditions are not documented.
• References are model-generated. MISTT treats a reference as suspect where at least 70% of models agree with each other and not with it; 156 such clips were kept out of the paired subset.
• Some turns are fragments without a complete request.

Distributed as Parquet on Hugging Face, one row per utterance, with 16 kHz audio.

Native format

Container

Parquet, single train split, 1,000 rows

Fields

sample_id, audio (16 kHz), duration_seconds, transcription

Audio

16-bit PCM, 16 kHz, mono

As sampled into MISTT

Selection

all 1,000 rows, single speaker, 2–15 s, ≥ 3 words, deduplicated

Processing

16 kHz mono PCM16, RMS −20 dBFS, Silero VAD speech bounds

Independence

each turn is its own resampling unit for intervals

Manifest row (clips-v0.jsonl)

{
  "clip_id": "pipecat-001122785206",
  "dataset": "pipecat",
  "language": "en",
  "duration_s": 9.089,
  "sha256": "27988540cd05bd957215c9da3c30c89622e1097d2fdfc3f798809a0c4b563179",
  "reference": "I want to start a small vegetable garden in my own backyard and need some helpful advice on which plants are best suited for my particular climate and soil type.",
  "sample_rate": 16000,
  "speech_start_ms": 992,
  "speech_end_ms": 8324,
  "license": "CC-BY-4.0"
}

81 models scored; median WER 2.40%. The ten lowest are listed. Intervals and the full ranking are on the leaderboard.

1. 1Gemini 3.5 TranscribeGoogle · batch1.49%
2. 2Gemini 3.1 Flash-Lite (minimal)Google · batch1.52%
3. 3Gemini 3.5 Transcribe LiveGoogle · streaming1.55%
4. 4Qwen3 ASR FlashAlibaba · batch1.71%
5. 5Step-Audio 3 ASR MaxStepFun · batch1.71%
6. 6Universal-3 ProAssemblyAI · batch1.73%
7. 7Saaras v4Sarvam · streaming1.74%
8. 8U3.6 Pro Streaming (max accuracy)AssemblyAI · streaming1.76%
9. 9Universal-3.5 ProAssemblyAI · batch1.78%
10. 10Qwen3 ASR Flash RealtimeAlibaba · streaming1.81%

Hugging Face

huggingface.co/datasets/pipecat-ai/stt-benchmark-data

Code

github.com/pipecat-ai/stt-benchmark

Pinned revision

ea9d721e

Licence

CC BY 4.0

Attribution

Pipecat STT benchmark data by the Pipecat project (pipecat-ai), CC BY 4.0. No formal citation is published.