Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
The large model of Pipecat's second PhoneLLM release, open weights. Runs with microthinking: one short sentence of reasoning before every reply, the mode it was trained for.
| Scope | pass^5 | pass@1 | E / M / H | Tools | Handoff | Final DB | TTFT | Cost/hr |
|---|---|---|---|---|---|---|---|---|
| Cross-industry | 42.8% | 67.8% | 87 / 68 / 58 | 69.9% | 93.6% | 87.2% | 499ms | $0.222 |
| Healthcare | 40.0% | 59.3% | 100 / 64 / 34 | 59.7% | 91.0% | 87.7% | 510ms | $0.228 |
| Legal | 41.7% | 78.3% | 82 / 82 / 73 | 79.7% | 99.0% | 86.7% | 494ms | $0.220 |
| Customer support | 46.7% | 65.7% | 78 / 59 / 66 | 70.3% | 90.7% | 87.3% | 497ms | $0.218 |
E / M / H is pass@1 on easy, medium, and hard tasks. Gate columns are the share of conversations that cleared each check.
