Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
OpenAI's small GPT-5 model, run with minimal reasoning effort as a phone deployment would.
| Scope | pass^5 | pass@1 | E / M / H | Tools | Handoff | Final DB | TTFT | Cost/hr |
|---|---|---|---|---|---|---|---|---|
| Cross-industry | 12.8% | 37.1% | 63 / 31 / 30 | 38.0% | 63.0% | 66.4% | 784ms | $0.088 |
| Healthcare | 10.0% | 27.3% | 57 / 30 / 10 | 28.0% | 34.0% | 60.0% | 792ms | $0.069 |
| Legal | 0.0% | 36.7% | 42 / 37 / 34 | 38.3% | 89.7% | 62.3% | 765ms | $0.119 |
| Customer support | 28.3% | 47.3% | 92 / 26 / 47 | 47.7% | 65.3% | 77.0% | 785ms | $0.075 |
E / M / H is pass@1 on easy, medium, and hard tasks. Gate columns are the share of conversations that cleared each check.
