Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
Google's open-weights dense Gemma 4 model, thinking off.
| Scope | pass^5 | pass@1 | E / M / H | Tools | Handoff | Final DB | TTFT | Cost/hr |
|---|---|---|---|---|---|---|---|---|
| Cross-industry | 46.7% | 71.5% | 88 / 69 / 66 | 72.5% | 93.8% | 89.9% | 383ms | $0.306 |
| Healthcare | 48.3% | 71.0% | 100 / 73 / 54 | 71.0% | 92.7% | 89.7% | 416ms | $0.303 |
| Legal | 41.7% | 74.3% | 65 / 79 / 74 | 74.7% | 99.3% | 85.3% | 370ms | $0.276 |
| Customer support | 50.0% | 69.3% | 100 / 54 / 69 | 71.7% | 89.3% | 94.7% | 383ms | $0.340 |
E / M / H is pass@1 on easy, medium, and hard tasks. Gate columns are the share of conversations that cleared each check.
