Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
Meta's open-weights Llama 4 Maverick, mixture of experts with 17B active parameters (FP8 weights).
| Scope | pass^5 | pass@1 | E / M / H | Tools | Handoff | Final DB | TTFT | Cost/hr |
|---|---|---|---|---|---|---|---|---|
| Cross-industry | 33.9% | 61.7% | 87 / 64 / 46 | 63.1% | 84.8% | 77.1% | 486ms | $0.484 |
| Healthcare | 18.3% | 34.0% | 90 / 35 / 5 | 34.3% | 62.0% | 55.3% | 485ms | $0.540 |
| Legal | 26.7% | 72.7% | 75 / 73 / 71 | 72.7% | 95.0% | 86.0% | 493ms | $0.371 |
| Customer support | 56.7% | 78.3% | 95 / 85 / 63 | 82.3% | 97.3% | 90.0% | 483ms | $0.541 |
E / M / H is pass@1 on easy, medium, and hard tasks. Gate columns are the share of conversations that cleared each check.
