Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
NVIDIA's open-weights Nemotron 3 Nano 30B-A3B with reasoning off, the base model of PhoneLLM Alpha 1.
| Scope | pass^5 | pass@1 | E / M / H | Tools | Handoff | Final DB | TTFT | Cost/hr |
|---|---|---|---|---|---|---|---|---|
| Cross-industry | 7.8% | 22.0% | 45 / 19 / 14 | 22.3% | 59.1% | 52.7% | 223ms | $0.084 |
| Healthcare | 11.7% | 22.0% | 60 / 22 / 3 | 22.3% | 44.3% | 67.7% | 222ms | $0.067 |
| Legal | 3.3% | 16.3% | 28 / 14 / 13 | 16.7% | 71.0% | 27.3% | 208ms | $0.071 |
| Customer support | 8.3% | 27.7% | 47 / 21 / 25 | 28.0% | 62.0% | 63.0% | 228ms | $0.113 |
E / M / H is pass@1 on easy, medium, and hard tasks. Gate columns are the share of conversations that cleared each check.
