Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
Every LLM on the leaderboard, in rank order. Open one for its per-industry and per-difficulty results.
Claude Haiku 5.5
Anthropic
Anthropic's fastest Claude model.
Tell-1
Retell AI
Retell AI's own LLM for phone agents, run through Retell's agent playground API with the agent handbooks off.
DeepSeek V4.1 Flash
DeepSeek
DeepSeek's open-weights V4.1 Flash, thinking off.
Gemma 4 31B
Google's open-weights dense Gemma 4 model, thinking off.
PhoneLLM Max Alpha 2 (microthinking)
Pipecat
The large model of Pipecat's second PhoneLLM release, open weights. Runs with microthinking: one short sentence of reasoning before every reply, the mode it was trained for.
GPT-4.1
OpenAI
OpenAI's non-reasoning GPT-4.1, the default LLM in many production voice-agent stacks.
GPT-4.1 mini
OpenAI
OpenAI's small non-reasoning GPT-4.1 model.
Qwen3.5 397B A17B
Alibaba
Alibaba's largest open-weights Qwen3.5 model, mixture of experts with 17B active parameters, thinking off.
Gemini 2.5 Flash
Google's fast Gemini 2.5 model, a common choice for low-latency voice agents.
Claude Haiku 4.5
Anthropic
Anthropic's previous Haiku model.
Llama 4 Maverick
Meta
Meta's open-weights Llama 4 Maverick, mixture of experts with 17B active parameters (FP8 weights).
GPT-4o
OpenAI
OpenAI's GPT-4o, the earlier flagship that many voice-agent stacks were built on.
Gemma 4 26B A4B
Google's open-weights mixture-of-experts Gemma 4 model (4B active parameters), thinking off.
gpt-oss-120b
OpenAI
OpenAI's larger open-weights model, served by Groq at low reasoning effort.
GPT-4o mini
OpenAI
OpenAI's small GPT-4o model.
Qwen3.5 122B A10B
Alibaba
Alibaba's open-weights Qwen3.5 mixture-of-experts model with 10B active parameters, thinking off.
Ministral 3 14B
Mistral AI
Mistral AI's open-weights 14B Ministral 3 instruct model.
Gemini 2.5 Flash-Lite
Google's smallest and fastest Gemini 2.5 model.
Llama 3.3 70B
Meta
Meta's open-weights Llama 3.3 70B instruct model.
gpt-oss-20b
OpenAI
OpenAI's smaller open-weights model, served by Groq at low reasoning effort.
PhoneLLM Mini Alpha 2
Pipecat
The small model of Pipecat's second PhoneLLM release, open weights.
GPT-5 mini
OpenAI
OpenAI's small GPT-5 model, run with minimal reasoning effort as a phone deployment would.
Think-Fast
Deepgram
Deepgram's Think-Fast, an LLM built for Deepgram's voice agents, called through its Think API (Early Access).
Qwen3.6 35B A3B
Alibaba
Alibaba's open-weights Qwen3.6 mixture-of-experts model with 3B active parameters, thinking off.
Mistral Small 4
Mistral AI
Mistral AI's open-weights Mistral Small 4 (119B), reasoning off.
PhoneLLM Alpha 1
Pipecat
Pipecat's open-weights phone-agent LLM: a full fine-tune of NVIDIA Nemotron 3 Nano 30B-A3B (3.5B active parameters, 256K context).
Llama 4 Scout
Meta
Meta's open-weights Llama 4 Scout, mixture of experts with 17B active parameters.
Nemotron 3 Nano 30B A3B
NVIDIA
NVIDIA's open-weights Nemotron 3 Nano 30B-A3B with reasoning off, the base model of PhoneLLM Alpha 1.
Qwen3.5 9B
Alibaba
Alibaba's small dense open-weights Qwen3.5 model, thinking off.
