CALLM

Most phone agents in production are cascaded. Speech recognition turns the caller's audio into text, a language model decides what to say, which tool to call, and when to hand off, and speech synthesis speaks the reply. Recognition and synthesis carry the words; the language model is the only stage that reads the business policy, calls the business's systems, and writes to its records. Speech-to-speech models, which fold the three stages into one network, are outside the category.

Speed is part of the definition because the caller waits in silence until the model produces its first token. Synthesis streams, so how long the model takes to finish a reply matters far less than how long it takes to start. People answer each other within a few hundred milliseconds (Stivers et al., 2009), and the recognizer and synthesizer add their own delay on top of the model's. A model that needs several seconds to begin is not usable on a phone line, however accurate it is, and ranking it beside models that are would compare two different products.

CALLM therefore admits a model only if its median time to first token is 1,000 ms or less, measured as described under Metrics. For models that reason before they speak, the clock runs to the first spoken token. A model above the line is not a conversational AI LLM, however accurate it is: GPT-6.1 Sol, which plays the caller, measures 1,405 ms and is not ranked, and neither is Nemotron 3 Nano with reasoning on (3,014 ms; with reasoning off it is listed) or Amazon Nova 2 Lite (1,136 ms). Every model is measured again with each release, so membership in the category can change.

General LLM benchmarks evaluate knowledge or reasoning in a single prompt. A deployed phone agent fails differently. It books an appointment before the caller's identity is verified. It answers a coverage question from its prior instead of calling the eligibility tool. It agrees to a refund that policy forbids, or writes a misheard date of birth into the patient record. These failures are observable in the tool trace and the final database even when the transcript reads well.

CALLM therefore scores outcomes, not text quality. For a fixed company, caller, tool set, and channel, we ask two questions: does the model complete the task as the business requires, and does it do so on every attempt?

Let a task τ consist of a caller brief, a set of caller traits, a list of required tool calls E, an expected handoff path H, and an expected final database state S*. The environment starts every call from the same seeded database and at the same entry agent. At each step the model receives the active agent's system prompt, that agent's tools, and the conversation so far, and it either calls a tool or speaks.

Tool calls are scoped to the agent that owns them in agent_blueprint.json; a call to a tool the active agent does not own is refused. A handoff tool replaces the system prompt and tool set in place while the conversation history is preserved, matching the voice harnesses this environment was ported from. Within one agent turn the model may make up to eight requests (tool calls followed by speech). The call ends when the caller hangs up, an agent ends the call, or after 30 exchanges; a call that reaches the cap is scored as it stands, not discarded. The resulting trajectory, consisting of the transcript, the tool trace with the agent that issued each call, and the final database, is the only input to scoring.

Prompts, tools, seed data, caller configuration, phone line, and verifier are identical for every row on the leaderboard.

Each company is a directed graph of specialist agents. A reception agent routes the call, gated agents verify identity before any account-bound action, and each specialist owns the writes in its domain. Authorization rules, policy limits, and fee arithmetic are encoded in the prompts and enforced by the tools, so they are part of what the model must get right. The median agent prompt is about 2,500 tokens in healthcare and legal and 3,200 in customer support. Company descriptions, handoff graphs, and every prompt are published on the industries pages.

Each industry ships a FastAPI tool server backed by SQLite. Every tool request carries the call's identifier. On the first request for an identifier, the server creates a database from schema.sql and seed.sql; later requests reopen the same file. Concurrent calls therefore never share state, and the final database of a call reflects only that call's actions. A call that issued no tool requests has a well-defined final state: the unmodified seed.

The caller is simulated by a fixed model (GPT-6.1 Sol) that is never the model under test, run at the provider's default sampling settings. It is conditioned on the task's brief, traits, and scripted responses, and it adapts to what the agent says: it repeats a request when the answer is missing and declines offers the task rules out, such as a waitlist, a transfer, or a confirmation text. Because the caller samples, two attempts of the same task can unfold differently; this is one reason every task is attempted five times.

The caller writes only what it says. A simulated phone line then writes what a phone recognizer would transcribe, and only that reaches the model. The line sorts each caller turn into a kind (whether it has the caller's name, how long a number it holds, how many words it has) and copies the whole set of errors from a randomly drawn real turn of the same kind, out of 4,769 caller turns in our voice-agent benchmark calls where we have both the caller's scripted text and the transcript each agent's speech recognizer produced over a phone line. Errors therefore happen as often, and as often together, as on the real calls. Code writes numbers the way the recognizer does and applies its punctuation and casing habits at their measured rates; a misheard number keeps the shape of the real error it was drawn from, digits lost or added (applied in code) or heard as words; a misheard name, word, or phrase, or a number heard as words, is written by a small model prompt that may change only that span; clipped and split turns, a missing final period, and lowercase are applied in code. A clean swap of one digit for another, which no agent could catch without a read-back, is never used. Draws are seeded by task, attempt, and turn, so attempt k of a task meets the same errors for every model; the wording of each error comes from a model and can vary. When the agent reads back a detail that would sound different from what was said, the caller corrects it once.

To check the channel, we compared simulated transcripts with 80 real calls held out from calibration. A judge model shown one real and one simulated transcript of the same call identified the real one 58% of the time, where 50% would mean the two cannot be told apart. Across the 64,203 caller turns of the published runs, each error type landed within 0.5 points of its real rate.

Healthcare C3-H2 illustrates why the checks are scored separately. A new patient with Cigna asks whether the Park Avenue office accepts the plan, then asks for the first routine visit there and a waitlist spot. The eligibility check is a read and leaves no row behind, so a model that skips it and books correctly reaches exactly the expected database. Only the tool check catches the omission.

Every listed model meets the conversational AI LLM definition: a median time to first token of 1,000 ms or less.

Hosted models are called through their providers' APIs with default sampling. GPT-5 mini runs with minimal reasoning effort, the setting a phone deployment would use. Providers whose APIs require the first message to come from the user receive one fixed line stating that the call has connected. Deepgram's Think-Fast is called through Deepgram's Think API, which is in Early Access and free, so it has no cost per hour.

Retell's Tell-1 runs through Retell's agent playground API with the agent handbooks off, so it sees the same prompts and tools as every other model. Retell mocks each tool call on its side; CALLM executes the call against the real state API and returns the real result to the conversation.

Pipecat's PhoneLLM models are open weights. We serve them with vLLM on dedicated NVIDIA B200 GPUs at temperature 0, each in the mode Pipecat specifies. Alpha 1 and Mini Alpha 2 run with thinking off. Max Alpha 2 runs with microthinking, the mode it was trained for: one short sentence of reasoning before every reply, returned separately and withheld from the caller. Its latency is measured to the first spoken token and therefore includes that sentence.

gpt-oss-120b and gpt-oss-20b run on Groq at low reasoning effort. When Groq rejects a malformed tool call, the request is sent again, as a production client would, up to three times. The other open weights models (Gemma 4, Qwen3.5 and Qwen3.6, DeepSeek V4.1 Flash, Nemotron 3 Nano, Mistral Small 4, Ministral 3, Llama 3.3) are served on NVIDIA B200 GPUs on Modal, and Llama 4 Scout and Maverick on H200s, with thinking off where the model has it and the authors' default sampling. Their latency is measured on that deployment; a different host or GPU will measure differently. Llama 3.3 gets its tool list in the system prompt (Meta's template otherwise puts it in the first user turn with an instruction to answer with a function call), and Llama 4 uses vLLM's pythonic tool template with its format example quoted. Llama 4 often writes a call inside a spoken reply ("Let me check. [lookup(...)]"), which vLLM's parser accepts only as the whole reply; the client lifts such calls to known tools out of the text, speaks what precedes the first call, and drops what follows the last, which was written before the tool returned.

Each industry has six categories of ten tasks. Five cover the main workflows of the company. The sixth, R, covers regulation, refusals, escalation, impersonation, and adversarial requests. Within each category, two tasks are easy, four medium, and four hard. Difficulty reflects the number of agents and tools the task requires and how much the caller withholds or changes during the call.

Every task.json locks the caller brief and traits, the expected handoff path, the required tool calls with their constrained arguments, and the final database. The expected database is produced by replaying the required calls on a fresh seed. An oracle then replays each answer key with no model and no caller and confirms that every call is owned by an agent on the expected path, returns its expected status, and scores 1.0. The full suite is held out to limit contamination, although the agent prompts and nine sample tasks are public; the Tasks pages publish three hard tasks per industry.

Tool adherence. Each required call must match a distinct call in the trace with the same name. Constrained arguments, such as identifiers, dates, codes, and amounts, must equal the expected values after normalization (case, phone digits, numeric value, location and carrier aliases, and any alternatives the task lists). Free-text arguments are not checked. Where the answer key records a call's result, the actual call must return the same status. Additional calls are permitted, with one exception: in healthcare any text message is either required or forbidden by the task, never optional.

Handoff adherence. The expected handoffs must appear in the trace in order. Extra handoffs are permitted as long as the expected path is a subsequence of the actual one. 34 of the 180 tasks expect no handoff, and for them this check passes by definition.

Database adherence. The final database is compared with the expected state on a fixed set of tables per industry, with timestamps compared to the minute. Healthcare compares patients, appointments, and the waitlist. Legal compares intake, document, hold, evaluation, message, and escalation records, and excludes free-text fields and slot details the task leaves to the agent. Customer support compares customers, orders, and every return, refund, membership, service, and escalation table, excluding free-text reasons. Creation timestamps and free-text descriptions are excluded everywhere. A name or number that the model wrote down as misheard fails this check.

We use conjunction rather than a weighted sum because each check observes a different part of the outcome and none is sufficient alone: the C3-H2 example passes the database check without the required read, and a call can make every required call and still write a wrong value. No check calls a language model, so a saved trajectory always receives the same score and can be rescored when the verifier changes.

For a task with n scored attempts of which c pass, pass@1 is c / n, pooled over all scored attempts. pass^k is the probability that k attempts of a task all pass, estimated as C(c, k) / C(n, k) and averaged over tasks with equal weight (Yao et al., 2024). We report k = n = 5, so a task contributes to pass⁵ only if all five attempts pass.

The leaderboard is ranked by pass⁵. A deployment handles the same request many times, and a model that completes a task in three of five attempts will fail a predictable share of real callers. pass@1 measures capability; the gap between pass@1 and pass⁵ measures inconsistency. Cross-industry values are the unweighted mean of the three industries.

Latency and cost are reported alongside the scores and do not affect pass or fail. Latency is the median time to the first text or tool-call token of each model request, measured by replaying the run's requests streamed and one at a time after two warm-up requests, so that concurrency during the run does not inflate it. Latency is measured from one host. Tell-1 does not stream through the playground API, so its latency is the full round trip of each request, replayed the same way, one at a time.

Cost applies list token prices to the tokens the model under test used on each call: uncached input, cached input, and output, reasoning included. Open-weights models are priced at their base model's rate on OpenRouter. A text call has no clock, so its duration is estimated from the words spoken by both sides at 0.605 seconds per word, the median pace of 187 voice calls on the same tasks with dead air and tool time included. Cost per hour is total cost over those estimated hours. Retell has not published a price for Tell-1, so its cost is left blank.