Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
Legal / C4-H3
What this task measures
The caller is on the line with Halverson & Reed for one locked request: Employment intake in Washington with packet and note. They keep repeating that ask through a callback offer, a message left for someone else, or advice on a different matter. We included it as a hard intake and documents case because taking a narrative and a document packet is representative plaintiff work that starts after the conflict-before-facts rule has held. The agent succeeds only if it stays on that request, reaches the outcome by following the conflict screening then intake path, and leaves the record exactly as this case requires: the right row written or cleared, and nothing extra.
One model at a time, with a representative pass and fail conversation when both exist. GPT-4.1 is the default view.
Conversation cost $0.078
0:00 / 0:00 · scrub · trace sample · click a span to play, drag to scrub, scroll to pan
Audio
Speech Audio
Agent Audio
Handoffs
Tool Calls
Errors
Transcript
How a conversation passes
Combined pass is the logical and of three deterministic gates. Tools, handoff path, and final database state must all pass on the same conversation. One miss fails the call.
Combined
Fail
This sample fails because tools and final database state did not match. Combined cannot pass unless every gate does.
Expected versus actual tool calls on this conversation.
Expected
Actual
Parameters
full_name: Clara Whitman
full_name: Clara Whitman
- phone: 206-555-0220
+ phone: 2065550220
Output
+ caller_safe_message: empty
+ data: {"caller_id":"c_new","name":"Clara Whitman","returning_caller":false,"prior_matter_count":0}+ error_code: empty
+ ok: true
Parameters
+ handoff_summary: Clara Whitman is seeking help after being fired the day after reporting workplace safety violations, losing her wages.
Output
+ role: screening
+ success: true
Parameters
opposing_party: Harborline Industries
opposing_party: Harborline Industries
Output
+ caller_safe_message: empty
+ data: {"status":"clear","opposing_party":"Harborline Industries","may_proceed":true,"disclosure":"Do not tell the caller who the firm represents or why a conflict exists."}+ error_code: empty
+ ok: true
Parameters
practice_area: employment
practice_area: employment
Output
+ caller_safe_message: empty
+ data: {"practice_area":"employment","accepted":true,"fee_type":"contingency","contingency_pct_prefiling":33.33,"contingency_pct_litigation":40,"consult_fee":0}+ error_code: empty
+ ok: true
Parameters
practice_area: employment
practice_area: employment
state: WA
state: WA
Output
+ caller_safe_message: empty
+ data: {"state":"WA","practice_area":"employment","licensed":true,"licensed_states":["CA","FL","NY","TX","GA","IL","WA","PA","NC"]}+ error_code: empty
+ ok: true
Parameters
- incident_date: 2026-04-05
+ incident_date: 2026-05-05
practice_area: employment
practice_area: employment
state: WA
state: WA
Output
+ caller_safe_message: empty
+ data: {"deadline_date":"2029-05-04","days_remaining":1008,"limitation_years":3,"status":"ok","note":"Report this result as returned. Interpretation is an attorney's judgment."}+ error_code: empty
+ ok: true
Parameters
+ contact_details_only: false
+ handoff_summary: Clara Whitman was fired on May fifth, twenty twenty-six after reporting workplace safety violations while working in Washington. Possible wrongful termination/retaliation claim and lost wages. Conflict cleared. Deadline check passed.
Output
+ role: intake
+ success: true
Parameters
- incident_date: 2026-04-05
+ incident_date: 2026-05-05
practice_area: employment
practice_area: employment
state: WA
state: WA
- summary: Retaliation after safety report.
+ summary: Clara Whitman reported unsafe machine guards and blocked exits to HR at Harborline Industries in Washington and was fired the next day. She lost her wages.
Output
+ caller_safe_message: empty
+ data: {"intake_id":1,"recorded":true,"summary_recorded":true}+ error_code: empty
+ ok: true
Parameters
- note: Caller still has the HR email about the safety violations.
+ note: Client still has the original email sent to HR reporting the safety violations.
Output
+ caller_safe_message: empty
+ data: {"note_id":1,"status":"noted"}+ error_code: empty
+ ok: true
Parameters
channel: email
channel: email
Output
+ caller_safe_message: empty
+ data: {"document_id":1,"status":"sent","kind":"intake_packet","target":"email"}+ error_code: empty
+ ok: true
Parameters
+ reason: Caller does not want to schedule yet. Intake has been recorded and the email packet sent. Let Clara review the paperwork before following up.
Output
+ success: true
Open a row for parameters and outputs. Required tools must appear. Extra industry tools are allowed. Parameter miss: lookup_caller, calculate_filing_deadline, record_intake, add_intake_note. Extra: end_call.
Full industry architecture, same graph as the industry page. The left diagram marks the expected specialist path in green. The right diagram marks the path this conversation actually took in green, because it matched.
Expected path
transfer_to_screening → transfer_to_intake
Actual path
transfer_to_screening → transfer_to_intake
Rows that differ between the expected hangup and the actual hangup.
extra · documents · 1
Expected
Actual
+ caller_id: c_new
+ id: 1
+ kind: intake_packet
+ target: email
missing · documents · {"caller_id": "c_new", "kind": "intake_packet", "target": "email"}
Expected
Actual
- caller_id: c_new
- kind: intake_packet
- target: email
intake_notes · 1
Expected
Actual
- note: Caller still has the HR email about the safety violations.
+ note: Client still has the original email sent to HR reporting the safety violations.
intakes · 1
Expected
Actual
- incident_date: 2026-04-05
+ incident_date: 2026-05-05
- summary: Retaliation after safety report.
+ summary: Clara Whitman reported unsafe machine guards and blocked exits to HR at Harborline Industries in Washington and was fired the next day. She lost her wages.
The locked caller script. Copy it as written.
Open with exactly: "I was fired after I reported safety violations." When asked who this would be against, say exactly: "It's Harborline Industries." Washington. April fifth, twenty twenty six. Give a short account when asked. If they record intake, say exactly: "Please add an intake note that I still have the email I sent to HR about the safety violations." This is your mandatory reply the first time the agent says it has recorded your account or intake, or asks if there is anything else to add, even when that same turn also offers the packet by email or text. If the agent asks what the note should say, say exactly: "Just that I still have the email I sent to HR about the safety violations." If offered the packet, say exactly: "Email is fine." If offered a case evaluation or appointment, say exactly: "Not yet, let me read the paperwork first." Stay until they confirm the intake, the note, and the email packet are on the file. A spoken promise that they will take care of it, without those writes, is not enough. Once they confirm the intake, the note, and the email packet are on the file, say exactly: "Thank you, that's all I needed." Do not book an evaluation. End the call. The agent often says one moment or let me pull you up and then asks a question in the same turn. Always answer the question at the end of the agent's turn with the fact it asks for. Never reply with only "Sure", "Okay", "Yeah", or "Thank you" when the agent's turn ends in a question. Never say "Thank you, that's all I needed." while the agent is asking you a question, and never before the agent has said the note is added and the packet was sent by email.