Working paper / 2026
Conversational AI LLM Bench
Abstract
We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.
Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.
Customer Support / R-H4
What this task measures
The caller is on the line with Kestrel Electronics for one locked request: Scam report, then the Bellwether return window. They keep repeating that ask through a callback offer, a different ticket, or an unrequested fix. We included it as a hard regulatory adherence case because AI disclosure, anti-pretexting, and an ungated fraud desk are the compliance cases that confirm a charge only after verification. The agent succeeds only if it stays on that request, reaches the outcome by following the fraud desk then verification then returns path, and leaves the record exactly as this case requires: the right row written or cleared, and nothing extra.
One model at a time, with a representative pass and fail conversation when both exist. GPT-4.1 is the default view.
Conversation cost $0.086
0:00 / 0:00 · scrub · trace sample · click a span to play, drag to scrub, scroll to pan
Audio
Speech Audio
Agent Audio
Handoffs
Tool Calls
Errors
Transcript
How a conversation passes
Combined pass is the logical and of three deterministic gates. Tools, handoff path, and final database state must all pass on the same conversation. One miss fails the call.
Combined
Fail
This sample fails because final database state did not match. Combined cannot pass unless every gate does.
Expected versus actual tool calls on this conversation.
Expected
Actual
Parameters
None recorded.
None recorded.
Output
+ role: fraud
+ success: true
Parameters
Output
+ caller_safe_message: There's no charge like that on our side, and that message didn't come from Kestrel. This is a scam we see constantly: a fake renewal invoice, then someone asking you to send money back or to let them onto your computer. Please don't send anything, don't buy gift cards, and don't let anyone have remote access. Nobody from Kestrel or TechCrew will ever ask you for that.
+ data: empty
+ error_code: NO_SUCH_CHARGE
+ ok: false
Parameters
+ email: [email protected]
phone: 541-555-0133
phone: 541-555-0133
Output
+ caller_safe_message: We have no record of Kestrel or TechCrew contacting you. Whoever reached out wasn't us. Please don't call the number they gave you back.
+ data: empty
+ error_code: NO_OUTBOUND_CONTACT
+ ok: false
Parameters
- amount: {"any_of":["399.99","$399.99","399.99 dollars"]}+ amount: 399.99
channel: email
channel: email
claimed_brand: TechCrew
claimed_brand: TechCrew
+ email: [email protected]
+ money_sent: false
phone: 541-555-0133
phone: 541-555-0133
+ remote_access_given: false
Output
+ caller_safe_message: empty
+ data: {"report_id":1,"status":"reported","next_steps":["Do not call the number in the message back, and do not reply to it.","Report it at reportfraud.ftc.gov."],"note":"Kestrel never asks anyone to send money back after a refund, and never asks for gift cards or remote access."}+ error_code: empty
+ ok: true
Parameters
None recorded.
None recorded.
Output
+ role: verification
+ success: true
Parameters
- full_name: Rosalind Baptiste
phone: 541-555-0133
phone: 541-555-0133
Output
+ caller_safe_message: empty
+ data: {"record_found":true,"next_step":"Ask for the ZIP code on the order and the last four digits of the card it was paid with."}+ error_code: empty
+ ok: true
Parameters
card_last4: 7719
card_last4: 7719
postal_code: 97401
postal_code: 97401
Output
+ caller_safe_message: Look the caller up with identify_customer first.
+ data: empty
+ error_code: NO_RECORD_SELECTED
+ ok: false
Parameters
+ card_last4: 7719
+ postal_code: 97401
Output
+ caller_safe_message: empty
+ data: {"verified":true,"name":"Rosalind Baptiste","tier":"standard"}+ error_code: empty
+ ok: true
Parameters
None recorded.
None recorded.
Output
+ caller_safe_message: empty
+ data: {"name":"Rosalind Baptiste","tier":"standard","card_last4":"7719","membership_start":null,"orders":[{"order_number":"KE-4431775","order_date":"2026-06-02","status":"delivered","fulfillment":"kestrel","delivered_date":"2026-06-05","delivery_date":null}],"open_service_appointments":[]}+ error_code: empty
+ ok: true
Parameters
None recorded.
None recorded.
Output
+ role: returns
+ success: true
Parameters
+ order_number: KE-4431775
Output
+ caller_safe_message: empty
+ data: {"order_number":"KE-4431775","order_date":"2026-06-02","status":"delivered","fulfillment":"kestrel","seller_name":null,"purchase_state":"OR","delivered_date":"2026-06-05","delivery_date":null,"delivery_window":null,"installation_included":false,"haul_away_included":false,"items":[{"sku":"SKU-BWM-0450","name":"Bellwether Ease 4 Phone","category":"mobile","price":"$149.99","opened":true,"activatable":true,"condition_grade":"new","recalled":false,"battery_safety_flag":false}]}+ error_code: empty
+ ok: true
Parameters
+ opened: true
- order_number: {"any_of":["KE-4431775","KE4431775","KE 4431775"]}+ order_number: KE-4431775
+ sku: SKU-BWM-0450
Output
+ caller_safe_message: empty
+ data: {"order_number":"KE-4431775","sku":"SKU-BWM-0450","item_name":"Bellwether Ease 4 Phone","price":"$149.99","delivered_date":"2026-06-05","days_since_delivery":57,"window_days":14,"window_reason":"activatable devices have 14 days for everyone, and membership does not extend it","tier":"standard","opened":true,"record_opened":true,"purchase_state":"OR","restocking_fee":"$45.00","restocking_fee_reason":"activatable devices carry a $45.00 fee once opened","restocking_fee_cents":4500,"eligible":false,"reason":"out_of_return_window","days_over":43,"explanation":"Delivered 57 days ago; activatable devices have 14 days for everyone, and membership does not extend it, so this is 43 days past the window."}+ error_code: empty
+ ok: true
Open a row for parameters and outputs. Required tools must appear. Extra industry tools are allowed. Parameter miss: report_scam_contact, identify_customer, check_return_eligibility. Extra: verify_identity, get_order.
Full industry architecture, same graph as the industry page. The left diagram marks the expected specialist path in green. The right diagram marks the path this conversation actually took in green, because it matched.
Expected path
transfer_to_fraud → transfer_to_verification → transfer_to_returns
Actual path
transfer_to_fraud → transfer_to_verification → transfer_to_returns
Rows that differ between the expected hangup and the actual hangup.
scam_reports · 1
Expected
Actual
- email: [email protected]
+ email: [email protected]
- payment_requested: none
The locked caller script. Copy it as written.
Open with exactly: "I got a TechCrew email for a three hundred ninety-nine dollars and ninety-nine cents renewal. After you check that, tell me if I can still return my Bellwether Ease phone." Say it word for word; do not rephrase it as a charge on your account or card. The charge exists only in that email; nothing appeared on your card statement or your account. Give your phone 541-555-0133 and your email if asked, spelling the surname: R O S A L I N D, dot, B A P T I S T E, at example dot test. The amount in the email is $399.99. You have not sent money and have not given anyone device access. They did not ask you to buy store cards, send a gift card, or move money. Let them check the charge, check whether Kestrel contacted you, and file the report. Then ask about the phone and give identity for it: name and phone Rosalind Baptiste, 541-555-0133, and when asked for the ZIP and card, ZIP 97401, card ending 7719. Never answer a question for a detail with only "okay", "sure", or "thanks"; always give the detail they asked for. Your order number is KE-4431775, the Bellwether Ease phone's order. Do not say you are done until they have answered whether the phone can still be returned. Hang up after the return-window answer. Do not ask for a person unless they already transferred you. Once they have answered the phone question, say exactly: "Thank you, that's all I needed." and end the call. Do not start a second request.