Bluejay / Labs
Open navigation

Working paper / 2026

CALLM

Conversational AI LLM Bench

Abstract

We define conversational AI LLMs as the language models at the center of cascaded voice pipelines, between speech recognition and speech synthesis, that return a first token within one second: a median time to first token of 1,000 ms or less. The Conversational AI LLM Bench (CALLM) benchmarks this category. Each model operates every agent of a multi-agent front desk in healthcare, legal, and customer support, with production-length prompts, executable tools, and per-call database state. A simulated caller pursues a locked task over a simulated phone line, so the model receives only what a speech recognizer would have transcribed, errors included. A deterministic verifier scores tool calls, handoffs, and final database state; pass@1 and pass5 measure single-attempt capability and five-attempt reliability.

GitHub repository ↗

Clone the repository, point a model at an industry pack, and reproduce the evaluation environment.

Leaderboard

Each row is one LLM running the same locked task suite and deterministic verifiers. pass@1 is single-run success, pass5 is five-run reliability, and cross-industry values are an unweighted mean of the published suites. Pick a scope once and the chart and table follow it.

Scope

Bluejay CALLM Bench: Cross-industry pass5

higher is better
bluejay-labs
Claude Haiku 5.5 · 62.2%62.2%Claude Haiku 5.5Tell-1 · 60.6%60.6%Tell-1DeepSeek V4.1 Flash · 57.8%57.8%DeepSeek V4.1 FlashGemma 4 31B · 46.7%46.7%Gemma 4 31BPhoneLLM Max Alpha 2 (microthinking) · 42.8%42.8%PhoneLLM Max Alpha 2 (microthinking)GPT-4.1 · 41.7%41.7%GPT-4.1GPT-4.1 mini · 40.0%40.0%GPT-4.1 miniQwen3.5 397B A17B · 39.4%39.4%Qwen3.5 397B A17BGemini 2.5 Flash · 38.9%38.9%Gemini 2.5 FlashClaude Haiku 4.5 · 36.7%36.7%Claude Haiku 4.5Llama 4 Maverick · 33.9%33.9%Llama 4 MaverickGPT-4o · 33.3%33.3%GPT-4oGemma 4 26B A4B · 27.8%27.8%Gemma 4 26B A4Bgpt-oss-120b · 26.1%26.1%gpt-oss-120bGPT-4o mini · 22.8%22.8%GPT-4o miniQwen3.5 122B A10B · 22.2%22.2%Qwen3.5 122B A10BMinistral 3 14B · 17.8%17.8%Ministral 3 14BGemini 2.5 Flash-Lite · 17.2%17.2%Gemini 2.5 Flash-LiteLlama 3.3 70B · 15.0%15.0%Llama 3.3 70Bgpt-oss-20b · 13.3%13.3%gpt-oss-20bPhoneLLM Mini Alpha 2 · 13.3%13.3%PhoneLLM Mini Alpha 2GPT-5 mini · 12.8%12.8%GPT-5 miniThink-Fast · 12.8%12.8%Think-FastQwen3.6 35B A3B · 12.2%12.2%Qwen3.6 35B A3BMistral Small 4 · 11.1%11.1%Mistral Small 4PhoneLLM Alpha 1 · 10.0%10.0%PhoneLLM Alpha 1Llama 4 Scout · 7.8%7.8%Llama 4 ScoutNemotron 3 Nano 30B A3B · 7.8%7.8%Nemotron 3 Nano 30B A3BQwen3.5 9B · 7.2%7.2%Qwen3.5 9B

Scroll for more →

CALLM BenchmarkUpdated November 11th 2026

Frontiers

Two metrics at a time

The staircase joins the models nothing else beats on both, at cross-industry scope.

Cost versus latency

$/hr · ms
200ms400ms600ms800ms1000ms1200ms$0.05$0.1$0.2$0.5$1$2Claude Haiku 5.5 · Cost per hour $0.206/hr · Median latency 963msDeepSeek V4.1 Flash · Cost per hour $0.031/hr · Median latency 476msGemma 4 31B · Cost per hour $0.306/hr · Median latency 383msPhoneLLM Max Alpha 2 (microthinking) · Cost per hour $0.222/hr · Median latency 499msGPT-4.1 · Cost per hour $0.894/hr · Median latency 693msGPT-4.1 mini · Cost per hour $0.245/hr · Median latency 653msQwen3.5 397B A17B · Cost per hour $0.234/hr · Median latency 395msGemini 2.5 Flash · Cost per hour $0.132/hr · Median latency 610msClaude Haiku 4.5 · Cost per hour $1.34/hr · Median latency 459msLlama 4 Maverick · Cost per hour $0.484/hr · Median latency 486msGPT-4o · Cost per hour $1.85/hr · Median latency 660msGemma 4 26B A4B · Cost per hour $0.058/hr · Median latency 399msgpt-oss-120b · Cost per hour $0.135/hr · Median latency 340msGPT-4o mini · Cost per hour $0.123/hr · Median latency 748msQwen3.5 122B A10B · Cost per hour $0.221/hr · Median latency 284msMinistral 3 14B · Cost per hour $0.244/hr · Median latency 989msGemini 2.5 Flash-Lite · Cost per hour $0.047/hr · Median latency 466msLlama 3.3 70B · Cost per hour $2.02/hr · Median latency 896msgpt-oss-20b · Cost per hour $0.073/hr · Median latency 657msPhoneLLM Mini Alpha 2 · Cost per hour $0.055/hr · Median latency 224msGPT-5 mini · Cost per hour $0.088/hr · Median latency 784msQwen3.6 35B A3B · Cost per hour $0.032/hr · Median latency 310msMistral Small 4 · Cost per hour $0.227/hr · Median latency 827msPhoneLLM Alpha 1 · Cost per hour $0.095/hr · Median latency 394msLlama 4 Scout · Cost per hour $0.093/hr · Median latency 544msNemotron 3 Nano 30B A3B · Cost per hour $0.084/hr · Median latency 223msQwen3.5 9B · Cost per hour $0.100/hr · Median latency 410msDeepSeek V4.1 FlashQwen3.6 35B A3BPhoneLLM Mini Alpha 2Nemotron 3 Nano 30B A3BCOST PER HOUR · USD / HR · LOG · ← betterMEDIAN LATENCY · TTFT · ← better

Latency versus accuracy

ms · pass5
020406080200ms400ms600ms800ms1000ms1200msClaude Haiku 5.5 · Median latency 963ms · pass^5 62.2%Tell-1 · Median latency 352ms · pass^5 60.6%DeepSeek V4.1 Flash · Median latency 476ms · pass^5 57.8%Gemma 4 31B · Median latency 383ms · pass^5 46.7%PhoneLLM Max Alpha 2 (microthinking) · Median latency 499ms · pass^5 42.8%GPT-4.1 · Median latency 693ms · pass^5 41.7%GPT-4.1 mini · Median latency 653ms · pass^5 40.0%Qwen3.5 397B A17B · Median latency 395ms · pass^5 39.4%Gemini 2.5 Flash · Median latency 610ms · pass^5 38.9%Claude Haiku 4.5 · Median latency 459ms · pass^5 36.7%Llama 4 Maverick · Median latency 486ms · pass^5 33.9%GPT-4o · Median latency 660ms · pass^5 33.3%Gemma 4 26B A4B · Median latency 399ms · pass^5 27.8%gpt-oss-120b · Median latency 340ms · pass^5 26.1%GPT-4o mini · Median latency 748ms · pass^5 22.8%Qwen3.5 122B A10B · Median latency 284ms · pass^5 22.2%Ministral 3 14B · Median latency 989ms · pass^5 17.8%Gemini 2.5 Flash-Lite · Median latency 466ms · pass^5 17.2%Llama 3.3 70B · Median latency 896ms · pass^5 15.0%gpt-oss-20b · Median latency 657ms · pass^5 13.3%PhoneLLM Mini Alpha 2 · Median latency 224ms · pass^5 13.3%GPT-5 mini · Median latency 784ms · pass^5 12.8%Think-Fast · Median latency 285ms · pass^5 12.8%Qwen3.6 35B A3B · Median latency 310ms · pass^5 12.2%Mistral Small 4 · Median latency 827ms · pass^5 11.1%PhoneLLM Alpha 1 · Median latency 394ms · pass^5 10.0%Llama 4 Scout · Median latency 544ms · pass^5 7.8%Nemotron 3 Nano 30B A3B · Median latency 223ms · pass^5 7.8%Qwen3.5 9B · Median latency 410ms · pass^5 7.2%Claude Haiku 5.5Tell-1gpt-oss-120bQwen3.5 122B A10BPhoneLLM Mini Alpha 2Nemotron 3 Nano 30B A3BMEDIAN LATENCY · TTFT · ← betterPASS^5 · RELIABILITY, % · better →

Cost versus accuracy

$/hr · pass5
020406080$0.05$0.1$0.2$0.5$1$2Claude Haiku 5.5 · Cost per hour $0.206/hr · pass^5 62.2%DeepSeek V4.1 Flash · Cost per hour $0.031/hr · pass^5 57.8%Gemma 4 31B · Cost per hour $0.306/hr · pass^5 46.7%PhoneLLM Max Alpha 2 (microthinking) · Cost per hour $0.222/hr · pass^5 42.8%GPT-4.1 · Cost per hour $0.894/hr · pass^5 41.7%GPT-4.1 mini · Cost per hour $0.245/hr · pass^5 40.0%Qwen3.5 397B A17B · Cost per hour $0.234/hr · pass^5 39.4%Gemini 2.5 Flash · Cost per hour $0.132/hr · pass^5 38.9%Claude Haiku 4.5 · Cost per hour $1.34/hr · pass^5 36.7%Llama 4 Maverick · Cost per hour $0.484/hr · pass^5 33.9%GPT-4o · Cost per hour $1.85/hr · pass^5 33.3%Gemma 4 26B A4B · Cost per hour $0.058/hr · pass^5 27.8%gpt-oss-120b · Cost per hour $0.135/hr · pass^5 26.1%GPT-4o mini · Cost per hour $0.123/hr · pass^5 22.8%Qwen3.5 122B A10B · Cost per hour $0.221/hr · pass^5 22.2%Ministral 3 14B · Cost per hour $0.244/hr · pass^5 17.8%Gemini 2.5 Flash-Lite · Cost per hour $0.047/hr · pass^5 17.2%Llama 3.3 70B · Cost per hour $2.02/hr · pass^5 15.0%gpt-oss-20b · Cost per hour $0.073/hr · pass^5 13.3%PhoneLLM Mini Alpha 2 · Cost per hour $0.055/hr · pass^5 13.3%GPT-5 mini · Cost per hour $0.088/hr · pass^5 12.8%Qwen3.6 35B A3B · Cost per hour $0.032/hr · pass^5 12.2%Mistral Small 4 · Cost per hour $0.227/hr · pass^5 11.1%PhoneLLM Alpha 1 · Cost per hour $0.095/hr · pass^5 10.0%Llama 4 Scout · Cost per hour $0.093/hr · pass^5 7.8%Nemotron 3 Nano 30B A3B · Cost per hour $0.084/hr · pass^5 7.8%Qwen3.5 9B · Cost per hour $0.100/hr · pass^5 7.2%Claude Haiku 5.5DeepSeek V4.1 FlashCOST PER HOUR · USD / HR · LOG · ← betterPASS^5 · RELIABILITY, % · better →

Cost versus latency

4 of 27 models sit on the frontier: DeepSeek V4.1 Flash ($0.031/hr, 476ms), Qwen3.6 35B A3B ($0.032/hr, 310ms), PhoneLLM Mini Alpha 2 ($0.055/hr, 224ms), and Nemotron 3 Nano 30B A3B ($0.084/hr, 223ms). Qwen3.6 35B A3B (12.2%), PhoneLLM Mini Alpha 2 (13.3%), and Nemotron 3 Nano 30B A3B (7.8%) make it on price alone with pass5 under 30%.

Latency versus accuracy

Only Nemotron 3 Nano 30B A3B (223ms, 7.8%), PhoneLLM Mini Alpha 2 (224ms, 13.3%), Qwen3.5 122B A10B (284ms, 22.2%), gpt-oss-120b (340ms, 26.1%), Tell-1 (352ms, 60.6%), and Claude Haiku 5.5 (963ms, 62.2%) are on the frontier; everything else is slower and less reliable than at least one of them. Between them the trade is 740 ms of median latency for 54.4 points of pass5.

Cost versus accuracy

2 of 27 models form the frontier: DeepSeek V4.1 Flash ($0.031/hr, 57.8%) and Claude Haiku 5.5 ($0.206/hr, 62.2%). Claude Haiku 5.5 tops the staircase at 62.2% for $0.206/hr.

Table

Every metric, one row per model

Cross-industry · 29 models

1
62.2%83.2%84.1%95.7%92.4%963ms$0.206/hr
2
Tell-1
Retell AI
60.6%80.0%80.4%96.1%93.9%352msN/A
3
57.8%81.1%82.7%95.7%92.4%476ms$0.031/hr
4
46.7%71.5%72.5%93.8%89.9%383ms$0.306/hr
5
42.8%67.8%69.9%93.6%87.2%499ms$0.222/hr
6
GPT-4.1
OpenAI
41.7%71.3%72.6%86.9%86.0%693ms$0.894/hr
7
40.0%66.8%67.5%90.3%87.1%653ms$0.245/hr
8
39.4%68.8%70.1%90.9%89.4%395ms$0.234/hr
9
38.9%66.0%67.6%89.7%84.0%610ms$0.132/hr
10
36.7%65.0%65.7%92.4%87.7%459ms$1.34/hr
11
33.9%61.7%63.1%84.8%77.1%486ms$0.484/hr
12
GPT-4o
OpenAI
33.3%62.3%63.7%91.2%81.7%660ms$1.85/hr
13
27.8%56.8%57.5%84.1%82.5%399ms$0.058/hr
14
26.1%55.5%57.3%79.3%76.0%340ms$0.135/hr
15
22.8%46.5%47.7%79.4%73.0%748ms$0.123/hr
16
22.2%47.9%48.7%77.4%75.8%284ms$0.221/hr
17
Ministral 3 14B
Mistral AI
17.8%48.0%50.0%85.5%72.0%989ms$0.244/hr
18
17.2%45.8%46.6%80.4%71.9%466ms$0.047/hr
19
15.0%27.2%28.1%56.4%45.2%896ms$2.02/hr
20
13.3%34.5%36.6%67.8%57.4%657ms$0.073/hr
21
13.3%25.1%25.3%62.9%51.6%224ms$0.055/hr
22
12.8%37.1%38.0%63.0%66.4%784ms$0.088/hr
23
Think-Fast
Deepgram
12.8%35.3%35.6%63.9%59.1%285msN/A
24
12.2%39.9%40.8%70.5%68.8%310ms$0.032/hr
25
Mistral Small 4
Mistral AI
11.1%38.5%39.6%65.1%64.1%827ms$0.227/hr
26
10.0%19.1%19.5%60.7%46.3%394ms$0.095/hr
27
7.8%23.0%24.2%53.6%48.8%544ms$0.093/hr
28
7.8%22.0%22.3%59.1%52.7%223ms$0.084/hr
29
Qwen3.5 9B
Alibaba
7.2%24.5%25.0%61.0%57.9%410ms$0.100/hr
  1. pass@1 is conversation-level combined pass, pooled over five independent trials per task.
  2. pass^5 is the unweighted mean, across tasks, of C(c,5)/C(n,5), the unbiased estimate that all five trials pass (tau-bench, Yao et al. 2024).
  3. Cross-industry values are the unweighted mean of healthcare, legal, and customer support.
  4. Every call runs over the simulated phone line: the model receives what a phone recognizer would transcribe, not what the caller said.
  5. Cost/hr is list price on the tokens each call used (open weights at the base model's OpenRouter price), over call hours estimated at 0.605 seconds per spoken word, the median pace of MIVAS voice calls on the same tasks. Tell-1 and Think-Fast have no published price (Think-Fast is free during Early Access).
  6. Admission: a model is listed only if its median time to first token is 1,000 ms or less, the line CALLM draws for a conversational AI LLM.