Open navigation

Working paper / 2026

MIVAS

Multi-Industry Voice Agent Simulation Bench

Abstract

Multi-Industry Voice Agent Simulation Bench (MIVAS) is an indicator of voice AI performance across economic sectors. MIVAS is comprised of tasks and verifiers with granular rewards executed in production-grade multi-agent RL environments across industries where voice AI is being adopted the fastest. Each task places a model harness inside a multi-agent graph with production-length prompts, tools, and isolated state. Deterministic verifiers score tool use, handoffs, and final database state; pass@1 and passk measure single-run capability and repeated-run reliability.

GitHub repository ↗

Clone the repository, pair a model harness with an industry task suite, and reproduce the evaluation environment.

Back to Model Harnesses

Phonic 1 (reasoning: high)

Speech-to-speech · phonic-1

What the model is

Phonic phonic-1 with intelligence_level high, the configuration Phonic recommends for benchmarking, served over the Phonic speech-to-speech WebSocket (wss://api.phonic.ai/v1/sts/ws). Audio is pcm_16000 in both directions. Price is $0.06 per conversation minute (Phonic list price); the socket reports no token usage. The leaderboard row is labeled Phonic 1 (reasoning: high) to match the other rows; the API setting is intelligence_level high.

Provider model card

How the harness was built

The harness provisions the industry pack into Phonic at startup: one Phonic project per industry, one Phonic agent per blueprint stage with that stage's prompt and only its tools. Industry tools are custom_websocket tools; Phonic sends tool_call, the harness POSTs {TOOL_SERVER_URL}/tools/{name} with X-Mivas-Call-Id and answers tool_call_output. Schema keywords Phonic does not accept (pattern, format, length and range bounds) are folded into the field description; the tool server still enforces them.

The simulation engine reaches the pod over CHIRP, 16 kHz pcm_s16le, which matches Phonic's pcm_16000, so caller audio passes through in 20 ms chunks with no resampling, gating, or mute. Phonic owns turn-taking and barge-in. The digital silence Phonic streams between turns is not forwarded, so a transient stall cannot become a lasting playback lag.

Phonic's recommended benchmark settings are applied: the built-in choose_not_to_respond tool on every agent with respond_after_sec 3, so the agent can hold its reply while the caller is mid-thought. Industry tools are marked uninterruptible so a tool call that already wrote state is never cancelled and retried. The entry agent speaks the pack greeting, or a generated welcome when the pack has none.

Multi-agent assumptions

Handoffs are native Phonic transfers: each handoff tool is a built_in_transfer_to_agent tool targeting the stage agent in the same project, and Phonic swaps prompt and tools on the same socket with history carried over. Phonic does not start the target agent's turn on its own after a transfer, so once the transfer turn's speech ends the harness sends generate_reply, the same role response.create plays after a handoff in the other harnesses.

Runtime

October 2026 evaluation: 1,080 scored conversations across three five-run industry suites (healthcare 503142, legal 503147, customer support 503194), 360 per industry. 3 calls that never connected (NO_ANSWER) were redialed on the same digital humans before export; there are no void rows.

Cost/hr is Phonic's $0.06 per minute list price on the scored conversation duration.

Deployment

Each industry runs its own Kubernetes Deployment (seven replicas), Ingress, and Bluejay agent (phonic-sts-phonic-1-high-<industry>). The Phonic workspace allows 60 concurrent conversations, so the three industries ran together at 20 concurrent calls each. Every scored conversation started from isolated seeded state and was checked against the same deterministic gates as the other MIVAS harnesses.

Benchmark definitions and deterministic verifiers: mivas-bench.

Scores

pass@1 among pass/fail only. pass5 is reliability from the five-run healthcare, legal, and customer-support suites.

Industrypass@1pass5ToolsHandoffFinal DBLatencyCost/hr
Healthcarescored85.6%58.3%85.8%96.9%95.3%1.42s$3.60/hr
Legalscored85.3%68.1%85.6%95.0%88.9%1.22s$3.60/hr
Customer supportscored73.3%44.4%74.2%88.9%94.4%1.62s$3.60/hr
Cross-industry81.4%56.9%81.9%93.6%92.9%1.42s$3.60/hr