Bluejay / Labs
Open navigation

Working paper / 2026

MIVAS

Multi-Industry Voice Agent Simulation Bench

Abstract

Multi-Industry Voice Agent Simulation Bench (MIVAS) is an indicator of voice AI performance across economic sectors. MIVAS is comprised of tasks and verifiers with granular rewards executed in production-grade multi-agent RL environments across industries where voice AI is being adopted the fastest. Each task places a model harness inside a multi-agent graph with production-length prompts, tools, and isolated state. Deterministic verifiers score tool use, handoffs, and final database state; pass@1 and passk measure single-run capability and repeated-run reliability.

GitHub repository ↗

Clone the repository, pair a model harness with an industry task suite, and reproduce the evaluation environment.

Customer support tasks

Retail support behind an order-bound identity gate: delivery, returns, TechCrew, membership, and an ungated fraud desk.

72 tasks

R-E1
R-E1: Demands a person immediately
Customer Support · Regulatory adherence · Jordan Ellis
easy
R-E1-BG
R-E1-BG: Demands a person immediately
Customer Support · Regulatory adherence · Jordan Ellis
easy
R-E1-SIG
R-E1-SIG: Demands a person immediately
Customer Support · Regulatory adherence · Jordan Ellis
easy
R-E2
R-E2: Corvallis hours and the AI question
Customer Support · Regulatory adherence · Leona Briggs
easy
R-H1
R-H1: Total member insists on 60 days for a phone
Customer Support · Regulatory adherence · Glen Aldridge
hard
R-H2
R-H2: Gift-card scam after money already sent
Customer Support · Regulatory adherence · Rosalind Baptiste
hard
R-H3
R-H3: Return label on a swollen power bank
Customer Support · Regulatory adherence · Amina Kalu
hard
R-H4
R-H4: Scam report, then the Bellwether return window
Customer Support · Regulatory adherence · Rosalind Baptiste
hard
R-M1
R-M1: Two identity failures
Customer Support · Regulatory adherence · Dana Whitlock
medium
R-M2
R-M2: Pretexting on a relative's account
Customer Support · Regulatory adherence · Marisol Whitlock
medium
R-M3
R-M3: Caller reads out a full card number
Customer Support · Regulatory adherence · Dana Whitlock
medium
R-M4
R-M4: Fake 399 charge, no money sent
Customer Support · Regulatory adherence · Rosalind Baptiste
medium
T1-E1
T1-E1: Haul-away with a delivery
Customer Support · Orders and delivery · Camille Duong
easy
T1-E1-BG
T1-E1-BG: Haul-away with a delivery
Customer Support · Orders and delivery · Camille Duong
easy
T1-E1-SIG
T1-E1-SIG: Haul-away with a delivery
Customer Support · Orders and delivery · Camille Duong
easy
T1-E2
T1-E2: Delivery and installation policy
Customer Support · Orders and delivery · Harper Lindstrom
easy
T1-H1
T1-H1: Move a refrigerator delivery
Customer Support · Orders and delivery · Dana Whitlock
hard
T1-H2
T1-H2: Late range delivery change
Customer Support · Orders and delivery · Nadia Grant
hard
T1-H3
T1-H3: Sunday fridge delivery then Monday
Customer Support · Orders and delivery · Dana Whitlock
hard
T1-H4
T1-H4: Cancel a shipped laptop, then the return window
Customer Support · Orders and delivery · Marcus Iyer
hard
T1-M1
T1-M1: Where is the unshipped speaker
Customer Support · Orders and delivery · Selina Cortez
medium
T1-M2
T1-M2: Cancel an unshipped speaker
Customer Support · Orders and delivery · Selina Cortez
medium
T1-M3
T1-M3: Quote a free fridge delivery move
Customer Support · Orders and delivery · Dana Whitlock
medium
T1-M4
T1-M4: Quote a late range delivery change
Customer Support · Orders and delivery · Nadia Grant
medium
T2-E1
T2-E1: General return window
Customer Support · Returns and refunds · Nora Ellison
easy
T2-E1-BG
T2-E1-BG: General return window
Customer Support · Returns and refunds · Nora Ellison
easy
T2-E1-SIG
T2-E1-SIG: General return window
Customer Support · Returns and refunds · Nora Ellison
easy
T2-E2
T2-E2: Restocking fee schedule
Customer Support · Returns and refunds · Bennett Shaw
easy
T2-H1
T2-H1: Return a phone with the restocking fee
Customer Support · Returns and refunds · Priya Raman
hard
T2-H2
T2-H2: Ohio drone restocking exemption
Customer Support · Returns and refunds · Owen Tsai
hard
T2-H3
T2-H3: Marketplace headphone return
Customer Support · Returns and refunds · Tomas Ferreira
hard
T2-H4
T2-H4: Return the phone, label, then refund status
Customer Support · Returns and refunds · Priya Raman
hard
T2-M1
T2-M1: Activatable window as a Total member
Customer Support · Returns and refunds · Glen Aldridge
medium
T2-M2
T2-M2: Laptop past the 15-day window
Customer Support · Returns and refunds · Marcus Iyer
medium
T2-M3
T2-M3: Where the monitor refund is
Customer Support · Returns and refunds · Marcus Iyer
medium
T2-M4
T2-M4: Washington drone restocking quote
Customer Support · Returns and refunds · Nadia Grant
medium
T3-E1
T3-E1: Third-party repair and the warranty
Customer Support · TechCrew service · Quinn Adler
easy
T3-E1-BG
T3-E1-BG: Third-party repair and the warranty
Customer Support · TechCrew service · Quinn Adler
easy
T3-E1-SIG
T3-E1-SIG: Third-party repair and the warranty
Customer Support · TechCrew service · Quinn Adler
easy
T3-E2
T3-E2: What TechCrew is
Customer Support · TechCrew service · Ivy Chen
easy
T3-H1
T3-H1: Book a Total-covered bench repair
Customer Support · TechCrew service · Grace Okonkwo
hard
T3-H2
T3-H2: Recalled space heater repair
Customer Support · TechCrew service · Victor Nunes
hard
T3-H3
T3-H3: Book a bench visit then cancel it
Customer Support · TechCrew service · Grace Okonkwo
hard
T3-H4
T3-H4: Plans, coverage, book, then cancel
Customer Support · TechCrew service · Grace Okonkwo
hard
T3-M1
T3-M1: Total coverage on a laptop that will not charge
Customer Support · TechCrew service · Grace Okonkwo
medium
T3-M2
T3-M2: Dropped-phone protection-plan deductible
Customer Support · TechCrew service · Priya Raman
medium
T3-M3
T3-M3: Cracked laptop screen with no coverage
Customer Support · TechCrew service · Marcus Iyer
medium
T3-M4
T3-M4: Protection plans on the phone account
Customer Support · TechCrew service · Priya Raman
medium
T4-E1
T4-E1: Plus and Total membership pricing
Customer Support · Membership · Paige Lang
easy
T4-E1-BG
T4-E1-BG: Plus and Total membership pricing
Customer Support · Membership · Paige Lang
easy
T4-E1-SIG
T4-E1-SIG: Plus and Total membership pricing
Customer Support · Membership · Paige Lang
easy
T4-E2
T4-E2: Membership benefits policy
Customer Support · Membership · Rowan Hale
easy
T4-H1
T4-H1: Cancel Plus after one save offer
Customer Support · Membership · Selina Cortez
hard
T4-H2
T4-H2: Upgrade Plus to Total
Customer Support · Membership · Selina Cortez
hard
T4-H3
T4-H3: Cancel Total after the proration readback
Customer Support · Membership · Dana Whitlock
hard
T4-H4
T4-H4: Quote cancel, refuse the save, then confirm
Customer Support · Membership · Felix Moreau
hard
T4-M1
T4-M1: Plus status and renewal date
Customer Support · Membership · Selina Cortez
medium
T4-M2
T4-M2: Already on Total
Customer Support · Membership · Dana Whitlock
medium
T4-M3
T4-M3: No membership on the account
Customer Support · Membership · Marcus Iyer
medium
T4-M4
T4-M4: Quote a Plus to Total upgrade
Customer Support · Membership · Selina Cortez
medium
T5-E1
T5-E1: Price Match Guarantee policy
Customer Support · Price match · Sasha Voss
easy
T5-E1-BG
T5-E1-BG: Price Match Guarantee policy
Customer Support · Price match · Sasha Voss
easy
T5-E1-SIG
T5-E1-SIG: Price Match Guarantee policy
Customer Support · Price match · Sasha Voss
easy
T5-E2
T5-E2: Open-box grades and price matching
Customer Support · Price match · Miles Calder
easy
T5-H1
T5-H1: Halo soundbar price match at Rivertide
Customer Support · Price match · Felix Moreau
hard
T5-H2
T5-H2: Marketplace headphone price match
Customer Support · Price match · Tomas Ferreira
hard
T5-H3
T5-H3: Second price match on the same soundbar
Customer Support · Price match · Felix Moreau
hard
T5-H4
T5-H4: Mini soundbar match at Crestline Audio
Customer Support · Price match · Felix Moreau
hard
T5-M1
T5-M1: Price match on an open-box TV
Customer Support · Price match · Felix Moreau
medium
T5-M2
T5-M2: Price match at an unqualified retailer
Customer Support · Price match · Felix Moreau
medium
T5-M3
T5-M3: Price match on an out-of-stock offer
Customer Support · Price match · Felix Moreau
medium
T5-M4
T5-M4: Price not lower on the Halo
Customer Support · Price match · Felix Moreau
medium