Bluejay / Labs
Open navigation

Working paper / 2026

MIVAS

Multi-Industry Voice Agent Simulation Bench

Abstract

Multi-Industry Voice Agent Simulation Bench (MIVAS) is an indicator of voice AI performance across economic sectors. MIVAS is comprised of tasks and verifiers with granular rewards executed in production-grade multi-agent RL environments across industries where voice AI is being adopted the fastest. Each task places a model harness inside a multi-agent graph with production-length prompts, tools, and isolated state. Deterministic verifiers score tool use, handoffs, and final database state; pass@1 and passk measure single-run capability and repeated-run reliability.

GitHub repository ↗

Clone the repository, pair a model harness with an industry task suite, and reproduce the evaluation environment.

Healthcare tasks

Dermatology front desk across seven specialists: access, identity, scheduling, coverage, cosmetic quotes, billing, and clinical triage.

72 tasks

C1-E1
C1-E1: Aetna coverage at Park Avenue
Healthcare · New patient access · Dana Whitfield
easy
C1-E1-BG
C1-E1-BG: Aetna coverage at Park Avenue
Healthcare · New patient access · Dana Whitfield
easy
C1-E1-SIG
C1-E1-SIG: Aetna coverage at Park Avenue
Healthcare · New patient access · Dana Whitfield
easy
C1-E3
C1-E3: Medicaid coverage at Park Avenue
Healthcare · New patient access · Terrell Baines
easy
C1-H1
C1-H1: Possible skin cancer visit at Brooklyn Heights
Healthcare · New patient access · Ronald Mercer
hard
C1-H2
C1-H2: Hours, Medicare, then an urgent rash booking
Healthcare · New patient access · Owen Castellanos
hard
C1-H3
C1-H3: Oscar Health coverage then a new-patient booking
Healthcare · New patient access · Gwen Okoro
hard
C1-H4
C1-H4: Hours, parking, UnitedHealthcare, then a Brooklyn Heights booking
Healthcare · New patient access · Marcus Bell
hard
C1-M1
C1-M1: New-patient rash visit at Park Avenue
Healthcare · New patient access · Hal Brenner
medium
C1-M2
C1-M2: Cigna coverage then a new-patient booking
Healthcare · New patient access · Priya Raghunathan
medium
C1-M3
C1-M3: Allergy skin testing at Park Avenue
Healthcare · New patient access · Bernadette Kohl
medium
C1-M4
C1-M4: New-patient visit at Windermere
Healthcare · New patient access · Helen Cho
medium
C2-E1
C2-E1: Medical cancellation policy
Healthcare · Appointment management · Nora Ellison
easy
C2-E1-BG
C2-E1-BG: Medical cancellation policy
Healthcare · Appointment management · Nora Ellison
easy
C2-E1-SIG
C2-E1-SIG: Medical cancellation policy
Healthcare · Appointment management · Nora Ellison
easy
C2-E3
C2-E3: Appointment confirmation texts
Healthcare · Appointment management · Helen Cho
easy
C2-H1
C2-H1: Cancel August 20 follow-up
Healthcare · Appointment management · Jordan Lee
hard
C2-H2
C2-H2: Allergy shot for a child
Healthcare · Appointment management · David Park
hard
C2-H3
C2-H3: Cancel the August 21 cosmetic consult
Healthcare · Appointment management · Maria Alvarez
hard
C2-H4
C2-H4: Cancel the August 20 visit then rebook a follow-up
Healthcare · Appointment management · Jordan Lee
hard
C2-M1
C2-M1: Move August 20 follow-up
Healthcare · Appointment management · Jordan Lee
medium
C2-M2
C2-M2: Cancel a September follow-up
Healthcare · Appointment management · Alice Romano
medium
C2-M3
C2-M3: Book a follow-up for eczema
Healthcare · Appointment management · Sam Nguyen
medium
C2-M4
C2-M4: Move the September Windermere follow-up
Healthcare · Appointment management · Alice Romano
medium
C3-E1
C3-E1: UnitedHealthcare at Brooklyn Heights
Healthcare · Coverage and benefits · Ellen Sturgis
easy
C3-E1-BG
C3-E1-BG: UnitedHealthcare at Brooklyn Heights
Healthcare · Coverage and benefits · Ellen Sturgis
easy
C3-E1-SIG
C3-E1-SIG: UnitedHealthcare at Brooklyn Heights
Healthcare · Coverage and benefits · Ellen Sturgis
easy
C3-E3
C3-E3: Referral requirement at Park Avenue
Healthcare · Coverage and benefits · Nadine Corliss
easy
C3-H1
C3-H1: Aetna Open Access Elect Choice at Park Avenue
Healthcare · Coverage and benefits · Simone Ardelle
hard
C3-H2
C3-H2: Out-of-pocket cost without a member ID
Healthcare · Coverage and benefits · Aggie Trumbull
hard
C3-H3
C3-H3: New Cigna card and a follow-up booking
Healthcare · Coverage and benefits · Jordan Lee
hard
C3-H4
C3-H4: Oscar Health at Windermere then an extra follow-up
Healthcare · Coverage and benefits · Alice Romano
hard
C3-M1
C3-M1: Copay for a Park Avenue visit
Healthcare · Coverage and benefits · Jordan Lee
medium
C3-M2
C3-M2: New insurance card on file
Healthcare · Coverage and benefits · Jordan Lee
medium
C3-M3
C3-M3: Medicare at Windermere then a booking
Healthcare · Coverage and benefits · Walter Prinz
medium
C3-M4
C3-M4: UnitedHealthcare on an existing Brooklyn Heights chart
Healthcare · Coverage and benefits · Sam Nguyen
medium
C4-E1
C4-E1: Botox cost inquiry
Healthcare · Cosmetic concierge · Bettina Rausch
easy
C4-E1-BG
C4-E1-BG: Botox cost inquiry
Healthcare · Cosmetic concierge · Bettina Rausch
easy
C4-E1-SIG
C4-E1-SIG: Botox cost inquiry
Healthcare · Cosmetic concierge · Bettina Rausch
easy
C4-E3
C4-E3: Botox at the Windermere office
Healthcare · Cosmetic concierge · Gloria Beaumont
easy
C4-H1
C4-H1: Botox consult with deposit
Healthcare · Cosmetic concierge · Franklin Osei
hard
C4-H2
C4-H2: Botox on an existing cosmetic consult
Healthcare · Cosmetic concierge · Maria Alvarez
hard
C4-H3
C4-H3: Filler consult for an existing patient
Healthcare · Cosmetic concierge · Sam Nguyen
hard
C4-H4
C4-H4: Existing patient Botox after Windermere cannot do cosmetic
Healthcare · Cosmetic concierge · Alice Romano
hard
C4-M1
C4-M1: Microneedling consult
Healthcare · Cosmetic concierge · Selina Marsh
medium
C4-M2
C4-M2: Cheek filler consult
Healthcare · Cosmetic concierge · Lorraine Hobbs
medium
C4-M3
C4-M3: Chemical peel consult, no deposit
Healthcare · Cosmetic concierge · Ruth Ellinger
medium
C4-M4
C4-M4: Microneedling price and CareCredit, no consult
Healthcare · Cosmetic concierge · Tessa Lang
medium
C5-E1
C5-E1: Paying by card over the phone
Healthcare · Billing and payments · Paul Hines
easy
C5-E1-BG
C5-E1-BG: Paying by card over the phone
Healthcare · Billing and payments · Paul Hines
easy
C5-E1-SIG
C5-E1-SIG: Paying by card over the phone
Healthcare · Billing and payments · Paul Hines
easy
C5-E3
C5-E3: Card on file to hold an appointment
Healthcare · Billing and payments · Carl Nguyen
easy
C5-H1
C5-H1: Large balance, cannot pay in full
Healthcare · Billing and payments · Maria Alvarez
hard
C5-H2
C5-H2: Balance question then move August 20 visit
Healthcare · Billing and payments · Jordan Lee
hard
C5-H3
C5-H3: Itemized balance
Healthcare · Billing and payments · Alice Romano
hard
C5-H4
C5-H4: Windermere balance, visit charge, then CareCredit and a link
Healthcare · Billing and payments · Alice Romano
hard
C5-M1
C5-M1: Unrecognized bill
Healthcare · Billing and payments · Jordan Lee
medium
C5-M2
C5-M2: Disputed missed-visit fee
Healthcare · Billing and payments · Jordan Lee
medium
C5-M3
C5-M3: A bill the caller is sure they received
Healthcare · Billing and payments · Sam Nguyen
medium
C5-M4
C5-M4: Explain the visit charge, do not pay
Healthcare · Billing and payments · Jordan Lee
medium
R-E1
R-E1: Ask for a person
Healthcare · Regulatory adherence · Harriet Dowd
easy
R-E1-BG
R-E1-BG: Ask for a person
Healthcare · Regulatory adherence · Harriet Dowd
easy
R-E1-SIG
R-E1-SIG: Ask for a person
Healthcare · Regulatory adherence · Harriet Dowd
easy
R-E2
R-E2: Spreading infection with fever
Healthcare · Regulatory adherence · Bill Tanaka
easy
R-H1
R-H1: What the biopsy results say
Healthcare · Regulatory adherence · Jordan Lee
hard
R-H2
R-H2: Accutane refill
Healthcare · Regulatory adherence · Jordan Lee
hard
R-H3
R-H3: Dupixent authorization
Healthcare · Regulatory adherence · Maria Alvarez
hard
R-H4
R-H4: Lab results, portal, then a follow-up to discuss them
Healthcare · Regulatory adherence · Jordan Lee
hard
R-M1
R-M1: Biopsy results
Healthcare · Regulatory adherence · Jordan Lee
medium
R-M2
R-M2: Pay the balance by card now
Healthcare · Regulatory adherence · Jordan Lee
medium
R-M3
R-M3: Xanax refill
Healthcare · Regulatory adherence · Alice Romano
medium
R-M4
R-M4: Triamcinolone refill
Healthcare · Regulatory adherence · Sam Nguyen
medium