MIVAS

Anatomy of a harness

Every harness is a folder in voice-agent-harnesses/ in the mivas-bench repo. It loads the industry's agent_blueprint.json and turns that graph into a runnable multi-agent voice runtime for one provider. The work splits into four parts: a transport that carries the caller's audio, a session layer that speaks the provider's realtime protocol, an audio bridge that carries speech between the caller and the model, and a tool router that connects tool calls to the shared task environment.

The harness stays deliberately thin. Prompts, greeting text, tool schemas, and the handoff graph all come from the industry pack, and the same locked suites run on every row of the leaderboard. When two harnesses score differently on the same task, the difference comes from the model and its runtime.

Three kinds of tools

The blueprint marks every tool as function, handoff, or session, and each kind takes a different path through the harness. The split keeps the parts of the benchmark that must be identical fully identical, while the parts that genuinely differ between runtimes stay native.

Function tools do the real work: booking an appointment, looking up an order, updating a record. Every harness sends these through the same generic pipe to the task environment and reads the result back to the caller. No harness carries special-case code for any tool, so the environment behaves the same no matter which model is on the call.

Handoff tools move the conversation between agents, and every runtime has its own native way of making that switch. The harness uses whichever mechanism the platform provides and carries the conversation history along. How gracefully a runtime handles those switches is part of what the benchmark measures.

Session tools end the call. The harness lets the goodbye finish, then closes the connection.

One database per call

Each industry ships a seeded database behind a small state API, and both live inside the same pod as the harness. When a new conversation starts, the environment creates a fresh copy of the seeded database just for that call. Parallel conversations never share state, and nothing from one call can leak into another.

That choice is what makes deterministic verification possible. Every call starts from the same known state, so the verifiers can diff exactly the rows one conversation touched and compare them, along with the tools used and the handoff path, against what the task required. At hangup the final state is frozen and stored, and the evals read that frozen copy.

How a run is deployed

Each harness and industry pair is packaged as one container: the voice runtime, the industry's tool server, and its seeded database travel together. Scored runs deploy those containers to Kubernetes, one deployment per pair, and Bluejay's digital humans dial each one from the outside like real callers.

Packing everything into one unit keeps the benchmark reproducible. A pair can be rebuilt, redeployed, or scaled without touching any other pair, and a scored run depends on nothing outside its own container except the model provider's API. Build scripts, cluster configuration, and the rest of the operational detail live in mivas-bench.