Test the brain
You can test every change to a brain without a microphone. The SDK ships
voqalize.conformance: a fake Voqalize that speaks
the wire. It hosts your real Brain on a real WebSocket,
mints a real brain-connection token,
and models playout and heard-truth the way
Voqalize does — and lets you drive it in text mode. user_says("…") in, a Turn with
.text out.
That makes a voice agent testable like any other service, and it is the eval primitive: a scenario is a conversation script plus assertions. Keep a scenario file per use case in your repo and run it in CI.
Set it up
Section titled “Set it up”from voqalize.conformance import ( DirectConnection, VoqalizeDriver, brain_server, generate_keypair, mint_voqalize_token,)from mybrain import MyBrain
keypair = generate_keypair()
async with brain_server(MyBrain, public_keys=keypair.public_pem) as server: token = mint_voqalize_token( private_key_pem=keypair.private_pem, session_id="s1", agent_id="agent_test", tenant_id="tenant_test", ) driver = VoqalizeDriver( DirectConnection(server.url, "s1", token=token), session_id="s1", default_timeout=10.0, ) await driver.open()brain_server binds an ephemeral port, so tests never collide, and closes on the
way out whatever the test did. It is a test server — production hosting is
run_session in your own web framework’s route (see
Deploy the brain); the SDK owns no server.
The brain verifies against the public half of the keypair and the driver signs with the private half, so token verification runs for real rather than being switched off.
Drive it
Section titled “Drive it”| Call | Drives | Returns |
|---|---|---|
start_session(init={…}) | SessionStart, which is turn 1; plays out the greeting. init reaches the brain as session.init. | Turn | None |
user_says("…") | One user turn, played out and finalized. | Turn |
barge_in("…") | Start a turn, let the brain speak, interrupt, finalize the cut with partial heard-truth. | Turn |
user_idle(level=1, idle_ms=30000) | An idle trigger; plays out on_user_idle. | Turn |
send_ui_event(event, payload) / send_rtvi(type, data) | One app→brain RTVI message, delivered to on_rtvi. That callback cannot speak and opens no turn, so there is nothing to wait for. | — |
collect_ui_commands(min_count=1) | Waits for and returns the ui-command bodies the brain fired — {"command": …, "payload": {…}}. | list[dict] |
end_session() / send_cancel() / aclose() | End, Cancel, teardown. | — |
Assert on it
Section titled “Assert on it”A Turn carries .turn_id, .text (everything spoken this turn), .completed,
.interrupted, .heard (for a barge-in: the partial the user actually heard), and
.units — one entry per speech unit, each with .speech_id, .text, .spoke and
.ended.
.completed is the most valuable single assertion — a false there usually means the
brain hung or raised.
async def test_answers(driver): await driver.start_session() turn = await driver.user_says("Add two oat milks to my cart.") assert turn.completed assert "oat milk" in turn.text.lower()
cmds = await driver.collect_ui_commands(min_count=1) add = next(c for c in cmds if c["command"] == "add_to_cart") assert add["payload"] == {"sku": "oat-milk", "qty": 2}The driver also accumulates driver.ui_commands, driver.errors and
driver.requests — every configure_* the brain made, in wire order — so “did the
brain switch to Hindi when asked?” is an assertion on driver.requests, not a
listening exercise. Set driver.reject[op] = "reason" to make Voqalize refuse one, and
driver.withhold.add(op) to make it never answer at all.
If the brain calls a real LLM, tests get slow and flaky for the usual reasons. Inject
a scripted fake (brain=lambda: MyBrain(llm=FakeLLM())) and keep one slow
test against the real model as a smoke check.
The built-in conformance suite
Section titled “The built-in conformance suite”Beyond your own scenarios, the harness ships a catalog of 17 scenarios: the bar a brain must clear to be wire-compatible: greeting, turn- and speech-id monotonicity, bracket integrity, the barge-in watermark, heard-truth reconciliation across multiple interruptions, action-outcome correlation, RTVI delivery, idle re-engagement, and bad-token rejection.
# Point it at any brain that speaks the wire.python -m voqalize.conformance --brain-url ws://127.0.0.1:8787 --private-key ./voqalize_dev.pem
# Prove the harness itself: host the bundled reference brain, run everything.python -m voqalize.conformance --self-testAdd --no-auth instead of --private-key if the brain runs allow_unverified;
the auth scenarios are then skipped, because an unverified brain has no bad token
to reject.
Most of them need a cooperating brain — one that speaks a private
command grammar (say banana, count slowly) and echoes its committed state back,
which is what voqalize.conformance.reference.ConformanceBrain is for. Yours
doesn’t, and shouldn’t. The suite probes for that grammar on connect and skips
what can’t apply, naming the reason and qualifying the verdict:
[PASS] greeting (299 ms) [PASS] single_turn (299 ms) [PASS] multi_turn (300 ms) [SKIP] two_units_one_turn [SKIP] barge_in … [PASS] reject_bad_token (86 ms) …
13 skipped: needs the reference command grammar — this brain answered the probe with its own words, which is what any real brain does
4 passed, 0 failed, 13 skipped — CONFORMANT on what ran (4 of 17 scenarios; see the skips above)That is the honest result for an ordinary brain: the wire-level tier is what the
suite can prove about it, and the verdict says so rather than claiming more. Force
the probe either way with --reference / --no-reference if you need to.
--only name1,name2 restricts the run; the exit code is 0 iff nothing failed
— skips don’t fail a build, so this is safe in CI. Programmatically it’s
run_suite(brain_url, private_key_pem=…) → a Report with .ok, .passed,
.failed, .skipped, .summary().
The deep tier is worth reaching for once your own scenarios are in place: to run
it, your brain needs a test mode that answers the grammar and answers a
client-message named by voqalize.conformance.CONFORMANCE_DUMP_EVENT with its
conversation. Read
ConformanceBrain for the shape — the cooperation is small, and it buys you
heard-truth reconciliation across multiple interruptions.
When a live call misbehaves
Section titled “When a live call misbehaves”Read it back as Reading a call back shows: the call record (what was said and heard, turn by turn), then the logs. When a live call fails in a way the offline suite passed, that gap is the next scenario: reproduce it here first, then fix it.
Read next
Section titled “Read next”- Interruption and heard truth: what the harness models when it replays a barge-in.
- Deploy the brain: put the tested brain where Voqalize can reach it.
- Reading a call back: the record, logs and events of a real call.