Designing for voice
Getting a call working is the quick part. Getting one people use twice is the rest of the work, and almost none of it is about the API. Speech has no scrollback. A user can interrupt you mid-word. The screen is right there and it holds detail the ear cannot.
What follows is the whole argument, written from running production voice agents rather than from first principles. It is one page because each of these ideas is the same idea from another side, and reading them apart made that harder to see.
Voice points, the screen holds
Section titled “Voice points, the screen holds”Speech is serial, it cannot be skimmed, and it is gone the moment it is played. Nine product codes read aloud are nine things the user has forgotten by the time the tenth arrives. The screen is random-access and it persists, and it cannot direct attention on its own.
So the division: voice carries intent, acknowledgement, and the one number that matters. The screen carries the record. A number the user must hold in their head belongs on the screen.
Speech holds the floor; an action does not
Section titled “Speech holds the floor; an action does not”A brain reaches the user by speaking and by dispatching an action, and the difference between them is audio.
Speech is yielded from the turn. SpeechStart, SpeechChunk, SpeechEnd are the
only yieldable types, and a unit of speech is one thing the
user can be interrupted out of.
An action is dispatched, never yielded. It carries no audio, so it holds no
floor, and session.dispatch(action) is callable from inside a turn, from a
callback that is not a generator at all, and from work that finished long after
the turn that started it. The mechanism is documented at
actions; this page is about what to send and when.
That asymmetry is the whole design. Speaking is a turn and costs the user time. Rendering costs them nothing, so it can happen at any moment, including while the agent is saying something else.
Speak the pointer, render the payload
Section titled “Speak the pointer, render the payload”All of these are running today:
| Shape | What voice says |
|---|---|
| Options on a row the user can tap | ”drops or ointment?” — never the prices |
| A video seeked to the second that answers the question | one line pointing at the screen |
| A ring drawn around the field in question | the field’s name, and nothing else |
orderdesk states the rule in its own prompt, because a text-to-speech engine
reads pharmaceutical brand names badly and the screen spells them correctly:
“NEVER read out a list of options, prices, pack sizes or SKU codes. Ever. That
is what the screen is for”. aura plays
its own how-to video muted and seeked to the right chapter while the agent
narrates over it, and carries the chapter map in the prompt marked as material
for choosing a timestamp rather than a script to read out.
An action is a shape, and the shape is the contract
Section titled “An action is a shape, and the shape is the contract”Subclass Action and the fields are the payload. The class name becomes the wire
name — OpenItinerary becomes open_itinerary — and can be pinned with name=
when you would rather the class be free to move.
Every declared field is emitted, including the ones that are None. The wire
shape of an action is a function of the class rather than of which fields
happened to be set on this call, so the browser declares one total interface and
handles one shape. JSON Schema export is what makes the TypeScript half
generatable rather than hand-copied (TypeScript types).
Inside a turn, a dispatch reaches the wire in the order it runs, so it cannot jump ahead of speech already yielded. An action that belongs with a sentence goes next to that sentence.
The failure this prevents is invisible
Section titled “The failure this prevents is invisible”An agent that narrates its own screen produces a correct transcript, a completed call, and a satisfied dashboard. Every instrument reports success. The only symptom is that the call took twice as long as it needed to, because the slow channel repeated what the fast one had already delivered — and no metric you have distinguishes that from a user who had more to say.
The proxy that does see it is the interruption rate. Users talk over an agent that is reading things to them.
The turn budget
Section titled “The turn budget”Between a user finishing a sentence and hearing the first syllable back, this happens in order:
- Voqalize decides the user has stopped and finalizes the recognizer’s text.
- Your callback runs, until it yields its first
SpeechChunk. - Voqalize synthesizes that chunk and plays it out.
The middle interval is the only latency in the product your code controls. It has a measurable name — time to first chunk — and it has no idle time you can reclaim later. If the user waited, they waited.
So a fast turn is not a turn that finishes quickly. It is a turn that starts quickly. After the first chunk you have the user’s attention and the rest of your generation runs underneath it; before the first chunk you have a person listening to nothing.
greet is not a turn
Section titled “greet is not a turn”A model round trip in greet is the most expensive latency in the product: the
user is connected and hearing nothing. Return a fixed line or a template
(The greeting is a string),
or None to open silently, which is what an ambient agent wants.
The demos open with a one-word hello in the session’s language followed by a written opener, so the user hears a syllable the moment the session connects. Anything that needs a model belongs in the first turn.
Where your half actually goes
Section titled “Where your half actually goes”Time to the model’s first token, not its last. Awaiting a completion before speaking converts a streaming system into a batch one. Stream chunks as you produce them.
Tool round trips. A tool’s result reaches the model with the next request,
which is normally the user’s next message, so a tool costs the turn nothing but
its own run time. A tool marked @needs_result_now costs a whole model round
trip: the model is asked again, in silence, before it can finish its reply. A
network call inside a tool is a round trip the user waits through — start it in
the background instead. See tool design.
Retrieval as a serial hop, unless it was started before it was needed — see parallel workstreams.
Work that was serial for no reason. Three sequential awaits cost the sum;
one asyncio.gather costs the slowest.
Prompt size and cache misses, which is prompt design.
Thinking budget is a latency setting
Section titled “Thinking budget is a latency setting”A reasoning budget on a voice turn is spent in silence the user sits through,
and thought tokens are never spoken, so the cost has no audible half at all. The
SDK’s VOICE_THINKING asks for the least thinking the default model allows.
The right setting is model-specific: models differ in which thinking levels they accept, whether their lowest level still spends thought tokens, and how reliably they call tools at a low level. Measure on real turns whenever you change the model.
Speak first, and call in the same response
Section titled “Speak first, and call in the same response”The SDK’s model-backed brains (GeminiBrain, OpenAIBrain, LiteLLMBrain) ask
the model once per turn, plus once more after each @needs_result_now tool
returns. The model speaks, calls its tools in the same response, and the turn
ends when it stops. Each tool runs as its call arrives, after the speech ahead of
it has been sent, so the screen changes while the line is playing.
So the line said with the call is the reply, not a promise to report back:
nothing after the call is spoken unless the tool is marked @needs_result_now.
The wait before that line is already covered: with ack on (the default),
Voqalize acknowledges the user when their turn ends, so the line carries
substance (what you are doing or showing) and never filler like “Let me check”.
legal puts it in its prompt: “EVERY RESPONSE STARTS WITH WORDS. Write your one
short line first, then make the call, in that same response”. The rule is
written down once, in Tools.
A model that calls a tool and says nothing leaves the user listening to silence
until they speak again. If a user turn sends no SpeechChunk within 15 seconds,
Voqalize says a line of its own
about the wait. The fix is the prompt, or a line your brain speaks itself when a
call arrives with no speech ahead of it; the SDK does not fill the silence for
you, because a silent call can be the right answer when the screen already is
one.
Instruments you already have
Section titled “Instruments you already have”on_finalize fires once for every speech unit you opened, after playout,
carrying what was heard and whether it was interrupted. Stamp a monotonic clock at callback entry and
close it at your first SpeechChunk; that is your half of the budget, measured on
every turn, in your own process.
The model-backed brains log one turn: line per turn with that interval, its model round
trips and its tool calls. speechless=yes on it is a turn that called tools and
said nothing — the silence above, counted.
Interruption rate is the cheapest quality proxy in the product. Users talk over an agent that is too slow, too long, or wrong, and those are hard to tell apart from a transcript and easy to tell apart from a clock.
get_call_record over the MCP server returns Voqalize’s half
of the same call, joined on the same session_id — each turn’s marks and
the session’s pace, and every unit the user cut short.
There are no latency numbers on this page
Section titled “There are no latency numbers on this page”Voqalize publishes no latency figures for its half yet. This section names the moments instead, and the instruments above measure your half, which is the half you can change.
Interruption and heard truth
Section titled “Interruption and heard truth”Your brain yields three sentences. The user cuts in after the first one.
What the user knows is one sentence. What your code produced is three. If the three go into history, every later turn is planned against a version of the call that never happened — the agent thanks the user for a detail they never gave, or declines to repeat something it believes it already covered.
Nothing catches this. There is no error, no dropped frame, no latency spike, no failed eval. The transcript your own code wrote agrees with itself. The only artifact that disagrees is the recording, and nobody plays the recording of a call that went fine.
Interruption is the normal case
Section titled “Interruption is the normal case”On any turn longer than a sentence, a user who cuts in is a user working correctly: they got what they needed and moved on. Building for it as an exception gets the arithmetic backwards — the uninterrupted turn is the one worth treating as a special case, because it is the one where the user had nothing to add.
on_finalize hands you the prefix
Section titled “on_finalize hands you the prefix”After a unit of speech finishes playing, one callback fires:
async def on_finalize(self, session: Session, fin: Finalize) -> None: ...Finalize carries speech_id, heard and generated.
heard is the delivered prefix — the text that reached the user’s ear, not
the text you yielded. generated is the text you yielded, kept by the SDK so the
two can be set side by side; fin.interrupted is that comparison and nothing
more. This is the
one place Voqalize knows something your process cannot: playout happens on
Voqalize’s side of the wire, so where the audio actually stopped is Voqalize’s
to report and yours to record.
The callback fires long after the generator that produced the speech has returned. That ordering is the point — the truth about a unit is not available while it is still playing.
The obligation, stated plainly
Section titled “The obligation, stated plainly”Whatever you persist as the assistant’s turn — a Gemini Content, a row in your
own table, a line in a log — is built from fin.heard. Not from the string you
yielded, and not from the accumulated chunks.
The same lie twice:
- History for the next model call. The model plans its next turn against what the user knows.
- Everything downstream of the transcript — summaries, QA scoring, handoff notes, the “what did we tell this customer” audit that someone runs six months later during a dispute.
The downstream record is worse, because by then the recording is gone.
Not everything that makes noise is an interruption
Section titled “Not everything that makes noise is an interruption”A user who says “mm-hm” while your agent is talking is agreeing, not cutting in, and an agent that stops dead for every backchannel is unusable. So barge-in is gated on sustained voiced speech: while the agent is speaking, the user must speak for about 600 ms before the agent stops. A cough, a phantom detection or an “mm-hm” stays under the bar; a real interruption crosses it.
The bar drops to zero in two cases: when nothing is playing, and for the first three seconds after the user’s own turn ended, so a user who pauses and carries on is heard as continuing rather than as interrupting.
What follows are design constraints on what you write rather than knobs you can turn:
A short imperative may not land. “Stop” shouted over a long answer is exactly the shape that stays below the bar: over before 600 ms of speech. That is the accepted cost of not stopping for “mm-hm”, and it is a reason to keep a spoken answer short enough that the user does not need to shout it down.
An idle user is a different case. When your agent is not speaking, any speech starts a turn instantly — the bar applies only to cutting off speech in progress.
A tool call is invisible to Voqalize. Your brain’s tools run in your process, between the speech it sends, so nothing mutes the user while one runs and a barge-in can land at any point in it. Speech the brain sent ahead of the call is what the user is hearing while the tool runs, and that speech is what they interrupt.
What happens to a turn that gets cut
Section titled “What happens to a turn that gets cut”Voqalize sends an interruption naming the last turn it applies to. The SDK marks that number as a watermark, and every turn at or below it is dead: its task is cancelled, and any chunk still in flight from it is discarded rather than spoken.
Consequences worth knowing:
Your finally runs. The generator is closed, not abandoned — the SDK calls
aclose(), so cleanup, span exits and released locks all execute on an
interrupted turn exactly as on a completed one.
Turn ids fence the turns from each other. The turn that replaces a cancelled one carries a higher number, and a watermark never rises to reach it. Late work from a dead turn cannot leak a sentence into its successor.
What interruption does not undo
Section titled “What interruption does not undo”An action already dispatched has already arrived. The screen does not roll back when the user cuts in, and it should not: the itinerary they are looking at is the itinerary they asked for, whether or not the sentence describing it finished.
Work a tool started in the background is not cancelled either. A charge that was authorized was authorized. The cut turn’s own task is cancelled, so a tool still awaiting inside it stops at that await — one more reason a tool returns at once and hands slow work to a task of its own. If that work must not outlive its turn, that is a decision for your code to make, and it needs its own idempotency rather than a hope about timing.
Testing it without a microphone
Section titled “Testing it without a microphone”The conformance harness models playout and heard-truth finalization the way
Voqalize does, so heard truth is an assertion your brain passes rather than a
behaviour described on a page. VoqalizeDriver records what was delivered per
unit, which means a test can assert on the heard text instead of the generated
text — and the strongest property available is the one worth asserting:
the history your brain wrote equals what the driver says was heard.
See Test the brain.
How exact heard is
Section titled “How exact heard is”heard is a prefix of your text, and playout can be cut in the middle of a
word. Whether that prefix is character-exact or rounds to a boundary is not
specified. Treat heard as the account of what was delivered, and do not build
anything that depends on its last character.
Misunderstanding and reversal
Section titled “Misunderstanding and reversal”The same things go wrong on every voice deployment, and none of them are going away.
The recognizer mishears. The model misinterprets what it heard correctly. And the user changes their mind halfway through the sentence — which is not an error at all. It is how people talk, and a system that treats it as a fault is a system that argues with its user.
So “how do I prevent mistakes” is the wrong question to build against. The questions that produce a working design are: how fast does a mistake become visible, and how cheap is it to undo?
The answers follow, and withholding the authority that matters is the one that saves you.
Show what the agent believes, including that it is unsure
Section titled “Show what the agent believes, including that it is unsure”Uncertainty is a state to render, rather than a null to hide. orderdesk gives
every row on the cart a status:
LineItemStatus = Literal["multi_family", "multi_variant", "matched", "not_found"]A row appears on screen the moment it is named, before the catalog has answered, and then shows its verdict: matched, one of several brands, one of several variants, or not stocked. The pharmacist can see which rows still need a choice while the rest are already matched, and can settle one by hand without waiting to be asked.
spoken_text and source do the same job at a finer grain. Each row keeps
spoken_text —
the raw heard phrase — beside the SKU it resolved to, so the evidence for a
mistake survives the resolution. Seeing “amlong” next to a row that resolved to
the wrong brand tells the pharmacist immediately whether the recognizer or the
matcher was at fault. And source: "agent" | "manual" records who put the row
there, so a hand correction is distinguishable from the agent’s own work.
None of that is possible if the screen only shows conclusions.
Make correction cheap, by voice and by hand
Section titled “Make correction cheap, by voice and by hand”By voice, correction is a taxonomy of tools rather than one re-add:
| What the user says | Tool | What it preserves |
|---|---|---|
| ”make it three” | set_quantity | the row |
| ”one more” | adjust_quantity | the row, and the distinction between delta and absolute |
| ”the syrup, the tablets” | change_variant | the row and its quantity |
| ”I meant the 40” | refine_item | the row and its history |
| ”drop that” | remove_items | — |
| picking from options | choose | the row |
The shared property is identity. The row stays where it is, so the user’s eye
does not have to re-find it. orderdesk’s prompt gives the reason in five words:
remove and re-add, and “he loses his place on the screen.”
By hand, correction is the same event arriving from the other side. The pharmacist taps a variant pill, edits a quantity, deletes a row, adds something from the search bar — and the page sends the brain an event naming the row and what it now says, and the brain yields to it. That path is a supported one, and it is how the agent notices a correction it never heard.
The instruction that closes the loop is the reciprocal of everything above: “NEVER redo what he already did himself.”
Withhold the authority that matters
Section titled “Withhold the authority that matters”The strongest correction mechanism is not correcting at all. Anything important or irreversible is committed by a human, with a click, and the agent has no tool for it.
orderdesk: “You have no confirm tool and no confirm authority.” The pharmacist presses Confirm; the brain sees the screen sayconfirmedand closes in one line.servicing:submit_packetgoes through after the advisor approves.aura: the sign-in is a sheet the customer authorises on their own screen, and the brain is told when they have.
Why a click and not a spoken “yes.” A spoken yes can be misheard. It can be background noise the recognizer resolved into a word. It can be a genuine yes to a question the user only half-heard, because they started talking over the second half of it — and what the user heard is the part that finished playing, which your agent does not know at the moment it asks. A click has none of those failure modes, and it lands on a screen showing exactly what is being agreed to.
This is also what makes every earlier tool safe. Corrections stay cheap right up to the commit boundary, and the boundary is a human.
Where the story stops
Section titled “Where the story stops”What this section does not cover:
Correcting something already committed. The compensating-call taxonomy above runs up to the confirm click and stops. Whatever undoes a placed order is your system’s problem, and it is a different kind of problem.
All of this assumes a screen. Every mechanism here — visible uncertainty, identity-preserving edits, the click that commits — needs somewhere to render. An agent with no UI has none of them, and has to fall back to spoken confirmation with all the failure modes just listed. If you are building voice-only, the honest version is fewer irreversible actions rather than better confirmations.
Parallel workstreams
Section titled “Parallel workstreams”A form makes you wait for each field. You type, it validates, it renders the next one, and the round trip repeats until the form is done. Speech has no such round trip: a user can name six items in one breath and stop, having spent four seconds on what the form spends four minutes on.
That burst is where the speed comes from. It is also the thing an agent gives back first. Handle the six items one at a time — ask, resolve, confirm, ask again — and you have rebuilt the form, out loud, at the pace of a conversation. Slower than the form it replaced, because now every field costs a spoken turn.
So parallelism is not a later optimization here. Without it the user’s natural behaviour becomes your worst case.
Take the burst whole
Section titled “Take the burst whole”orderdesk’s prompt says it in the imperative, because a model left to itself
will take one item and stop:
The moment he names a product, say a tiny line and call add_items. Do not wait for the previous one to resolve; do not ask a question in between. He can list six items in one breath — take them all in ONE add_items call with a list.
One tool call, six items, six rows. Each row gets its own verdict (matched, or ambiguous between two pack sizes, or not stocked) and only the ambiguous ones need the user. The user is still talking while they appear.
Work that outlives its turn
Section titled “Work that outlives its turn”servicing makes the same shape explicit for long work. prepare_case puts the
workup on screen, where it runs, and returns immediately:
f"preparing {action.ref} in the background{blocker_note}; the workup shows ""on the case when the prep finishes"The return value reports what was started; it is not an order to speak. The line the advisor hears is the one the prompt has the agent say with the call: that the case is under way, and the headline of what it found. The model reads the return value with the advisor’s next message, by which time it has already said that line and carried on.
What makes this safe:
session.dispatch(...) never blocks and is callable from anywhere, including from
a task that outlived the turn that started it. A background job finishing ninety
seconds later can still paint the screen.
An action carries no audio, so it needs no floor. The screen can change while the agent is mid-sentence, and neither one waits for the other.
The real design question is how a result comes back
Section titled “The real design question is how a result comes back”Voice is one serial channel and there are now three finished jobs to report. Reading them out is almost never the answer. The ways back are not equally priced:
| Way | When to use it | Shipped in |
|---|---|---|
| A row changes on screen, silently | The result is legible and unambiguous | orderdesk, forge |
| One short spoken line | It changes what the user should do next | servicing |
| A spoken question | Genuinely ambiguous, and only the user can resolve it | orderdesk |
The default is the silent row change, and forge’s prompt tells the model to
trust it:
Every tool you call also shows up as a live task on screen (a small “activity” checklist), so your actions are already acknowledged visually — trust it and stay quiet.
Speaking costs a turn, so a spoken line and a spoken question are spent, not
spread. When
several rows do turn out to need the user, batch them the way you batched the
intake — orderdesk again:
Batch your questions. Let him finish his run of items, then at the natural pause ask about the ambiguous rows, one short question each.
Nothing blocks, a person’s decision included
Section titled “Nothing blocks, a person’s decision included”Waiting on the user is still waiting. aura’s sign-in puts a sheet on the
customer’s screen and returns; the customer authorises it in their own time, and
the page’s event tells the brain when. The agent keeps talking in between. The
shape is in tool design.
Ordering and failure are yours to decide
Section titled “Ordering and failure are yours to decide”Ordering. Three background jobs finishing at once produce three dispatches in
the order they completed. orderdesk makes each action name one row and one
change to it (RowMatched, RowQuantity, RowQuestion), so a late action
changes only what it names and arrival order between rows stops mattering. That
is the convention to copy.
Failure. Every worked example here succeeds. A fan-out where one branch fails is the case you will hit first in production, and the failed row still has to reach the user through one of the ways above — most often as a row that says so.
Prompt design for voice
Section titled “Prompt design for voice”A chat prompt can afford to be thin. The model can look things up, and the reader watches a spinner while it does — a two-second tool call reads as work happening.
Say the same two seconds out loud. Nothing happens, on a channel where nothing happening is the one thing a user reacts to. So the arithmetic changes: what the agent needs, it should mostly already have.
That is the whole design rule, and it is not “be concise.”
The 80/10/10 target
Section titled “The 80/10/10 target”| Share of what the agent needs | Where it lives | What it costs |
|---|---|---|
| 80% | In the prompt already | tokens |
| 10% | One fast tool call away — in your process, no network, marked @needs_result_now | a model round trip |
| 10% | Genuinely slow — remote, expensive | a background workstream, with an answer to “what does the user hear meanwhile” |
These numbers are a design target, not a measured ratio. Their job is the genuinely slow tenth: anything that lands there needs a plan for the silence, which is parallel workstreams.
What a voice prompt does that a chat prompt need not
Section titled “What a voice prompt does that a chat prompt need not”Talk less, do more. The default reply is one short line, and everything longer is on the screen. The shipped prompts say it in the imperative, because a model’s default register is a paragraph:
orderdesk: “No markdown, no lists, no stage directions. Never narrate your own actions (…) — say only what the pharmacist should hear, and make the call with it.”forge: “your actions are already acknowledged visually — trust it and stay quiet.”sugar: “better, don’t say units at all; the screen shows them.”support: “Never read out ids or order numbers as raw text unless the shopper asks — say “your order from May 28th” instead.”
Hold its prompt still. The system prompt is the cache prefix. Set it once per session and it matches turn after turn; rebuild it — even to append one fresh line — and the provider re-reads the whole thing on every turn, which the user pays for in silence. Volatile context goes at the tail, next to the latest user message, where a change costs the provider only the small new suffix. This is the cheapest latency win available and the easiest to throw away by accident, because nothing in a transcript shows it.
Know where the user is. The agent needs a way to answer “what is on screen
right now,” and the answer has to be current rather than remembered. Both aura and
servicing do the same thing: the page sends an event for each change, the brain
keeps its own mirror of the screen up to date from those events and from its own
actions, and a tool marked @needs_result_now reads the mirror on demand —
aura’s get_screen_context, servicing’s get_advisor_context. The reciprocal instruction matters as much.
The screen is authoritative over the agent’s own memory of it, because the user
has hands: orderdesk’s prompt ends that thought with “NEVER redo what he
already did himself.”
Track a task list. Several threads are open at once and the prompt has to
name them and say how each one closes. servicing’s prompt is explicit that
background prep runs while the advisor keeps working: “prepare the other case
quietly, and in the line you say with prepare_case tell them it is under way and
the headline of what you found; the prep finishes on screen by itself. They are
never blocked.”
Assume it misheard. A recognizer at a busy pharmacy counter will get things wrong. Correction paths belong in the prompt as first-class instructions rather than as a fallback paragraph at the end — see misunderstanding and reversal.
Never leave silence, in the shipped prompts
Section titled “Never leave silence, in the shipped prompts”legal: “nothing a tool returns reaches you before your turn ends, and you get no further word until they speak again. So the line you say with the call is the whole of your reply: make it the substance, not a promise to report back.”aura: “a screen call’s result reaches you only with the customer’s next message, so you get no second word after it. The line you say with the calls is your whole answer: never a promise of more (“let me check…”)”
These are the prompt doing latency work.
A prompt fragment worth copying
Section titled “A prompt fragment worth copying”orderdesk has to disambiguate a spoken drug name against two dozen SKUs, by
voice, in Hindi, without reading a list aloud. Its prompt teaches an
information-theoretic rule in plain language, and the whole block is worth reading
as a model of how specific this gets:
Four or fewer choices need no machinery: the pills are already on his screen, so just ask the one short thing that differs (the rule above).
Five or more, and the tool hands you a CANDIDATE TABLE instead of options. Never read it. Never try to show it all. Call ask_choice ONCE with one short English question and TWO TO FOUR choices that split those candidates as evenly as you can. The sharpest question is the one that eliminates the most candidates WHATEVER he answers — a choice that keeps twenty-three of twenty-four is a wasted turn.
Group on the axis that actually partitions the list: the suffix line first (plain / H / AM / CT / BETA…), then form, then a strength band. Never split on pack size while a bigger axis still divides the list.
TWO ROUNDS AT MOST. Round one cuts twenty-four to a handful; round two is leaf pills he can tap, or he simply says which one and you call choose.
What to take from it: it names the threshold at which machinery starts (five), it gives the model a decision rule rather than an example, and it caps the interaction in turns, because a turn is the unit the user feels.
The wording of the question is the model’s; the shape is not. ask_choice
rejects a set that has fewer than two or more than four choices, or that leaves a
candidate uncovered — and the prompt tells the model what happens when it does, so
a rejection is a retry rather than a dead turn.
Tool design for voice
Section titled “Tool design for voice”A tool call in a chat app is a pause. In a voice call, a pause is dead air. So the SDK’s model-backed brains do not wait on a tool to finish the reply: the model speaks its line, calls the tool in the same response, and the tool’s result reaches the model with the user’s next message. The user hears the line while the tool changes the screen.
The properties that follow are properties of the tool rather than of the prompt around it.
A tool returns immediately
Section titled “A tool returns immediately”A tool returns within 20 ms. The model-backed brains time every call, and one that runs longer is logged as a warning that the user waited for it. Nothing is cancelled or timed out; the warning is how you find it.
servicing’s prepare_case puts minutes of work on screen and returns at once,
with a short report of what it started (work that outlives its
turn). Whenever the work is slower than a
sentence, that is what a voice tool returns: what was started, for the model to
read on its next turn.
A tool that needs the network, or anything else slow, has been mis-split. Break it into a cheap dispatch that starts a background task and returns, and let the result land on screen or in the state your other tools read, the way parallel workstreams describes.
Mark a tool only when the reply needs its result
Section titled “Mark a tool only when the reply needs its result”@needs_result_now marks a tool whose result the model must read before it
finishes this reply:
from voqalize.sdk.gemini import GeminiBrain, needs_result_now
class Desk(GeminiBrain): @property def tools(self): return [self.show_card_controls, self.get_account_balance]
async def show_card_controls(self) -> str: "Put the card controls on screen." self.session.dispatch(ShowCardControls()) return "shown"
@needs_result_now async def get_account_balance(self, args: Account) -> dict[str, str]: "The balance of one of the customer's accounts." return self.accounts[args.number].balance()Mark a tool when it reads data the model needs to answer correctly — a balance, the cart, what is on the screen, an eligibility check — from memory your brain already holds. Leave it off actions, screen changes, sign-in prompts, language switches, and a tool whose result only repeats what the model already said. Unmarked is the default, and the right answer for most tools.
A marked tool makes the model answer again as soon as it returns, so it costs the user a model round trip of silence. The mark does not make a slow tool acceptable: a marked tool returns within the same 20 ms.
What the user hears when the mark is wrong:
- Missing. The agent says its line, calls the tool, and goes quiet until the user speaks again — then answers from the result, a turn late.
- Extra. A pause before every reply that calls the tool.
A tool is not rolled back
Section titled “A tool is not rolled back”Barge-in cancels the turn. It does not un-dispatch an action already sent, and it does not reach work a tool started in the background. A tool that returns at once has finished before the user can cut in; one still awaiting when they do is cancelled at that await, which is one more reason to hand slow work to a task of its own.
This sounds untidy and is correct: the user interrupted the speech. They did not interrupt the lookup, and half-applied work is worse to reason about than completed work. The screen showing what they asked for is right, whether or not the sentence describing it finished.
Tools also run one at a time, in the order the model produced them. Tools racing would put the user’s display in an order the model never asked for, and the screen is the thing the user is reading.
A tool is undone by another tool
Section titled “A tool is undone by another tool”The undo for a voice tool is a compensating call, and the compensating calls are
worth enumerating separately rather than collapsing into a re-add. orderdesk
gives the model an edit tool per operation and shouts why:
A QUANTITY TWEAK IS NEVER A RE-ADD. An absolute number (…) is set_quantity; a relative one (…) is adjust_quantity with a delta — plus ten is 10, “ten less” is -10. If he wants none of it, that is remove_items, not a delta down to zero.
A VARIANT SWAP IS NEVER A RE-ADD EITHER. … Never remove the row and add it again — he loses his place on the screen.
The reason in that second line is the general rule. Each of these preserves the identity of the row it touches, so the display updates in place and the user’s eye keeps its position. A re-add is correct in the database and wrong on the screen.
The premise underneath is that tools are cheap enough to undo this way. A tool expensive enough that you want to abort it mid-flight is the one to split.
A failed call is a result, not an exception
Section titled “A failed call is a result, not an exception”A tool that raises comes back to the model as {"error": …}, with the
exception’s message, and the model-backed brains log tool … failed in your process. A
call to a tool that is not declared this turn answers the same way. The model
reads it like any other result and can call again.
It reads it when it reads any other result. For an unmarked tool that is the
user’s next message, after the line that announced the call has already played.
An agent cheerfully telling a user their order is placed, because the failure
came back later as {"error": …} and looked like data, is the shape of the worst
bug in this category — and the log line is the only place it shows on your side.
Use the same seam for validation. orderdesk’s ask_choice is rejected unless it
has two to four choices, uses known codes, and covers every candidate; a separate
validator rejects non-Latin labels headed for a screen that must stay Latin.
These come back as retriable errors, and the prompt warns the model in advance that they
can. So the shape of the question is guaranteed even though its wording is the
model’s.
A tool never holds irreversible authority
Section titled “A tool never holds irreversible authority”orderdesk’s prompt draws the line in one sentence:
You have no confirm tool and no confirm authority. Your job is a fully matched cart: every row green, every quantity set.
The pharmacist presses Confirm. servicing is the same shape — submit_packet
goes through after the advisor approves. The agent gets the work to the edge of
the commit and stops there, which is also what makes every earlier tool safe to
undo.
Some events must not take the floor
Section titled “Some events must not take the floor”orderdesk answers the manual search bar’s catalog_searched and the
variants_opened event floor-free: it dispatches the results as an action,
with no model call and no speech. The user is typing in a search box; a
keystroke must not make the agent start talking over them.
If something exists to serve the screen rather than the conversation, answer it without taking the floor. Anything that can be triggered by a tap or a keystroke belongs in this category.
No tool waits for the user
Section titled “No tool waits for the user”A tool that needs the user — to sign in, to approve, to pick — puts the request on
screen, says so, and returns. aura’s show_auth_popup returns as soon as the
sheet is up:
The sign-in is on screen and the customer is NOT signed in yet. You will be told when they authorise it, and handed an authenticated_context then.
The user’s answer arrives as an event from the page, which your brain appends to
the context; it is never the tool’s return value. A tool that depends on it takes
a parameter only that answer can supply — aura’s account tools take the
authenticated_context the brain mints when the sign-in completes — so the model
cannot invent it, and a call without it is refused with a reply that names the
missing step.
The checklist
Section titled “The checklist”- Returns within 20 ms, starting anything slower in the background.
- Marked
@needs_result_nowonly if the reply needs what it reads from memory. - Typed arguments; bad ones come back as an error the model can read.
- Has an undo that is another tool call.
- Preserves the identity of what it touched, so the screen does not jump.
- Holds no authority over anything irreversible.
- Never waits on the user; the user’s answer arrives as an event.
When you are done here
Section titled “When you are done here”You have an application worth putting in front of people. Keeping it working is Operate.