Skip to content

Designing for voice

Getting a call working is the quick part. Getting one people use twice is the rest of the work, and almost none of it is about the API. Speech has no scrollback. A user can interrupt you mid-word. The screen is right there and it holds detail the ear cannot.

What follows is the whole argument, written from running production voice agents rather than from first principles. It is one page because each of these ideas is the same idea from another side, and reading them apart made that harder to see.

Speech is serial, it cannot be skimmed, and it is gone the moment it is played. Nine product codes read aloud are nine things the user has forgotten by the time the tenth arrives. The screen is random-access and it persists, and it cannot direct attention on its own.

So the division: voice carries intent, acknowledgement, and the one number that matters. The screen carries the record. A number the user must hold in their head belongs on the screen.

Speech holds the floor; an action does not

Section titled “Speech holds the floor; an action does not”

A brain reaches the user by speaking and by dispatching an action, and the difference between them is audio.

Speech is yielded from the turn. SpeechStart, SpeechChunk, SpeechEnd are the only yieldable types, and a unit of speech is one thing the user can be interrupted out of.

An action is dispatched, never yielded. It carries no audio, so it holds no floor, and session.dispatch(action) is callable from inside a turn, from a callback that is not a generator at all, and from work that finished long after the turn that started it. The mechanism is documented at actions; this page is about what to send and when.

That asymmetry is the whole design. Speaking is a turn and costs the user time. Rendering costs them nothing, so it can happen at any moment, including while the agent is saying something else.

All of these are running today:

ShapeWhat voice says
Options on a row the user can tap”drops or ointment?” — never the prices
A video seeked to the second that answers the questionone line pointing at the screen
A ring drawn around the field in questionthe field’s name, and nothing else

orderdesk states the rule in its own prompt, because a text-to-speech engine reads pharmaceutical brand names badly and the screen spells them correctly: “NEVER read out a list of options, prices, pack sizes or SKU codes. Ever. That is what the screen is for”. aura plays its own how-to video muted and seeked to the right chapter while the agent narrates over it, and carries the chapter map in the prompt marked as material for choosing a timestamp rather than a script to read out.

An action is a shape, and the shape is the contract

Section titled “An action is a shape, and the shape is the contract”

Subclass Action and the fields are the payload. The class name becomes the wire name — OpenItinerary becomes open_itinerary — and can be pinned with name= when you would rather the class be free to move.

Every declared field is emitted, including the ones that are None. The wire shape of an action is a function of the class rather than of which fields happened to be set on this call, so the browser declares one total interface and handles one shape. JSON Schema export is what makes the TypeScript half generatable rather than hand-copied (TypeScript types).

Inside a turn, a dispatch reaches the wire in the order it runs, so it cannot jump ahead of speech already yielded. An action that belongs with a sentence goes next to that sentence.

An agent that narrates its own screen produces a correct transcript, a completed call, and a satisfied dashboard. Every instrument reports success. The only symptom is that the call took twice as long as it needed to, because the slow channel repeated what the fast one had already delivered — and no metric you have distinguishes that from a user who had more to say.

The proxy that does see it is the interruption rate. Users talk over an agent that is reading things to them.

Between a user finishing a sentence and hearing the first syllable back, this happens in order:

  1. Voqalize decides the user has stopped and finalizes the recognizer’s text.
  2. Your callback runs, until it yields its first SpeechChunk.
  3. Voqalize synthesizes that chunk and plays it out.

The middle interval is the only latency in the product your code controls. It has a measurable name — time to first chunk — and it has no idle time you can reclaim later. If the user waited, they waited.

So a fast turn is not a turn that finishes quickly. It is a turn that starts quickly. After the first chunk you have the user’s attention and the rest of your generation runs underneath it; before the first chunk you have a person listening to nothing.

A model round trip in greet is the most expensive latency in the product: the user is connected and hearing nothing. Return a fixed line or a template (The greeting is a string), or None to open silently, which is what an ambient agent wants.

The demos open with a one-word hello in the session’s language followed by a written opener, so the user hears a syllable the moment the session connects. Anything that needs a model belongs in the first turn.

Time to the model’s first token, not its last. Awaiting a completion before speaking converts a streaming system into a batch one. Stream chunks as you produce them.

Tool round trips. A tool’s result reaches the model with the next request, which is normally the user’s next message, so a tool costs the turn nothing but its own run time. A tool marked @needs_result_now costs a whole model round trip: the model is asked again, in silence, before it can finish its reply. A network call inside a tool is a round trip the user waits through — start it in the background instead. See tool design.

Retrieval as a serial hop, unless it was started before it was needed — see parallel workstreams.

Work that was serial for no reason. Three sequential awaits cost the sum; one asyncio.gather costs the slowest.

Prompt size and cache misses, which is prompt design.

A reasoning budget on a voice turn is spent in silence the user sits through, and thought tokens are never spoken, so the cost has no audible half at all. The SDK’s VOICE_THINKING asks for the least thinking the default model allows.

The right setting is model-specific: models differ in which thinking levels they accept, whether their lowest level still spends thought tokens, and how reliably they call tools at a low level. Measure on real turns whenever you change the model.

Speak first, and call in the same response

Section titled “Speak first, and call in the same response”

The SDK’s model-backed brains (GeminiBrain, OpenAIBrain, LiteLLMBrain) ask the model once per turn, plus once more after each @needs_result_now tool returns. The model speaks, calls its tools in the same response, and the turn ends when it stops. Each tool runs as its call arrives, after the speech ahead of it has been sent, so the screen changes while the line is playing.

So the line said with the call is the reply, not a promise to report back: nothing after the call is spoken unless the tool is marked @needs_result_now. The wait before that line is already covered: with ack on (the default), Voqalize acknowledges the user when their turn ends, so the line carries substance (what you are doing or showing) and never filler like “Let me check”. legal puts it in its prompt: “EVERY RESPONSE STARTS WITH WORDS. Write your one short line first, then make the call, in that same response”. The rule is written down once, in Tools.

A model that calls a tool and says nothing leaves the user listening to silence until they speak again. If a user turn sends no SpeechChunk within 15 seconds, Voqalize says a line of its own about the wait. The fix is the prompt, or a line your brain speaks itself when a call arrives with no speech ahead of it; the SDK does not fill the silence for you, because a silent call can be the right answer when the screen already is one.

on_finalize fires once for every speech unit you opened, after playout, carrying what was heard and whether it was interrupted. Stamp a monotonic clock at callback entry and close it at your first SpeechChunk; that is your half of the budget, measured on every turn, in your own process.

The model-backed brains log one turn: line per turn with that interval, its model round trips and its tool calls. speechless=yes on it is a turn that called tools and said nothing — the silence above, counted.

Interruption rate is the cheapest quality proxy in the product. Users talk over an agent that is too slow, too long, or wrong, and those are hard to tell apart from a transcript and easy to tell apart from a clock.

get_call_record over the MCP server returns Voqalize’s half of the same call, joined on the same session_id — each turn’s marks and the session’s pace, and every unit the user cut short.

Voqalize publishes no latency figures for its half yet. This section names the moments instead, and the instruments above measure your half, which is the half you can change.

Your brain yields three sentences. The user cuts in after the first one.

What the user knows is one sentence. What your code produced is three. If the three go into history, every later turn is planned against a version of the call that never happened — the agent thanks the user for a detail they never gave, or declines to repeat something it believes it already covered.

Nothing catches this. There is no error, no dropped frame, no latency spike, no failed eval. The transcript your own code wrote agrees with itself. The only artifact that disagrees is the recording, and nobody plays the recording of a call that went fine.

On any turn longer than a sentence, a user who cuts in is a user working correctly: they got what they needed and moved on. Building for it as an exception gets the arithmetic backwards — the uninterrupted turn is the one worth treating as a special case, because it is the one where the user had nothing to add.

After a unit of speech finishes playing, one callback fires:

async def on_finalize(self, session: Session, fin: Finalize) -> None:
...

Finalize carries speech_id, heard and generated.

heard is the delivered prefix — the text that reached the user’s ear, not the text you yielded. generated is the text you yielded, kept by the SDK so the two can be set side by side; fin.interrupted is that comparison and nothing more. This is the one place Voqalize knows something your process cannot: playout happens on Voqalize’s side of the wire, so where the audio actually stopped is Voqalize’s to report and yours to record.

The callback fires long after the generator that produced the speech has returned. That ordering is the point — the truth about a unit is not available while it is still playing.

Whatever you persist as the assistant’s turn — a Gemini Content, a row in your own table, a line in a log — is built from fin.heard. Not from the string you yielded, and not from the accumulated chunks.

The same lie twice:

  • History for the next model call. The model plans its next turn against what the user knows.
  • Everything downstream of the transcript — summaries, QA scoring, handoff notes, the “what did we tell this customer” audit that someone runs six months later during a dispute.

The downstream record is worse, because by then the recording is gone.

Not everything that makes noise is an interruption

Section titled “Not everything that makes noise is an interruption”

A user who says “mm-hm” while your agent is talking is agreeing, not cutting in, and an agent that stops dead for every backchannel is unusable. So barge-in is gated on sustained voiced speech: while the agent is speaking, the user must speak for about 600 ms before the agent stops. A cough, a phantom detection or an “mm-hm” stays under the bar; a real interruption crosses it.

The bar drops to zero in two cases: when nothing is playing, and for the first three seconds after the user’s own turn ended, so a user who pauses and carries on is heard as continuing rather than as interrupting.

What follows are design constraints on what you write rather than knobs you can turn:

A short imperative may not land. “Stop” shouted over a long answer is exactly the shape that stays below the bar: over before 600 ms of speech. That is the accepted cost of not stopping for “mm-hm”, and it is a reason to keep a spoken answer short enough that the user does not need to shout it down.

An idle user is a different case. When your agent is not speaking, any speech starts a turn instantly — the bar applies only to cutting off speech in progress.

A tool call is invisible to Voqalize. Your brain’s tools run in your process, between the speech it sends, so nothing mutes the user while one runs and a barge-in can land at any point in it. Speech the brain sent ahead of the call is what the user is hearing while the tool runs, and that speech is what they interrupt.

Voqalize sends an interruption naming the last turn it applies to. The SDK marks that number as a watermark, and every turn at or below it is dead: its task is cancelled, and any chunk still in flight from it is discarded rather than spoken.

Consequences worth knowing:

Your finally runs. The generator is closed, not abandoned — the SDK calls aclose(), so cleanup, span exits and released locks all execute on an interrupted turn exactly as on a completed one.

Turn ids fence the turns from each other. The turn that replaces a cancelled one carries a higher number, and a watermark never rises to reach it. Late work from a dead turn cannot leak a sentence into its successor.

An action already dispatched has already arrived. The screen does not roll back when the user cuts in, and it should not: the itinerary they are looking at is the itinerary they asked for, whether or not the sentence describing it finished.

Work a tool started in the background is not cancelled either. A charge that was authorized was authorized. The cut turn’s own task is cancelled, so a tool still awaiting inside it stops at that await — one more reason a tool returns at once and hands slow work to a task of its own. If that work must not outlive its turn, that is a decision for your code to make, and it needs its own idempotency rather than a hope about timing.

The conformance harness models playout and heard-truth finalization the way Voqalize does, so heard truth is an assertion your brain passes rather than a behaviour described on a page. VoqalizeDriver records what was delivered per unit, which means a test can assert on the heard text instead of the generated text — and the strongest property available is the one worth asserting: the history your brain wrote equals what the driver says was heard.

See Test the brain.

heard is a prefix of your text, and playout can be cut in the middle of a word. Whether that prefix is character-exact or rounds to a boundary is not specified. Treat heard as the account of what was delivered, and do not build anything that depends on its last character.

The same things go wrong on every voice deployment, and none of them are going away.

The recognizer mishears. The model misinterprets what it heard correctly. And the user changes their mind halfway through the sentence — which is not an error at all. It is how people talk, and a system that treats it as a fault is a system that argues with its user.

So “how do I prevent mistakes” is the wrong question to build against. The questions that produce a working design are: how fast does a mistake become visible, and how cheap is it to undo?

The answers follow, and withholding the authority that matters is the one that saves you.

Show what the agent believes, including that it is unsure

Section titled “Show what the agent believes, including that it is unsure”

Uncertainty is a state to render, rather than a null to hide. orderdesk gives every row on the cart a status:

LineItemStatus = Literal["multi_family", "multi_variant", "matched", "not_found"]

A row appears on screen the moment it is named, before the catalog has answered, and then shows its verdict: matched, one of several brands, one of several variants, or not stocked. The pharmacist can see which rows still need a choice while the rest are already matched, and can settle one by hand without waiting to be asked.

spoken_text and source do the same job at a finer grain. Each row keeps spoken_text — the raw heard phrase — beside the SKU it resolved to, so the evidence for a mistake survives the resolution. Seeing “amlong” next to a row that resolved to the wrong brand tells the pharmacist immediately whether the recognizer or the matcher was at fault. And source: "agent" | "manual" records who put the row there, so a hand correction is distinguishable from the agent’s own work.

None of that is possible if the screen only shows conclusions.

Make correction cheap, by voice and by hand

Section titled “Make correction cheap, by voice and by hand”

By voice, correction is a taxonomy of tools rather than one re-add:

What the user saysToolWhat it preserves
”make it three”set_quantitythe row
”one more”adjust_quantitythe row, and the distinction between delta and absolute
”the syrup, the tablets”change_variantthe row and its quantity
”I meant the 40”refine_itemthe row and its history
”drop that”remove_items—
picking from optionschoosethe row

The shared property is identity. The row stays where it is, so the user’s eye does not have to re-find it. orderdesk’s prompt gives the reason in five words: remove and re-add, and “he loses his place on the screen.”

By hand, correction is the same event arriving from the other side. The pharmacist taps a variant pill, edits a quantity, deletes a row, adds something from the search bar — and the page sends the brain an event naming the row and what it now says, and the brain yields to it. That path is a supported one, and it is how the agent notices a correction it never heard.

The instruction that closes the loop is the reciprocal of everything above: “NEVER redo what he already did himself.”

The strongest correction mechanism is not correcting at all. Anything important or irreversible is committed by a human, with a click, and the agent has no tool for it.

  • orderdesk: “You have no confirm tool and no confirm authority.” The pharmacist presses Confirm; the brain sees the screen say confirmed and closes in one line.
  • servicing: submit_packet goes through after the advisor approves.
  • aura: the sign-in is a sheet the customer authorises on their own screen, and the brain is told when they have.

Why a click and not a spoken “yes.” A spoken yes can be misheard. It can be background noise the recognizer resolved into a word. It can be a genuine yes to a question the user only half-heard, because they started talking over the second half of it — and what the user heard is the part that finished playing, which your agent does not know at the moment it asks. A click has none of those failure modes, and it lands on a screen showing exactly what is being agreed to.

This is also what makes every earlier tool safe. Corrections stay cheap right up to the commit boundary, and the boundary is a human.

What this section does not cover:

Correcting something already committed. The compensating-call taxonomy above runs up to the confirm click and stops. Whatever undoes a placed order is your system’s problem, and it is a different kind of problem.

All of this assumes a screen. Every mechanism here — visible uncertainty, identity-preserving edits, the click that commits — needs somewhere to render. An agent with no UI has none of them, and has to fall back to spoken confirmation with all the failure modes just listed. If you are building voice-only, the honest version is fewer irreversible actions rather than better confirmations.

A form makes you wait for each field. You type, it validates, it renders the next one, and the round trip repeats until the form is done. Speech has no such round trip: a user can name six items in one breath and stop, having spent four seconds on what the form spends four minutes on.

That burst is where the speed comes from. It is also the thing an agent gives back first. Handle the six items one at a time — ask, resolve, confirm, ask again — and you have rebuilt the form, out loud, at the pace of a conversation. Slower than the form it replaced, because now every field costs a spoken turn.

So parallelism is not a later optimization here. Without it the user’s natural behaviour becomes your worst case.

orderdesk’s prompt says it in the imperative, because a model left to itself will take one item and stop:

The moment he names a product, say a tiny line and call add_items. Do not wait for the previous one to resolve; do not ask a question in between. He can list six items in one breath — take them all in ONE add_items call with a list.

One tool call, six items, six rows. Each row gets its own verdict (matched, or ambiguous between two pack sizes, or not stocked) and only the ambiguous ones need the user. The user is still talking while they appear.

servicing makes the same shape explicit for long work. prepare_case puts the workup on screen, where it runs, and returns immediately:

f"preparing {action.ref} in the background{blocker_note}; the workup shows "
"on the case when the prep finishes"

The return value reports what was started; it is not an order to speak. The line the advisor hears is the one the prompt has the agent say with the call: that the case is under way, and the headline of what it found. The model reads the return value with the advisor’s next message, by which time it has already said that line and carried on.

What makes this safe:

session.dispatch(...) never blocks and is callable from anywhere, including from a task that outlived the turn that started it. A background job finishing ninety seconds later can still paint the screen.

An action carries no audio, so it needs no floor. The screen can change while the agent is mid-sentence, and neither one waits for the other.

The real design question is how a result comes back

Section titled “The real design question is how a result comes back”

Voice is one serial channel and there are now three finished jobs to report. Reading them out is almost never the answer. The ways back are not equally priced:

WayWhen to use itShipped in
A row changes on screen, silentlyThe result is legible and unambiguousorderdesk, forge
One short spoken lineIt changes what the user should do nextservicing
A spoken questionGenuinely ambiguous, and only the user can resolve itorderdesk

The default is the silent row change, and forge’s prompt tells the model to trust it:

Every tool you call also shows up as a live task on screen (a small “activity” checklist), so your actions are already acknowledged visually — trust it and stay quiet.

Speaking costs a turn, so a spoken line and a spoken question are spent, not spread. When several rows do turn out to need the user, batch them the way you batched the intake — orderdesk again:

Batch your questions. Let him finish his run of items, then at the natural pause ask about the ambiguous rows, one short question each.

Nothing blocks, a person’s decision included

Section titled “Nothing blocks, a person’s decision included”

Waiting on the user is still waiting. aura’s sign-in puts a sheet on the customer’s screen and returns; the customer authorises it in their own time, and the page’s event tells the brain when. The agent keeps talking in between. The shape is in tool design.

Ordering. Three background jobs finishing at once produce three dispatches in the order they completed. orderdesk makes each action name one row and one change to it (RowMatched, RowQuantity, RowQuestion), so a late action changes only what it names and arrival order between rows stops mattering. That is the convention to copy.

Failure. Every worked example here succeeds. A fan-out where one branch fails is the case you will hit first in production, and the failed row still has to reach the user through one of the ways above — most often as a row that says so.

A chat prompt can afford to be thin. The model can look things up, and the reader watches a spinner while it does — a two-second tool call reads as work happening.

Say the same two seconds out loud. Nothing happens, on a channel where nothing happening is the one thing a user reacts to. So the arithmetic changes: what the agent needs, it should mostly already have.

That is the whole design rule, and it is not “be concise.”

Share of what the agent needsWhere it livesWhat it costs
80%In the prompt alreadytokens
10%One fast tool call away — in your process, no network, marked @needs_result_nowa model round trip
10%Genuinely slow — remote, expensivea background workstream, with an answer to “what does the user hear meanwhile”

These numbers are a design target, not a measured ratio. Their job is the genuinely slow tenth: anything that lands there needs a plan for the silence, which is parallel workstreams.

What a voice prompt does that a chat prompt need not

Section titled “What a voice prompt does that a chat prompt need not”

Talk less, do more. The default reply is one short line, and everything longer is on the screen. The shipped prompts say it in the imperative, because a model’s default register is a paragraph:

  • orderdesk: “No markdown, no lists, no stage directions. Never narrate your own actions (…) — say only what the pharmacist should hear, and make the call with it.”
  • forge: “your actions are already acknowledged visually — trust it and stay quiet.”
  • sugar: “better, don’t say units at all; the screen shows them.”
  • support: “Never read out ids or order numbers as raw text unless the shopper asks — say “your order from May 28th” instead.”

Hold its prompt still. The system prompt is the cache prefix. Set it once per session and it matches turn after turn; rebuild it — even to append one fresh line — and the provider re-reads the whole thing on every turn, which the user pays for in silence. Volatile context goes at the tail, next to the latest user message, where a change costs the provider only the small new suffix. This is the cheapest latency win available and the easiest to throw away by accident, because nothing in a transcript shows it.

Know where the user is. The agent needs a way to answer “what is on screen right now,” and the answer has to be current rather than remembered. Both aura and servicing do the same thing: the page sends an event for each change, the brain keeps its own mirror of the screen up to date from those events and from its own actions, and a tool marked @needs_result_now reads the mirror on demand — aura’s get_screen_context, servicing’s get_advisor_context. The reciprocal instruction matters as much. The screen is authoritative over the agent’s own memory of it, because the user has hands: orderdesk’s prompt ends that thought with “NEVER redo what he already did himself.”

Track a task list. Several threads are open at once and the prompt has to name them and say how each one closes. servicing’s prompt is explicit that background prep runs while the advisor keeps working: “prepare the other case quietly, and in the line you say with prepare_case tell them it is under way and the headline of what you found; the prep finishes on screen by itself. They are never blocked.”

Assume it misheard. A recognizer at a busy pharmacy counter will get things wrong. Correction paths belong in the prompt as first-class instructions rather than as a fallback paragraph at the end — see misunderstanding and reversal.

Never leave silence, in the shipped prompts

Section titled “Never leave silence, in the shipped prompts”
  • legal: “nothing a tool returns reaches you before your turn ends, and you get no further word until they speak again. So the line you say with the call is the whole of your reply: make it the substance, not a promise to report back.”
  • aura: “a screen call’s result reaches you only with the customer’s next message, so you get no second word after it. The line you say with the calls is your whole answer: never a promise of more (“let me check…”)”

These are the prompt doing latency work.

orderdesk has to disambiguate a spoken drug name against two dozen SKUs, by voice, in Hindi, without reading a list aloud. Its prompt teaches an information-theoretic rule in plain language, and the whole block is worth reading as a model of how specific this gets:

Four or fewer choices need no machinery: the pills are already on his screen, so just ask the one short thing that differs (the rule above).

Five or more, and the tool hands you a CANDIDATE TABLE instead of options. Never read it. Never try to show it all. Call ask_choice ONCE with one short English question and TWO TO FOUR choices that split those candidates as evenly as you can. The sharpest question is the one that eliminates the most candidates WHATEVER he answers — a choice that keeps twenty-three of twenty-four is a wasted turn.

Group on the axis that actually partitions the list: the suffix line first (plain / H / AM / CT / BETA…), then form, then a strength band. Never split on pack size while a bigger axis still divides the list.

TWO ROUNDS AT MOST. Round one cuts twenty-four to a handful; round two is leaf pills he can tap, or he simply says which one and you call choose.

What to take from it: it names the threshold at which machinery starts (five), it gives the model a decision rule rather than an example, and it caps the interaction in turns, because a turn is the unit the user feels.

The wording of the question is the model’s; the shape is not. ask_choice rejects a set that has fewer than two or more than four choices, or that leaves a candidate uncovered — and the prompt tells the model what happens when it does, so a rejection is a retry rather than a dead turn.

A tool call in a chat app is a pause. In a voice call, a pause is dead air. So the SDK’s model-backed brains do not wait on a tool to finish the reply: the model speaks its line, calls the tool in the same response, and the tool’s result reaches the model with the user’s next message. The user hears the line while the tool changes the screen.

The properties that follow are properties of the tool rather than of the prompt around it.

A tool returns within 20 ms. The model-backed brains time every call, and one that runs longer is logged as a warning that the user waited for it. Nothing is cancelled or timed out; the warning is how you find it.

servicing’s prepare_case puts minutes of work on screen and returns at once, with a short report of what it started (work that outlives its turn). Whenever the work is slower than a sentence, that is what a voice tool returns: what was started, for the model to read on its next turn.

A tool that needs the network, or anything else slow, has been mis-split. Break it into a cheap dispatch that starts a background task and returns, and let the result land on screen or in the state your other tools read, the way parallel workstreams describes.

Mark a tool only when the reply needs its result

Section titled “Mark a tool only when the reply needs its result”

@needs_result_now marks a tool whose result the model must read before it finishes this reply:

from voqalize.sdk.gemini import GeminiBrain, needs_result_now
class Desk(GeminiBrain):
@property
def tools(self):
return [self.show_card_controls, self.get_account_balance]
async def show_card_controls(self) -> str:
"Put the card controls on screen."
self.session.dispatch(ShowCardControls())
return "shown"
@needs_result_now
async def get_account_balance(self, args: Account) -> dict[str, str]:
"The balance of one of the customer's accounts."
return self.accounts[args.number].balance()

Mark a tool when it reads data the model needs to answer correctly — a balance, the cart, what is on the screen, an eligibility check — from memory your brain already holds. Leave it off actions, screen changes, sign-in prompts, language switches, and a tool whose result only repeats what the model already said. Unmarked is the default, and the right answer for most tools.

A marked tool makes the model answer again as soon as it returns, so it costs the user a model round trip of silence. The mark does not make a slow tool acceptable: a marked tool returns within the same 20 ms.

What the user hears when the mark is wrong:

  • Missing. The agent says its line, calls the tool, and goes quiet until the user speaks again — then answers from the result, a turn late.
  • Extra. A pause before every reply that calls the tool.

Barge-in cancels the turn. It does not un-dispatch an action already sent, and it does not reach work a tool started in the background. A tool that returns at once has finished before the user can cut in; one still awaiting when they do is cancelled at that await, which is one more reason to hand slow work to a task of its own.

This sounds untidy and is correct: the user interrupted the speech. They did not interrupt the lookup, and half-applied work is worse to reason about than completed work. The screen showing what they asked for is right, whether or not the sentence describing it finished.

Tools also run one at a time, in the order the model produced them. Tools racing would put the user’s display in an order the model never asked for, and the screen is the thing the user is reading.

The undo for a voice tool is a compensating call, and the compensating calls are worth enumerating separately rather than collapsing into a re-add. orderdesk gives the model an edit tool per operation and shouts why:

A QUANTITY TWEAK IS NEVER A RE-ADD. An absolute number (…) is set_quantity; a relative one (…) is adjust_quantity with a delta — plus ten is 10, “ten less” is -10. If he wants none of it, that is remove_items, not a delta down to zero.

A VARIANT SWAP IS NEVER A RE-ADD EITHER. … Never remove the row and add it again — he loses his place on the screen.

The reason in that second line is the general rule. Each of these preserves the identity of the row it touches, so the display updates in place and the user’s eye keeps its position. A re-add is correct in the database and wrong on the screen.

The premise underneath is that tools are cheap enough to undo this way. A tool expensive enough that you want to abort it mid-flight is the one to split.

A failed call is a result, not an exception

Section titled “A failed call is a result, not an exception”

A tool that raises comes back to the model as {"error": …}, with the exception’s message, and the model-backed brains log tool … failed in your process. A call to a tool that is not declared this turn answers the same way. The model reads it like any other result and can call again.

It reads it when it reads any other result. For an unmarked tool that is the user’s next message, after the line that announced the call has already played. An agent cheerfully telling a user their order is placed, because the failure came back later as {"error": …} and looked like data, is the shape of the worst bug in this category — and the log line is the only place it shows on your side.

Use the same seam for validation. orderdesk’s ask_choice is rejected unless it has two to four choices, uses known codes, and covers every candidate; a separate validator rejects non-Latin labels headed for a screen that must stay Latin. These come back as retriable errors, and the prompt warns the model in advance that they can. So the shape of the question is guaranteed even though its wording is the model’s.

orderdesk’s prompt draws the line in one sentence:

You have no confirm tool and no confirm authority. Your job is a fully matched cart: every row green, every quantity set.

The pharmacist presses Confirm. servicing is the same shape — submit_packet goes through after the advisor approves. The agent gets the work to the edge of the commit and stops there, which is also what makes every earlier tool safe to undo.

orderdesk answers the manual search bar’s catalog_searched and the variants_opened event floor-free: it dispatches the results as an action, with no model call and no speech. The user is typing in a search box; a keystroke must not make the agent start talking over them.

If something exists to serve the screen rather than the conversation, answer it without taking the floor. Anything that can be triggered by a tap or a keystroke belongs in this category.

A tool that needs the user — to sign in, to approve, to pick — puts the request on screen, says so, and returns. aura’s show_auth_popup returns as soon as the sheet is up:

The sign-in is on screen and the customer is NOT signed in yet. You will be told when they authorise it, and handed an authenticated_context then.

The user’s answer arrives as an event from the page, which your brain appends to the context; it is never the tool’s return value. A tool that depends on it takes a parameter only that answer can supply — aura’s account tools take the authenticated_context the brain mints when the sign-in completes — so the model cannot invent it, and a call without it is refused with a reply that names the missing step.

  • Returns within 20 ms, starting anything slower in the background.
  • Marked @needs_result_now only if the reply needs what it reads from memory.
  • Typed arguments; bad ones come back as an error the model can read.
  • Has an undo that is another tool call.
  • Preserves the identity of what it touched, so the screen does not jump.
  • Holds no authority over anything irreversible.
  • Never waits on the user; the user’s answer arrives as an event.

You have an application worth putting in front of people. Keeping it working is Operate.