Skip to content

Build the brain

A brain is a class. You subclass Brain, implement on_user_message, and yield speech. Everything else is a callback you add when you need it, or a method on session when it is not speech. New to the terms? The glossary defines each one.

from voqalize.sdk import Brain, SpeechChunk, SpeechEnd, SpeechStart
class Concierge(Brain):
async def greet(self, session):
return "Hi, I'm the concierge. What can I do for you?"
async def on_user_message(self, session, msg):
yield SpeechStart()
yield SpeechChunk("You said: " + msg.text)
yield SpeechEnd()

That brain runs. The Quickstart runs it over Cortex and takes a call; this page is about what you add to it.

Speech is yielded; everything else is a session method

Section titled “Speech is yielded; everything else is a session method”

on_user_message and on_user_idle are async generators, and they are the only callbacks that are. They yield speech and nothing else, because speech is the only output with a position on the audio timeline. Showing something, switching language and hanging up are methods on session, callable from any callback:

MethodWhat it does
session.dispatch(action)Sends a typed action to your app.
await session.configure(config)Changes voice, language, turn-taking or the idle timer (Voice and language).
session.send_rtvi(type, data)Sends a raw RTVI message to your app.
session.end(reason)Hangs up.
CallbackReturnsWhen it runs
on_user_messagespeech (generator)The user finished a turn. Required.
on_user_idlespeech (generator)The user has been quiet; you may re-engage or end.
on_session_startNoneThe session opened, before the greeting. Read session.init here.
greetstr or NoneOnce, for the first line.
on_rtviNoneThe user did something in your app (Context and history).
on_finalizeNoneA speech unit finished playing; reports what the user heard.
on_errorNoneVoqalize sent a signal, such as OVERLOAD.
on_session_endNoneThe socket is closing. Runs at least once; keep it idempotent.

Only the two generators can speak, so a click or an error never talks over the user. Signatures and arguments are in the Brain API.

The greeting is a string, not a model call

Section titled “The greeting is a string, not a model call”

greet returns a string or None. It is async so you can look something up (for example in session.init), not so you can generate the sentence: the user is connected and hearing nothing while it runs.

  1. Speaking: the speech frames, units, streaming and acknowledgements.
  2. Actions: put things on the user’s screen.
  3. Tools: function calls on the shipped model brains, and the 20 ms budget.
  4. Context and history: what the user does in your app, and session.init.
  5. Transcripts and heard truth: what goes into history.
  6. TypeScript types: generate the browser half of actions and events.

Then test it and deploy it. Already have an agent? Use another agent framework.