Build the brain
A brain is a class. You subclass Brain, implement on_user_message, and yield
speech. Everything else is a callback you add when you need it, or a method on
session when it is not speech. New to the terms? The
glossary defines each one.
from voqalize.sdk import Brain, SpeechChunk, SpeechEnd, SpeechStart
class Concierge(Brain): async def greet(self, session): return "Hi, I'm the concierge. What can I do for you?"
async def on_user_message(self, session, msg): yield SpeechStart() yield SpeechChunk("You said: " + msg.text) yield SpeechEnd()That brain runs. The Quickstart runs it over Cortex and takes a call; this page is about what you add to it.
Speech is yielded; everything else is a session method
Section titled “Speech is yielded; everything else is a session method”on_user_message and on_user_idle are async generators, and they are the only
callbacks that are. They yield speech and nothing else, because speech is the
only output with a position on the audio timeline. Showing something, switching
language and hanging up are methods on session, callable from any callback:
| Method | What it does |
|---|---|
session.dispatch(action) | Sends a typed action to your app. |
await session.configure(config) | Changes voice, language, turn-taking or the idle timer (Voice and language). |
session.send_rtvi(type, data) | Sends a raw RTVI message to your app. |
session.end(reason) | Hangs up. |
The callbacks
Section titled “The callbacks”| Callback | Returns | When it runs |
|---|---|---|
on_user_message | speech (generator) | The user finished a turn. Required. |
on_user_idle | speech (generator) | The user has been quiet; you may re-engage or end. |
on_session_start | None | The session opened, before the greeting. Read session.init here. |
greet | str or None | Once, for the first line. |
on_rtvi | None | The user did something in your app (Context and history). |
on_finalize | None | A speech unit finished playing; reports what the user heard. |
on_error | None | Voqalize sent a signal, such as OVERLOAD. |
on_session_end | None | The socket is closing. Runs at least once; keep it idempotent. |
Only the two generators can speak, so a click or an error never talks over the user. Signatures and arguments are in the Brain API.
The greeting is a string, not a model call
Section titled “The greeting is a string, not a model call”greet returns a string or None. It is async so you can look something up
(for example in session.init), not so you can generate the sentence: the user
is connected and hearing nothing while it runs.
The brain pages, in order
Section titled “The brain pages, in order”- Speaking: the speech frames, units, streaming and acknowledgements.
- Actions: put things on the user’s screen.
- Tools: function calls on the shipped model brains, and the 20 ms budget.
- Context and history: what the user does in your app, and
session.init. - Transcripts and heard truth: what goes into history.
- TypeScript types: generate the browser half of actions and events.
Then test it and deploy it. Already have an agent? Use another agent framework.