How Voqalize works
An integration has four parts. You own three of them, and you create one record that joins them:
| Part | What it is | Who runs it |
|---|---|---|
| Your page | Your client on @voqalize/client-transport. It asks your server for a session and connects to it. | You |
| Your server | One HTTP request, sessions.connect, that decides who may start a session and with what. | You |
| Your brain | One WebSocket endpoint. It receives each finished user turn as text and returns speech and actions. | You |
| Voqalize | WebRTC, recognition, synthesis, turn-taking, interruption, the avatar and recording. | Voqalize |
| The agent | A record that says how Voqalize reaches your brain, and whether its sessions record by default. | You create it; Voqalize stores it |
Your page and your brain never talk to each other directly. Everything between them crosses Voqalize, and none of it is audio on your side: your brain sees text in and text out.
What happens in a session
Section titled “What happens in a session”A session is one conversation, from connect to hang-up. It has one WebSocket to your brain, opened when it starts and closed when it ends.
sequenceDiagram
participant P as Your page
participant S as Your server
participant V as Voqalize
participant B as Your brain
P->>S: start a session (your own auth)
S->>V: POST sessions.connect: agent_id, config, init
V-->>S: connect params
S-->>P: the same body, unchanged
P->>V: client.connect(params), then WebRTC
V->>B: open the WebSocket
V->>B: SessionStart, carrying init
B-->>V: the greeting
loop every turn
P->>V: the user speaks
V->>B: UserMessage, the finished turn as text
B-->>V: SpeechStart, SpeechChunk…, SpeechEnd
V-->>P: audio, transcript and actions
V->>B: Finalize, what the user heard of each unit
P->>V: a click or form, as an app event
V->>B: the same event, unchanged
end
B-->>V: End
- Your page asks your server. How that request is authenticated is yours: a
session cookie, a bearer token, whatever your app already does. A public page
can skip this hop and call Voqalize with a publishable
pk_key instead. - Your server calls
sessions.connectwith the agent’s secretsk_key. The body names the agent and may carryconfig(how this session sounds and listens, and whether it records) andinit(anything your brain should know about this user). Connect your app has the request. - Your page connects. It hands the response, unchanged, to
client.connect. Media then flows directly between the browser and the Voqalize machine running the session. - Voqalize dials your brain at
{brain_url}?session_id={session_id}and sendsSessionStartwithinit. Youron_session_startruns, thengreetreturns the first line. The greeting is a fixed string or a template overinit, because the user is already listening and a model call here is silence. - The user speaks. Voqalize decides when the turn is finished and sends it to
your brain as
UserMessagetext. With acknowledgements on (the default), Voqalize says a short line such as “Sure.” while your brain works. - Your brain answers in speech units. It yields
SpeechStart, one or moreSpeechChunks andSpeechEnd. Speaking starts on the first chunk, so the reply plays while it is still being generated. During a turn it can also dispatch actions to the page, call tools, or change the voice and language withsession.configure(...). - The user may interrupt. Voqalize stops the audio and sends
Interruption. For every unit,Finalizesays what the user actually heard. Keep that in your history, not what you generated: Transcripts and heard truth explains how. - The page talks back. A click or a form submitted in your page reaches the
brain as an app event at
on_rtvi. It does not take a turn, so the agent never talks over a user who just clicked. - The session ends when the user hangs up, the brain calls
session.end(), or the session reaches the one-hour cap. The session record, events, logs and any recording are then readable bysession_id; Reading a call back shows how.
The wire specifies every frame. In Python you never build
them: you subclass Brain and implement callbacks, as
Build the brain shows.
How Voqalize reaches your brain
Section titled “How Voqalize reaches your brain”The agent record says how, in one of two ways, and the brain code is the same
for both. In inbound mode the record holds a brain_url, a wss:// route in
your app that Voqalize dials once per session; this is the production default.
In Cortex mode your brain dials out to the Cortex relay with the agent’s
sk_ key and Voqalize meets it there, so nothing has to accept an inbound
connection. Cortex is the local-development default (no tunnel) and fits
serverless functions and egress-only networks. An agent with neither refuses
every session with 409 agent_not_configured.
Deploy the brain sets up both.
Where each setting comes from
Section titled “Where each setting comes from”The agent record holds no voice, language or turn-taking settings, because the right language usually depends on the user, who does not exist until the session starts. Each layer below overrides the one before it:
- Voqalize’s defaults: English on both legs, acknowledgements on, no idle timeout.
- The session creator’s
configatsessions.connect: your server with ansk_, or a page with apk_. Use it when your server already knows the user’s language. - The brain, with
await session.configure(Config(...)), fromon_session_start(in time for the greeting) or at any point during the session.
Recording is the exception: it is the agent’s default unless config.record
overrides it at connect, and it cannot change mid-session.
Voice and language covers voice, language,
patience, idle and acknowledgements; Recordings covers
recording and consent.
What you can add
Section titled “What you can add”None of these is needed for a first session, and each is a change to one side:
| Capability | What changes | Page |
|---|---|---|
| Voice, language and turn-taking | The connect config or the brain | Voice and language |
| Screen actions | The brain dispatches, the page renders | Actions |
| Stage | A spec, the brain and the page | Stage |
| The avatar | The page only | The avatar |
| Camera and screen recording | The page turns lanes on | Camera and screen |
| Recording | The agent default or the connect config | Recordings |
Glossary
Section titled “Glossary”| Term | Meaning |
|---|---|
| Brain | Your code: one WebSocket endpoint that receives user turns as text and returns speech and actions. In Python, a Brain subclass. Build the brain. |
| Agent | The Voqalize record that names how to reach a brain (mode and brain_url) and its recording default. Configuration, not intelligence. Your own “agent” is the brain. |
| Tenant | The account that owns your agents, keys, sessions and members. |
| Session | One conversation: one connect, one WebSocket to the brain, one record. It lasts at most one hour. |
| Turn | One finished utterance from the user, delivered as UserMessage text, and the brain’s reply to it. |
| Speech unit | One SpeechStart … SpeechEnd span of the brain’s reply. Speaking starts on its first chunk, and it is the piece a user can be interrupted out of. Speaking. |
| Heard text | What the user actually heard of a unit, delivered in Finalize. Shorter than what you generated when the user interrupted. Transcripts and heard truth. |
| Acknowledgement (ack) | The short line Voqalize says while your brain works on a reply. On by default. Voice and language. |
| Action | A typed message from the brain to the page (session.dispatch(...)), delivered as an RTVI ui-command. It renders and is never spoken. Actions. |
| App event | A typed message from the page to the brain (AppEvent), arriving at on_rtvi. It informs the brain and does not take a turn. Context and history. |
| Tool | A function your model calls during a turn. It runs in your process. Tools. |
| The wire | The published protobuf frames on the brain’s WebSocket. Any language can implement it. The wire. |
| RTVI | The message format your page and your brain share over the session’s data channel. Voqalize carries it between them. RTVI messages. |
| Inbound route | A wss:// route in your app that Voqalize dials once per session. Deploy the brain. |
| Cortex | The relay for a brain that cannot accept inbound connections: your brain dials out to it and Voqalize meets it there. Deploy the brain. |
sk_ key | A secret key for one agent. Your server uses it for sessions.connect, and a Cortex brain uses it to dial the relay. Keys. |
pk_ key | A publishable key for one agent, locked to a list of browser origins. A browser page may start sessions with it. Keys. |
| Session token | The short-lived credential in the connect response. The browser presents it to the Voqalize machine running the session. |
| Brain-connection token | The short-lived credential Voqalize presents when it dials your brain. The SDK verifies it for you. Deploy the brain. |
config | How a session sounds and listens: tts, stt, idle, ack, and record at connect. Set at connect and by the brain. |
init | Your data for the brain, passed at connect and read as session.init. Stored on the session, so send identifiers rather than personal data. Context and history. |
| Stage | The framework for an agent that shows and does things on the page while it talks, declared as moves in a spec. Stage. |
| The avatar | An animated character drawn in the page, lip-synced to the agent and driven by the session’s own messages. The avatar. |
| Recording | The session’s audio, plus camera and screen when the page turns them on. Recordings. |
| Conformance harness | The SDK’s test driver: your brain over the real wire, with no microphone and no model. Test the brain. |
| MCP server | The hosted tool server your coding agent uses to create agents, mint keys and read sessions. Work with a coding agent. |
Read next
Section titled “Read next”- Quickstart: hear an agent you wrote.
- Build an agent: every step, in order.
- Examples: complete apps that use each part.