Skip to content

How Voqalize works

An integration has four parts. You own three of them, and you create one record that joins them:

PartWhat it isWho runs it
Your pageYour client on @voqalize/client-transport. It asks your server for a session and connects to it.You
Your serverOne HTTP request, sessions.connect, that decides who may start a session and with what.You
Your brainOne WebSocket endpoint. It receives each finished user turn as text and returns speech and actions.You
VoqalizeWebRTC, recognition, synthesis, turn-taking, interruption, the avatar and recording.Voqalize
The agentA record that says how Voqalize reaches your brain, and whether its sessions record by default.You create it; Voqalize stores it

Your page and your brain never talk to each other directly. Everything between them crosses Voqalize, and none of it is audio on your side: your brain sees text in and text out.

A session is one conversation, from connect to hang-up. It has one WebSocket to your brain, opened when it starts and closed when it ends.

sequenceDiagram
  participant P as Your page
  participant S as Your server
  participant V as Voqalize
  participant B as Your brain

  P->>S: start a session (your own auth)
  S->>V: POST sessions.connect: agent_id, config, init
  V-->>S: connect params
  S-->>P: the same body, unchanged
  P->>V: client.connect(params), then WebRTC
  V->>B: open the WebSocket
  V->>B: SessionStart, carrying init
  B-->>V: the greeting
  loop every turn
    P->>V: the user speaks
    V->>B: UserMessage, the finished turn as text
    B-->>V: SpeechStart, SpeechChunk…, SpeechEnd
    V-->>P: audio, transcript and actions
    V->>B: Finalize, what the user heard of each unit
    P->>V: a click or form, as an app event
    V->>B: the same event, unchanged
  end
  B-->>V: End
  1. Your page asks your server. How that request is authenticated is yours: a session cookie, a bearer token, whatever your app already does. A public page can skip this hop and call Voqalize with a publishable pk_ key instead.
  2. Your server calls sessions.connect with the agent’s secret sk_ key. The body names the agent and may carry config (how this session sounds and listens, and whether it records) and init (anything your brain should know about this user). Connect your app has the request.
  3. Your page connects. It hands the response, unchanged, to client.connect. Media then flows directly between the browser and the Voqalize machine running the session.
  4. Voqalize dials your brain at {brain_url}?session_id={session_id} and sends SessionStart with init. Your on_session_start runs, then greet returns the first line. The greeting is a fixed string or a template over init, because the user is already listening and a model call here is silence.
  5. The user speaks. Voqalize decides when the turn is finished and sends it to your brain as UserMessage text. With acknowledgements on (the default), Voqalize says a short line such as “Sure.” while your brain works.
  6. Your brain answers in speech units. It yields SpeechStart, one or more SpeechChunks and SpeechEnd. Speaking starts on the first chunk, so the reply plays while it is still being generated. During a turn it can also dispatch actions to the page, call tools, or change the voice and language with session.configure(...).
  7. The user may interrupt. Voqalize stops the audio and sends Interruption. For every unit, Finalize says what the user actually heard. Keep that in your history, not what you generated: Transcripts and heard truth explains how.
  8. The page talks back. A click or a form submitted in your page reaches the brain as an app event at on_rtvi. It does not take a turn, so the agent never talks over a user who just clicked.
  9. The session ends when the user hangs up, the brain calls session.end(), or the session reaches the one-hour cap. The session record, events, logs and any recording are then readable by session_id; Reading a call back shows how.

The wire specifies every frame. In Python you never build them: you subclass Brain and implement callbacks, as Build the brain shows.

The agent record says how, in one of two ways, and the brain code is the same for both. In inbound mode the record holds a brain_url, a wss:// route in your app that Voqalize dials once per session; this is the production default. In Cortex mode your brain dials out to the Cortex relay with the agent’s sk_ key and Voqalize meets it there, so nothing has to accept an inbound connection. Cortex is the local-development default (no tunnel) and fits serverless functions and egress-only networks. An agent with neither refuses every session with 409 agent_not_configured. Deploy the brain sets up both.

The agent record holds no voice, language or turn-taking settings, because the right language usually depends on the user, who does not exist until the session starts. Each layer below overrides the one before it:

  1. Voqalize’s defaults: English on both legs, acknowledgements on, no idle timeout.
  2. The session creator’s config at sessions.connect: your server with an sk_, or a page with a pk_. Use it when your server already knows the user’s language.
  3. The brain, with await session.configure(Config(...)), from on_session_start (in time for the greeting) or at any point during the session.

Recording is the exception: it is the agent’s default unless config.record overrides it at connect, and it cannot change mid-session. Voice and language covers voice, language, patience, idle and acknowledgements; Recordings covers recording and consent.

None of these is needed for a first session, and each is a change to one side:

CapabilityWhat changesPage
Voice, language and turn-takingThe connect config or the brainVoice and language
Screen actionsThe brain dispatches, the page rendersActions
StageA spec, the brain and the pageStage
The avatarThe page onlyThe avatar
Camera and screen recordingThe page turns lanes onCamera and screen
RecordingThe agent default or the connect configRecordings
TermMeaning
BrainYour code: one WebSocket endpoint that receives user turns as text and returns speech and actions. In Python, a Brain subclass. Build the brain.
AgentThe Voqalize record that names how to reach a brain (mode and brain_url) and its recording default. Configuration, not intelligence. Your own “agent” is the brain.
TenantThe account that owns your agents, keys, sessions and members.
SessionOne conversation: one connect, one WebSocket to the brain, one record. It lasts at most one hour.
TurnOne finished utterance from the user, delivered as UserMessage text, and the brain’s reply to it.
Speech unitOne SpeechStart … SpeechEnd span of the brain’s reply. Speaking starts on its first chunk, and it is the piece a user can be interrupted out of. Speaking.
Heard textWhat the user actually heard of a unit, delivered in Finalize. Shorter than what you generated when the user interrupted. Transcripts and heard truth.
Acknowledgement (ack)The short line Voqalize says while your brain works on a reply. On by default. Voice and language.
ActionA typed message from the brain to the page (session.dispatch(...)), delivered as an RTVI ui-command. It renders and is never spoken. Actions.
App eventA typed message from the page to the brain (AppEvent), arriving at on_rtvi. It informs the brain and does not take a turn. Context and history.
ToolA function your model calls during a turn. It runs in your process. Tools.
The wireThe published protobuf frames on the brain’s WebSocket. Any language can implement it. The wire.
RTVIThe message format your page and your brain share over the session’s data channel. Voqalize carries it between them. RTVI messages.
Inbound routeA wss:// route in your app that Voqalize dials once per session. Deploy the brain.
CortexThe relay for a brain that cannot accept inbound connections: your brain dials out to it and Voqalize meets it there. Deploy the brain.
sk_ keyA secret key for one agent. Your server uses it for sessions.connect, and a Cortex brain uses it to dial the relay. Keys.
pk_ keyA publishable key for one agent, locked to a list of browser origins. A browser page may start sessions with it. Keys.
Session tokenThe short-lived credential in the connect response. The browser presents it to the Voqalize machine running the session.
Brain-connection tokenThe short-lived credential Voqalize presents when it dials your brain. The SDK verifies it for you. Deploy the brain.
configHow a session sounds and listens: tts, stt, idle, ack, and record at connect. Set at connect and by the brain.
initYour data for the brain, passed at connect and read as session.init. Stored on the session, so send identifiers rather than personal data. Context and history.
StageThe framework for an agent that shows and does things on the page while it talks, declared as moves in a spec. Stage.
The avatarAn animated character drawn in the page, lip-synced to the agent and driven by the session’s own messages. The avatar.
RecordingThe session’s audio, plus camera and screen when the page turns them on. Recordings.
Conformance harnessThe SDK’s test driver: your brain over the real wire, with no microphone and no model. Test the brain.
MCP serverThe hosted tool server your coding agent uses to create agents, mint keys and read sessions. Work with a coding agent.