Realtime voice SDK

This guide shows how to open a voice live session (brick F-08) from a browser or a server, stream audio, read the transcripts and the spoken answers, and close the session. The TypeScript client is the package @enterprise-ai/realtime (WebRTC through LiveKit with a fallback to the WebSocket protocol, browser and Node.js); the Python SDK (enterprise-ai[realtime]) and the Go realtime package expose the same event model. Source: sdks/typescript/packages/realtime/src.

Version française : realtime.fr.md

Overview

A session lives on the Control Plane: it holds the pipeline (echo or stt), the agent, the language, a quota reservation and a time to live. Clients connect with a short single use ws_token, never with the API key, and exchange JSON events with the server.

Requirements and compatibility

Client	Requirements
TypeScript @enterprise-ai/realtime	A standard WebSocket (browsers, Node.js 22+), or the ws package passed as WebSocket on Node.js 18 and 20; optional peer dependency livekit-client 2 or later for WebRTC
Python	the realtime extra of the wheel pulled from Harbor (websockets, livekit-api), asyncio
Go	the realtime package of the Go SDK (golang.org/x/net WebSocket)

Installation

1. Install the packages from the tarballs pulled from Harbor (see the TypeScript guide).

   npm install ./sdk-typescript/enterprise-ai-sdk-1.0.2.tgz ./sdk-typescript/enterprise-ai-realtime-1.0.2.tgz

   Expected result: both packages are in package.json. Python and Go users install the SDK of their language (Python, Go).

Session lifecycle

1. Your backend creates the session. It calls POST /v1/realtime/sessions with its API key and returns {session_id, ws_url, ws_token, livekit: {url, token, room, configured}} to the browser. The API key never leaves the server.
2. The browser connects with the short lived token. RealtimeClient.connect(session) opens WebRTC when LiveKit is configured, else the WebSocket /v1/realtime/ws/{session_id}.
3. Events flow on the data channel or on the WebSocket (table below).
4. The session ends. POST /v1/realtime/sessions/{id}/end closes it and commits the quota reservation.

// server
import { EnterpriseAI } from "@enterprise-ai/sdk";
const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, locale: "en" });
const session = await client.realtime.createSession({ pipeline: "stt", ttl_seconds: 900, config: { agent_id: "agt_support_voice", language: "fr" } });

// browser
import { RealtimeClient } from "@enterprise-ai/realtime";
const rtClient = new RealtimeClient({ baseUrl: "<control-plane-url>", locale: "fr" });
const rt = await rtClient.connect(session);
rt.on("transcript.delta", (e) => ui.caption(e.delta, e.final));
rt.on("response.audio.delta", () => ui.speaking(true));
rt.on("usage.update", (e) => ui.budget(e.usage));
rt.on("error", (e) => ui.error(e.error.message));
rt.updateSession({ instructions: "Réponds en français", voice: "fr_FR-siwis-medium" }); // session.update
rt.close(1000, "user_hangup");

Client configuration

new RealtimeClient(options: RealtimeClientOptions):

Name	Type	Default	Description
baseUrl	string	none, required	Control Plane URL.
apiKey	string	none	API key, for a server side client only.
token	string | (() => string | Promise<string>)	none	OIDC token or provider.
tenant, project, environment	string	none	Tenancy headers, as in the SDK.
locale	"en" | "fr"	"en"	Language of the error messages.
fetch	FetchLike	global fetch	HTTP calls (session, token, end).
WebSocket	WebSocketCtor	globalThis.WebSocket	WebSocket implementation; pass ws on Node.js 18 and 20.
webSocketOptions	Record<string, unknown>	none	Third constructor argument of the WebSocket (for example { headers } with ws).
reconnect	{ enabled?, maxAttempts?, baseDelayMs?, maxDelayMs? }	{ enabled: true, maxAttempts: 5, baseDelayMs: 500, maxDelayMs: 10000 }	Reconnection policy after a network loss.
maxQueue	number	500	Outbound events queued while disconnected.
now	() => number	Date.now	Clock of the latency metrics, injectable in tests.

RealtimeClient also has createSession, getSession, endSession, refreshToken, wsUrl and start (create then connect). A RealtimeConnection has send, updateSession, appendAudio, commitAudio, createResponse, cancelResponse, speechStarted, metrics and close.

Events

Event	Direction	Payload
session.update	client to server	instructions, voice, vad, language
input_audio.append	client to server	audio (PCM16 20 ms, base64) on WebSocket
transcript.delta	server to client	text, final, speaker
response.audio.delta	server to client	audio chunk
usage.update	server to client	audio_seconds, input_tokens, output_tokens, audio_tokens, estimated cost
error	server to client	code, message, message_i18n
response.create	client to server	{text, voice?}: the server speaks the text (rt.createResponse(text, voice))
response.cancel	client to server	stops the running response (rt.cancelResponse()), error no_active_response when none runs
input_audio.speech_started	client to server	barge-in from a client VAD (rt.speechStarted()); any input_audio.append during a response also cancels it
input_audio.commit	client to server	closes the user turn (rt.commitAudio()), saves the recording when config.record is set
response.created	server to client	turn_id, trace_id
response.done	server to client	turn_id, chunks_sent, latency_ms {tts_ms, first_audio_ms, total_ms}
response.cancelled	server to client	turn_id, reason (barge_in or client_cancel), chunks_sent
recording.saved	server to client	recording_id, sha256, bytes, expires_at

rt.on("response.created", (e) => ui.turn(e.turn_id));
rt.on("response.cancelled", (e) => ui.stopSpeaking(e.reason));
rt.on("response.done", (e) => metrics.record(e.latency_ms));
rt.on("recording.saved", (e) => ui.saved(e.recording_id));
rt.createResponse("Your order ships tomorrow.", "fr_FR-siwis-medium");
mic.onSpeech(() => rt.speechStarted()); // barge-in

Transcription and recording require an active consent: record it with client.realtime.recordConsent(sessionId, {purpose, retention_days, recording_allowed, transcription_allowed}), otherwise the server answers an error event with the code consent_required. client.realtime.turns(id) and client.realtime.transcripts(id) read the turns and the DLP filtered transcripts.

Errors and resilience

• Errors of the HTTP calls and of the connection are RealtimeError (code, status, details), with messages in the client locale; server error events carry code, message and message_i18n.
• quota_exceeded during a call: a warning usage.update at 90 % of the reservation, then error and a graceful end.
• Network change: reconnection with the same session id and a fresh ws_token while expires_at is not reached, following the reconnect policy; outbound events wait in a queue of maxQueue events.
• WebRTC blocked: TURN over TLS 443, then the WebSocket fallback.

Python and Go

Python: client.realtime.connect(session) is an async context manager; events() yields the server events and raises RealtimeError on an error event unless raise_on_error=False.

session = client.realtime.create_session("stt", config={"agent_id": "agt_support_voice", "language": "fr"})
async with client.realtime.connect(session) as s:
    await s.append_audio(pcm16_frame)
    async for ev in s.events():
        if ev.type == "transcript.delta":
            print(ev)

Go: realtime.Dial requests a new single use token, then opens the WebSocket; Recv returns the next Event (Type, Raw) and Close ends the connection.

sess, err := realtime.Dial(ctx, client, sessionID)
defer sess.Close()

Testing

Inject the transports: fetch and WebSocket in TypeScript (a fake socket class is enough), a mocked httpx transport plus a connector in Python, an httptest server in Go. The package tests sdks/typescript/packages/realtime/test/realtime.test.ts show a complete fake WebSocket.

FinOps

Usage type voice_live: audio_seconds, audio_tokens, text input and output tokens, sessions, duration_ms. See F-08.

Security

wsUrl only returns wss URLs for non local hosts (ws is allowed for localhost and 127.0.0.1), and a server provided ws_url must stay on the base URL host. The API key stays on the server: the browser only holds the single use token of one session.

Reference

• Generated reference: TypeScript SDK (client.realtime), Python, Go.
• TypeScript SDK, Python SDK, Go SDK, Java SDK.