Voice runtime: consent, identity, interruption, PII, engines

French version: ../voice-runtime.md. Bricks F-05 to F-12, requirements MULT-001 to MULT-012 and RELE-011, kit scenario 18.

Consent and retention (MULT-008)

• One consent record per session and participant: POST /v1/realtime/sessions/{id}/consents with purpose, recording_allowed, transcription_allowed, retention_days (1 to 3650) and method (verbal, click, written, api). The caller is recorded as captured_by, the time as captured_at. Revocation: POST /v1/realtime/sessions/{id}/consents/{consent_id}/revoke, effective on the next audio event.
• EAI_REALTIME_CONSENT_REQUIRED (unset: required outside dev and test; false refused at production startup). Without an active consent allowing transcription, every audio event gets the WebSocket error consent_required (purpose: transcription), no STT runs, and the refused turn is audited realtime.turn.refused with a deny decision.
• Recording (config.record: true) always needs recording_allowed, in every environment. The audio buffered between two input_audio.commit is sealed with the tenant key (eai.realtime_recordings, sha256 and size kept in clear) and the event recording.saved is sent.
• Retention: transcripts and recordings expire retention_days after capture. POST /v1/retention/run (resource type realtime, included in a run without types) deletes the expired rows and reports the counts.
• Tables (migration V145, RLS enabled and forced): realtime_consents, realtime_turns, realtime_transcripts, realtime_recordings; new columns participant_identity, on_behalf_of, agent_id on realtime_sessions.

Identity (MULT-012, RELE-011)

• A user opens a session for itself (on_behalf_of other than the caller: 403 identity_mismatch) and may name the agent_id it talks to. An agent principal opens it on_behalf_of a user; its id becomes agent_id.
• The LiveKit participant identity is the caller id; the token metadata carries tenant_id, session_id, on_behalf_of, agent_id and actor_kind. The real LiveKit server test shows them in ListParticipants.
• Every turn (input or response) stores and audits participant_id, user_id, agent_id, session_id and trace_id (the audit event carries the same trace id). GET /v1/realtime/sessions/{id}/turns lists them with the status and the measured latency.

Interruption (MULT-010)

response.create {text} starts a response turn: the TTS adapter runs in a worker thread and the audio is streamed as response.audio.delta chunks of 100 ms, paced at real time (config.response_pace, 0 for no pacing). A barge-in (input_audio.speech_started, any input_audio.append while the response streams, or response.cancel) stops the stream at the next chunk, discards a TTS call still in flight, records the turn cancelled with chunks_sent and the reason, audits realtime.turn.cancelled and sends response.cancelled. The next response.create starts a fresh turn.

Transcript PII (MULT-009)

The transcript goes through the privacy DLP (mode mask by default, tokenize or block; detect is refused for a realtime session) before it is stored, sent to the client or the model. Stored rows keep the filtered text and the finding types with counts only. The adapters never log audio or transcript; the tests check the stored rows, the logs, the audit events and the spans.

Engines on CPU (MULT-003, MULT-005, MULT-006)

tools/multimodal/setup.sh creates /tmp/toolchains/voice-venv (faster-whisper 1.2.1, ctranslate2 4.6.0, onnxruntime 1.22.1, numpy 2.2.6, kokoro-onnx 0.6.1 from PyPI) and /tmp/toolchains/voice-models, every file pinned and sha256 verified:

Engine	Source (pinned)	File	sha256	Licence
faster-whisper tiny.en	Hugging Face Systran/faster-whisper-tiny.en revision 0d3d19a32d3338f10357c0889762bd8d64bbdeba	model.bin	1a5afae06a4db91c975c9a9d78be5cc110ee4ea022ad57d55492e4550e936b2a	MIT
Silero VAD	GitHub snakers4/silero-vad tag v6.2.3	silero_vad.onnx	1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3	MIT
Kokoro v1.0 int8	GitHub thewh1teagle/kokoro-onnx release model-files-v1.0 (export of hexgrad/Kokoro-82M)	kokoro-v1.0.int8.onnx	6e742170d309016e5891a994e1ce1559c702a2ccd0075e67ef7157974f6406cb	Apache-2.0
Kokoro voices	same release	voices-v1.0.bin	bca610b8308e8d99f32e6fe4197e7ec01679264efed0cac9140fe9c29f1fbf7d	Apache-2.0
LiveKit server 1.13.11	GitHub livekit/livekit release v1.13.11	livekit_1.13.11_linux_amd64.tar.gz	1e9ee90ca301e1842b2887fc77cb122759756a9c8aeb4334ee91de3325ec43fb	Apache-2.0

The HTTP voice worker tools/multimodal/voice_worker.py (127.0.0.1 only) serves /audio/transcriptions, /audio/speech and /vad; the platform reaches it with the adapters openai-compatible-stt (EAI_STT_URL), openai-compatible-tts (EAI_TTS_URL) and http-vad (EAI_VAD_URL) through the egress guard. In process adapters exist as well: kokoro (EAI_KOKORO_MODEL, EAI_KOKORO_VOICES), silero-onnx (EAI_SILERO_VAD_MODEL), faster-whisper (EAI_WHISPER_MODEL).

Licence exceptions recorded: kokoro-onnx phonemizes with espeak-ng (GPL-3.0) through phonemizer (GPL-3.0) and espeakng-loader: they run only inside the separate voice worker process, never in the control plane. ffmpeg (/usr/bin/ffmpeg, LGPL-2.1 or GPL depending on the build) is called as an external program for video frames and audio extraction. Piper is not shipped: its maintained release piper1-gpl (piper-tts 1.3 and later) is GPL-3.0; Kokoro is the reference TTS.

Measured latencies (CPU, this host)

Written by tests/test_voice_engines_real.py to /tmp/eai-voice-latency.json: Kokoro synthesis of one sentence about 4.1 s on the first call (model load included) and 1.2 s warm, Silero VAD on 4.5 s of audio about 0.1 s, faster-whisper tiny.en about 1.0 s, input turn over the WebSocket (VAD, STT, DLP) about 1.9 s, DLP under 2 ms.

Limits

• MULT-001, MULT-002: the LiveKit server is started for real, the platform token is validated and the participant joins over the signalling WebSocket; WebRTC media (UDP, ICE, codecs) needs a browser or a WebRTC client and is not tested.
• MULT-004: pyannote weights are gated on Hugging Face (EAI_HF_TOKEN); the adapter is tested with a mock only.
• MULT-007: video frames are extracted by the real ffmpeg and analysed by the vision adapter against a fake VLM upstream; a real VLM needs a GPU and weights.