F-09 Voice activity detection

Brick F of the Enterprise AI Platform. Technology: Silero VAD (ONNX) embedded in STT workers and realtime pipelines. French version: ../../fr/bricks/F-09-vad.md.

Role and responsibilities

• Detect speech start and end, drive turn taking and barge-in in F-08.
• Remove silence before STT to reduce cost and hallucinations.

Position and interactions

• Execution path: realtime (in stream) and batch (pre-filter).
• Default threshold by deployment (EAI_VAD_THRESHOLD); a realtime session sets its own VAD in config.vad (at creation or with session.update): enabled (boolean, false marks every chunk as speech) and threshold (0 to 1, passed to Silero and to the energy VAD). Any other key or value gives a translated 422 parameter_invalid.

Open source and developed parts

• Open source reused: Silero VAD, onnxruntime.
• Developed in the platform: Tuning profiles (call center, meeting, noisy field), endpointing combined with semantic turn detection.

Data model

Control Plane tables (schema eai): eai.realtime_sessions, eai.multimodal_jobs.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/multimodal-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints

Method	Path
POST	/v1/realtime/sessions
POST	/v1/multimodal/stt

POST /v1/realtime/sessions

Request:

{
  "room": "support-voice-42",
  "pipeline": "stt",
  "ttl_seconds": 900,
  "config": {
    "language": "fr",
    "sample_rate": 16000,
    "dlp_mode": "mask"
  }
}

Response:

{
  "session_id": "rts_3K",
  "livekit_url": "wss://<livekit-host>",
  "token": "eyJhbGciOi...",
  "ws_url": "wss://<control-plane-host>/v1/realtime/ws/rts_3K",
  "expires_at": "2026-10-29T10:15:00Z"
}

POST /v1/multimodal/stt

Request:

{
  "media_id": "med_AUD1",
  "model": "faster-whisper-large-v3",
  "language": "fr",
  "dlp_mode": "detect"
}

Response:

{
  "job_id": "mmj_S01",
  "status": "succeeded",
  "text": "Bonjour, je vous appelle au sujet de ma facture.",
  "language": "fr",
  "segments": [
    {
      "start": 0.0,
      "end": 2.8,
      "text": "Bonjour, je vous appelle"
    }
  ],
  "usage": {
    "audio_seconds": 184.2,
    "output_tokens": 912
  }
}

SDK examples

Python

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
session = client.realtime.create_session("stt", config={"language": "fr"})
client.multimodal.stt.transcribe("med_AUD1", model="faster-whisper-large-v3", language="fr")

TypeScript

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
const session = await client.realtime.createSession({ pipeline: "stt", config: { language: "fr" } });
await client.multimodal.stt({ media_id: "med_AUD1", model: "faster-whisper-large-v3", language: "fr" });

Java

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
JsonNode session = client.realtime().createSession(Map.of("pipeline", "stt", "config", Map.of("language", "fr")));
client.multimodal().stt(Map.of("media_id", "med_AUD1", "model", "faster-whisper-large-v3", "language", "fr"));

Go

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Realtime.CreateSession(ctx, map[string]any{"pipeline": "stt", "config": map[string]any{"language": "fr"}})
_, err = client.Multimodal.Stt(ctx, map[string]any{"media_id": "med_AUD1", "model": "faster-whisper-large-v3", "language": "fr"})

// worker package: run a media worker consuming multimodal jobs
w, err := worker.New(worker.Config{Source: jobs, Handler: handle, Client: client, Concurrency: 2})
if err != nil { log.Fatal(err) }
err = w.Run(ctx)

Configuration

Environment variables: EAI_VAD_THRESHOLD.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): controlPlane.config, controlPlane.secrets.

Security

• Runs in process, audio never leaves the worker for VAD.
• Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
• Every mutation and policy decision writes a hash chained audit event (G-11).
• Errors use the standard envelope with message_i18n en and fr.

Observability

• Speech ratio and false barge-in rate per session.
• W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units

• No billable unit of its own; savings appear as lower audio_seconds sent to STT.

Failure modes and resilience

• Noisy environment -> profile switch or higher threshold, measured by false barge-in rate.

Standalone adoption

Comes with F-05 and F-08; no separate deployment needed.

Acceptance tests

1. On the reference set, speech segments detected with F1 above target.
2. A 30% silence file reduces billed STT seconds accordingly.

Implementation roadmap

Kit L4 voice evidence

Silero VAD v6.2.3 ONNX on onnxruntime (adapters silero-onnx and http-vad): speech segment found in Kokoro speech, none in silence (MULT-005). See the voice runtime guide.