F-06 Speaker diarization

Brick F of the Enterprise AI Platform. Technology: pyannote.audio (speaker-diarization pipeline) on GPU workers. French version: ../../fr/bricks/F-06-speaker-diarization.md.

Role and responsibilities

• Answer who spoke when: speaker turns, number of speakers, overlap detection.
• Merge turns with STT words to produce speaker-attributed transcripts.
• POST /v1/multimodal/diarize accepts min_speakers and max_speakers (1 to 32, min_speakers at most max_speakers), passed to pyannote; an adapter that cannot honour them, or word_timestamps on this route, gives a translated 422 parameter_unsupported.

Position and interactions

• Execution path: batch job, chained after or with STT.
• Called directly or through POST /v1/multimodal/stt with diarize=true.

Open source and developed parts

• Open source reused: pyannote.audio, PyTorch.
• Developed in the platform: Alignment with STT words, speaker label mapping, no persistent voiceprints by default.

Data model

Control Plane tables (schema eai): eai.media, eai.multimodal_jobs, eai.usage_events.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/multimodal-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints

Method	Path
POST	/v1/multimodal/diarize
GET	/v1/multimodal/jobs/{id}

POST /v1/multimodal/diarize

Request:

{
  "media_id": "med_AUD1",
  "language": "fr"
}

Response:

{
  "job_id": "mmj_D01",
  "status": "succeeded",
  "speakers": 2,
  "turns": [
    {
      "speaker": "SPEAKER_00",
      "start": 0.0,
      "end": 6.1
    },
    {
      "speaker": "SPEAKER_01",
      "start": 6.3,
      "end": 11.0
    }
  ],
  "usage": {
    "audio_seconds": 184.2,
    "gpu_seconds": 21.5
  }
}

GET /v1/multimodal/jobs/{id}

Response:

{
  "job_id": "mmj_V01",
  "operation": "video_process",
  "status": "succeeded",
  "outputs": {
    "frames": 118,
    "audio_media_id": "med_AUD7"
  },
  "attempts": 1
}

SDK examples

Python

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.multimodal.audio.diarize("med_AUD1", language="fr")
client.multimodal.job(job_id="mmj_V01")

TypeScript

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.multimodal.diarize({ media_id: "med_AUD1", language: "fr" });
await client.multimodal.getJob("mmj_V01");

Java

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.multimodal().diarize(Map.of("media_id", "med_AUD1", "language", "fr"));
client.multimodal().job("mmj_V01");

Go

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Multimodal.Diarize(ctx, map[string]any{"media_id": "med_AUD1", "language": "fr"})
_, err = client.Multimodal.Job(ctx, "mmj_V01")

// worker package: run a media worker consuming multimodal jobs
w, err := worker.New(worker.Config{Source: jobs, Handler: handle, Client: client, Concurrency: 2})
if err != nil { log.Fatal(err) }
err = w.Run(ctx)

Configuration

Environment variables: EAI_PYANNOTE_MODEL.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): workers.items[name=diarization].resources.

Security

• Speaker embeddings are biometric data: the pyannote adapter (EAI_PYANNOTE_MODEL) only returns speaking turns (start, end, speaker) and no embedding is stored.
• Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
• Every mutation and policy decision writes a hash chained audit event (G-11).
• Errors use the standard envelope with message_i18n en and fr.

Observability

• Diarization error rate tracked by G-09 evaluation suites on labelled audio.
• W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units

• Recorded under usage_type stt with labels.operation=diarize: audio_seconds and gpu_seconds.

Failure modes and resilience

• More speakers than the model can separate -> some turns are merged under one speaker label; check turns before relying on the number of speakers.

Standalone adoption

Meeting tools can call only diarization on media they already store elsewhere after uploading to /v1/media.

Acceptance tests

1. A 2 speaker recording returns 2 labels in speakers and turns covering the audio.
2. No embeddings are persisted with the default configuration.

Implementation roadmap

Kit L4 voice evidence

pyannote weights are gated on Hugging Face: the adapter is tested with a mock only (MULT-004 blocked on the weights). See the voice runtime guide.