F-05 Speech to text

Brick F of the Enterprise AI Platform. Technology: faster-whisper (CTranslate2) on GPU or CPU workers + LiteLLM transcription for external providers. French version: ../../fr/bricks/F-05-speech-to-text.md.

Role and responsibilities

• Transcribe audio with language detection, word timestamps, optional translation to English.
• Word timestamps: word_timestamps: true on POST /v1/multimodal/stt adds output.words (start, end, word; masked words when DLP masks); on POST /v1/gateway/audio/transcriptions, timestamp_granularities (word, segment) needs response_format=verbose_json and is forwarded as timestamp_granularities[]. An unsupported value or provider gives a translated 422.
• Optionally chain F-06 diarization and F-11 audio PII redaction on the transcript.

Position and interactions

• Execution path: synchronous for short audio, asynchronous job for long audio, streaming in F-08.
• POST /v1/multimodal/stt always creates a job in eai.multimodal_jobs: the local faster-whisper adapter (EAI_WHISPER_MODEL, EAI_WHISPER_DEVICE) or the OpenAI compatible adapter (EAI_STT_URL, EAI_STT_MODEL) processes it, and the client follows the job with GET /v1/multimodal/jobs/{id}.
• The transcript can be ingested in a knowledge base (brick E) with the media classification inherited.

Open source and developed parts

• Open source reused: faster-whisper, CTranslate2, Silero VAD for chunking.
• Developed in the platform: Chunking strategy, language policy, transcript format (segments, words), unified response across providers.

Data model

Control Plane tables (schema eai): eai.media, eai.multimodal_jobs, eai.usage_events.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/multimodal-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints

Method	Path
POST	/v1/multimodal/stt
POST	/v1/gateway/audio/transcriptions
GET	/v1/multimodal/jobs/{id}

POST /v1/multimodal/stt

Request:

{
  "media_id": "med_AUD1",
  "model": "faster-whisper-large-v3",
  "language": "fr",
  "dlp_mode": "detect"
}

Response:

{
  "job_id": "mmj_S01",
  "status": "succeeded",
  "text": "Bonjour, je vous appelle au sujet de ma facture.",
  "language": "fr",
  "segments": [
    {
      "start": 0.0,
      "end": 2.8,
      "text": "Bonjour, je vous appelle"
    }
  ],
  "usage": {
    "audio_seconds": 184.2,
    "output_tokens": 912
  }
}

POST /v1/gateway/audio/transcriptions

Request: multipart/form-data with a file part (the audio) and the fields model (required), language, prompt, response_format (json, text, srt, verbose_json, vtt), temperature and eai, for example curl -F file=@meeting.mp3 -F model=whisper-1 -F response_format=verbose_json. The same route also accepts JSON with the audio in content_base64 (plus mime_type and filename):

{
  "model": "whisper-1",
  "filename": "meeting.mp3",
  "mime_type": "audio/mpeg",
  "content_base64": "SUQzBAAAAAAA...",
  "language": "fr",
  "response_format": "verbose_json"
}

Response:

{
  "text": "Bonjour, je vous appelle au sujet de ma facture.",
  "duration": 184.2
}

GET /v1/multimodal/jobs/{id}

Response:

{
  "job_id": "mmj_V01",
  "operation": "video_process",
  "status": "succeeded",
  "outputs": {
    "frames": 118,
    "audio_media_id": "med_AUD7"
  },
  "attempts": 1
}

SDK examples

Python

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.multimodal.stt.transcribe("med_AUD1", model="faster-whisper-large-v3", language="fr")
client.gateway.transcribe("whisper-1", open("call.wav", "rb").read(), filename="call.wav", language="fr")
client.multimodal.job(job_id="mmj_V01")

TypeScript

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.multimodal.stt({ media_id: "med_AUD1", model: "faster-whisper-large-v3", language: "fr" });
await client.gateway.audio.transcribe({ model: "whisper-1", fileMediaId: "med_AUD1" });
await client.multimodal.getJob("mmj_V01");

Java

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.multimodal().stt(Map.of("media_id", "med_AUD1", "model", "faster-whisper-large-v3", "language", "fr"));
client.gateway().audioTranscriptions(Map.of("model", "whisper-1", "content_base64", "UklGRiQAAABXQVZF", "mime_type", "audio/wav"));
client.multimodal().job("mmj_V01");

Go

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Multimodal.Stt(ctx, map[string]any{"media_id": "med_AUD1", "model": "faster-whisper-large-v3", "language": "fr"})
_, err = client.Gateway.AudioTranscriptions(ctx, map[string]any{"model": "whisper-1", "content_base64": "UklGRiQAAABXQVZF", "mime_type": "audio/wav", "filename": "call.wav"})
_, err = client.Multimodal.Job(ctx, "mmj_V01")

// worker package: run a media worker consuming multimodal jobs
w, err := worker.New(worker.Config{Source: jobs, Handler: handle, Client: client, Concurrency: 2})
if err != nil { log.Fatal(err) }
err = w.Run(ctx)

Configuration

Environment variables: EAI_STT_MODEL.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): controlPlane.config.EAI_WHISPER_DEVICE, workers.items, workers.nats.

Security

• Audio and transcript keep the media classification; transcripts of secret media never leave local models.
• Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
• Every mutation and policy decision writes a hash chained audit event (G-11).
• Errors use the standard envelope with message_i18n en and fr.

Observability

• Real time factor (processing seconds / audio seconds) exported as eai_stt_rtf histogram.
• W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units

• usage_type stt: audio_seconds (billing basis of most providers) and output tokens when the provider bills tokens; local runs add gpu_seconds.

Failure modes and resilience

• Unsupported codec -> transcode by F-12 then retry.
• Hallucinated text on silence -> VAD pre-filter removes non speech segments.

Standalone adoption

A call center can adopt STT alone with the media and FinOps APIs, without agents.

Acceptance tests

1. A French 3 minute WAV gives WER below the target on the reference set and audio_seconds=184.2.
2. A 2 hour file returns a job id and completes with segment timestamps.

Implementation roadmap

Kit L4 voice evidence

faster-whisper tiny.en on CPU through the HTTP voice worker (tests/test_voice_engines_real.py): Kokoro speech transcribed, word overlap 1.00. Realtime transcripts filtered by the DLP before storage (MULT-003, MULT-009). See the voice runtime guide.