Skip to main content

F-07 Text to speech

Brick F of the Enterprise AI Platform. Technology: Kokoro v1.0 (Apache-2.0 weights, kokoro-onnx on CPU, reference engine) and provider TTS through an OpenAI compatible endpoint. Piper is not shipped (piper1-gpl is GPL-3.0). French version: ../../fr/bricks/F-07-text-to-speech.md.

Role and responsibilities​

  • Synthesize speech in French and English with approved voices, SSML subset, sample rate and format.
  • Keep a registry of validated engines and voices with license and consent status.

Position and interactions​

  • Execution path: synchronous or streaming (chunks used by F-08).
  • Used by agents to answer by voice, by channels (telephony) and by F-08 voice live.

Open source and developed parts​

  • Open source reused: Kokoro v1.0 ONNX weights and voices (Apache-2.0); espeak-ng phonemizer (GPL-3.0) isolated in the voice worker process.
  • Developed in the platform: Engine registry, voice approval workflow, text normalization fr/en (numbers, dates, currency), streaming chunker.

Data model​

Control Plane tables (schema eai): eai.media, eai.multimodal_jobs, eai.usage_events.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/multimodal-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
POST/v1/multimodal/tts
POST/v1/gateway/audio/speech

POST /v1/multimodal/tts​

Request:

{
  "text": "Votre dossier a bien été enregistré.",
  "voice": "ff_siwis",
  "model": "kokoro"
}

Response:

{
  "job_id": "mmj_T01",
  "status": "succeeded",
  "media_id": "med_TTS1",
  "usage": {
    "characters": 37,
    "audio_seconds": 2.6
  }
}

POST /v1/gateway/audio/speech​

Request:

{
  "model": "tts-1",
  "input": "Your request is registered.",
  "voice": "alloy",
  "response_format": "mp3"
}

Response:

{
  "media_id": "med_TTS2",
  "usage": {
    "characters": 27
  }
}

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")
client.multimodal.tts.synthesize("Votre dossier a bien été enregistré.", voice="ff_siwis", model="kokoro")
client.gateway.speech(model="tts-1", input="Your request is registered.", voice="alloy")

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.multimodal.tts({ text: "Votre dossier a bien été enregistré.", voice: "ff_siwis", model: "kokoro" });
await client.gateway.audio.speech({ model: "tts-1", input: "Your request is registered.", voice: "alloy" });

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.multimodal().tts(Map.of("text", "Votre dossier a bien été enregistré.", "voice", "ff_siwis", "model", "kokoro"));
byte[] audio = client.gateway().audioSpeech(Map.of("model", "tts-1", "input", "Your request is registered.", "voice", "alloy"));

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Multimodal.Tts(ctx, map[string]any{"text": "Votre dossier a bien été enregistré.", "voice": "ff_siwis", "model": "kokoro"})
_, err = client.Gateway.AudioSpeech(ctx, map[string]any{"model": "tts-1", "input": "Your request is registered.", "voice": "alloy"})

// worker package: run a media worker consuming multimodal jobs
w, err := worker.New(worker.Config{Source: jobs, Handler: handle, Client: client, Concurrency: 2})
if err != nil { log.Fatal(err) }
err = w.Run(ctx)

Configuration​

Environment variables: EAI_TTS_MODEL.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): workers.items[name=tts].replicas.

Security​

  • Voice cloning is disabled unless a signed consent of the voice owner is attached to the voice record.
  • Text goes through DLP before synthesis when the channel is public.
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Time to first audio chunk (eai_tts_ttfb_ms) for streaming usage.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • usage_type tts: characters, audio_tokens when the provider bills them, audio_seconds produced.

Failure modes and resilience​

  • Engine unavailable -> fallback to another approved voice of the same language, flagged in the response.

Standalone adoption​

A product can use TTS alone for accessible content generation.

Acceptance tests​

  1. French text with numbers and amounts is normalized and produces a WAV of the expected duration.
  2. Characters counted in the Usage Event equal the input length.

Implementation roadmap

Kit L4 voice evidence​

Kokoro v1.0 int8 (Apache-2.0, sha256 pinned) synthesizes the test speech; realtime response.create streams it and a barge-in cancels it (MULT-006, MULT-010). See the voice runtime guide.