Skip to main content

F-04 Video generation

Brick F of the Enterprise AI Platform. Technology: Specialized GPU workers (open video models, for example Wan or LTX-Video under compatible license) + provider adapters through /v1/gateway/video/generations. French version: ../../fr/bricks/F-04-video-generation.md.

Role and responsibilities​

  • Text to video and image to video generation with explicit seconds and size (resolution); FPS and mode are not request parameters of the API.
  • Schedule jobs on dedicated GPU pools (node selectors, tolerations, KEDA queue scaling).

Position and interactions​

  • Execution path: long asynchronous workflow with reservation.
  • Estimate (H-03) -> reservation (H-01) -> GPU job -> commit with measured GPU-seconds -> media stored with provenance.
  • Specialized adapters are listed by GET /v1/adapters (brick B).

Open source and developed parts​

  • Open source reused: Open video diffusion models, Diffusers pipelines, FFmpeg for muxing.
  • Developed in the platform: Job orchestration, GPU pool scheduler, progress events, provenance, provider adapter normalization.

Data model​

Control Plane tables (schema eai): eai.media, eai.multimodal_jobs, eai.quota_reservations, eai.gpu_samples, eai.usage_events.

Every tenant table has tenant_id uuid NOT NULL and a row level security policy on current_setting('app.tenant_id', true)::uuid. ClickHouse usage_events is the analytics replica of eai.usage_events (ReplacingMergeTree keyed by event_id).

Contract: contracts/v1/multimodal-contract.schema.json (schema_version 1.0). Usage is always reported with contracts/v1/usage-event.schema.json, audit with contracts/v1/audit-event.schema.json.

API endpoints​

MethodPath
POST/v1/multimodal/video/generate
POST/v1/gateway/video/generations
GET/v1/gateway/video/generations/{id}
GET/v1/multimodal/jobs/{id}

POST /v1/multimodal/video/generate​

Request:

{
  "model": "video-gen-gpu",
  "prompt": "15 second product teaser, slow dolly shot",
  "seconds": 15,
  "size": "1280x720"
}

Response:

{
  "job_id": "mmj_G15",
  "status": "queued",
  "estimated": {
    "amount": "1.80",
    "currency": "EUR"
  },
  "reservation_id": "qres_51"
}

POST /v1/gateway/video/generations​

Request:

{
  "model": "provider-video",
  "prompt": "Product teaser",
  "seconds": 8,
  "size": "1280x720"
}

Response:

{
  "id": "vid_88",
  "status": "in_progress"
}

GET /v1/gateway/video/generations/{id}​

Response:

{
  "id": "vid_88",
  "status": "completed",
  "media_id": "med_VID9"
}

GET /v1/multimodal/jobs/{id}​

Response:

{
  "job_id": "mmj_V01",
  "operation": "video_process",
  "status": "succeeded",
  "outputs": {
    "frames": 118,
    "audio_media_id": "med_AUD7"
  },
  "attempts": 1
}

SDK examples​

Python​

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_live_xxx", tenant="acme", locale="en")

with client.quotas.reservation(agent_id="agt_ap", amount="1.80", currency="EUR") as r:
    result = client.multimodal.video.generate("Teaser", model="video-gen-gpu", seconds=15)
    r.commit(cost=result.get("cost_measured"))  # released automatically on exception
estimate = client.finops.estimate("openai", usage_type="llm", model_id="gpt-4o-mini", units={"input_tokens": 12000, "output_tokens": 2000})
client.multimodal.video.generate("Product teaser", model="video-gen-gpu", seconds=15, size="1280x720")
client.gateway.video(model="provider-video", prompt="Product teaser", seconds=8)
client.gateway.video_status("vid_88")
client.multimodal.job(job_id="mmj_V01")

TypeScript​

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY, tenant: "acme", locale: "en" });
await client.multimodal.generateVideo({ model: "video-gen-gpu", prompt: "Product teaser", seconds: 15, size: "1280x720" });
await client.gateway.video.generate({ model: "provider-video", prompt: "Product teaser", seconds: 8 });
await client.gateway.video.get({ id: "vid_88" });
await client.multimodal.getJob("mmj_V01");

Java​

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .tenant("acme")
    .locale(Locale.ENGLISH)
    .build();
client.multimodal().generateVideo(Map.of("model", "video-gen-gpu", "prompt", "Product teaser", "seconds", 15, "size", "1280x720"));
client.gateway().videoGenerations(Map.of("model", "provider-video", "prompt", "Product teaser", "seconds", 8));
client.gateway().videoGeneration("vid_88");
client.multimodal().job("mmj_V01");

Go​

client, err := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")), eai.WithTenant("acme"), eai.WithLocale("en"))
if err != nil { log.Fatal(err) }
ctx := context.Background()
_, err = client.Multimodal.GenerateVideo(ctx, map[string]any{"model": "video-gen-gpu", "prompt": "Product teaser", "seconds": 15, "size": "1280x720"})
_, err = client.Gateway.VideoGenerations(ctx, map[string]any{"model": "provider-video", "prompt": "Product teaser", "seconds": 8})
_, err = client.Gateway.VideoGeneration(ctx, "vid_88")
_, err = client.Multimodal.Job(ctx, "mmj_V01")

// worker package: run a media worker consuming multimodal jobs
w, err := worker.New(worker.Config{Source: jobs, Handler: handle, Client: client, Concurrency: 2})
if err != nil { log.Fatal(err) }
err = w.Run(ctx)

Configuration​

Environment variables: EAI_VIDEO_GEN_URL, EAI_MULTIMODAL_API_KEY.

Helm values keys (deploy/helm/enterprise-ai/values.yaml): workers.items[name=video-gen].nodeSelector, workers.items[name=video-gen].keda.maxReplicaCount.

Security​

  • Generation above a cost threshold requires an approval (POST /v1/approvals) before the reservation.
  • Likeness of real persons is denied by policy unless an explicit consent record exists.
  • Tenant resolved from the credential, never from the body; PostgreSQL row level security on every tenant table.
  • Every mutation and policy decision writes a hash chained audit event (G-11).
  • Errors use the standard envelope with message_i18n en and fr.

Observability​

  • Progress events (percent, step) on NATS and job; GPU utilization joined from DCGM samples.
  • W3C traceparent propagated, X-Request-Id returned, spans carry eai.tenant_id, eai.project, eai.agent_id, eai.run_id.

FinOps units​

  • usage_type video_generation: video_seconds, video_resolution, video_fps, video_mode; local runs add gpu_seconds, gpu_memory_gb_seconds, energy_wh.

Failure modes and resilience​

  • Preemption of a spot GPU node -> resume from the last saved latent checkpoint.
  • Reservation expires during a long job -> automatic extension while heartbeats continue.

Standalone adoption​

A media team can run only the video generation workers and FinOps bricks, with its own GPU pool.

Acceptance tests​

  1. A 15 s 720p 24 fps request returns an estimate, a reservation and finally a media of 15 s.
  2. Cancelling a job releases the reservation and records only the consumed GPU-seconds.

Implementation roadmap