Skip to main content

E-13 Object Storage

Brick E: Data, RAG, context, memory. French version: E-13-object-storage

1. Role and responsibilities​

Validated S3 compatible storage (SeaweedFS, Ceph RGW, S3, GCS interop) for documents, media, exports and snapshots, encrypted.

  • Buckets per tenant or prefix
  • SSE encryption with E-04 keys
  • Object Lock for WORM audit

2. Position in the architecture and interactions​

  • Callers: Media (F), Document Processing, Audit export, Qdrant snapshots.
  • Execution path: Synchronous (presigned URLs).
  • Client storage: a storage profile (/v1/storage/profiles) names the bucket or container chosen by the client (AWS S3, Azure Blob, Google Cloud Storage, IBM COS, SeaweedFS or another S3 compatible service); the eai/storage/ adapters speak the REST API of each provider through the outbound guard eai.security.egress, without any vendor SDK.
  • The WORM audit export (G) writes through the tenant audit_worm profile or the platform profile EAI_AUDIT_WORM_STORAGE_PROFILE, refused without a verified Object Lock.
  • Every interaction goes through the Control Plane API v1 (/v1) behind the API Gateway (D-14).

3. Open source component and developed part​

  • Open source: SeaweedFS / Ceph / S3 compatible
  • Developed: Adapter, bucket policy.
  • Developed: unified interface eai/storage/base.py (put, get, head, list, delete, presign, multipart, sha256 verification, WORM write read back, capability discovery) and adapters s3.py (SigV4, IBM IAM), azure.py (Shared Key, SAS, Microsoft Entra ID, workload identity), gcs.py (service account, V4 signed URLs); domain eai/domains/storage/ (profiles, verification, presigned URLs).
  • External integration through an adapter eai/integrations/<system>.py with an in-process fallback when the URL is not configured.

4. Data model​

Tables (schema eai, tenant_id RLS):

  • eai.media
  • eai.kb_documents (object_key)
  • eai.storage_profiles (V125: provider, endpoint, region, bucket, credential_ref only, encryption, discovered capabilities, verified_at; platform profiles have no tenant)
  • eai.storage_capability_checks (V126: history of every verification)
  • eai.storage_presigned_grants (V127: scope, key, expiry and sha256 of each presigned URL, never the URL)
  • eai.audit_exports.storage_profile_id, worm_qualified (V128)

Common columns: id uuid, tenant_id uuid NOT NULL, created_at, updated_at; RLS policy current_setting('app.tenant_id', true)::uuid.

5. Contract​

  • Applied contract: multimodal-contract
  • JSON Schema: contracts/v1/multimodal-contract.schema.json
  • Key fields: media_id, object_key, sha256
  • Validation: POST /v1/contracts/{name}/validate; manifest apply: POST /v1/manifests/apply.
  • Storage profile contract: contracts/v1/storage-profile.schema.json (credential reference only, typed capabilities {supported, method, detail, checked_at}).

6. API endpoints​

  • POST /v1/media
  • GET /v1/media/{id}
  • DELETE /v1/media/{id}
  • GET, POST /v1/storage/profiles; GET, PATCH, DELETE /v1/storage/profiles/{profile_id}
  • POST /v1/storage/profiles/{profile_id}/verify (capability discovery by real calls), GET /v1/storage/profiles/{profile_id}/checks
  • POST /v1/storage/profiles/{profile_id}/presign (read or write scope, key prefix, bounded expiry)
  • Details and roles: control-plane-v1; guide: storage-administration

Request POST /v1/media:

{
  "filename": "contrat.pdf",
  "content_type": "application/pdf",
  "size": 482113
}

Response:

{
  "id": "med_44",
  "upload_url": "https://<s3-host>/eai-tenant-a/...",
  "status": "pending_upload"
}

7. SDK examples​

Python:

from enterprise_ai import EnterpriseAI

client = EnterpriseAI(base_url="<control-plane-url>", api_key="eai_xxx_secret")
med = client.multimodal.upload(open("contrat.pdf", "rb").read(), "contrat.pdf", "application/pdf")

TypeScript:

import { EnterpriseAI } from "@enterprise-ai/sdk";

const client = new EnterpriseAI({ baseUrl: "<control-plane-url>", apiKey: process.env.EAI_API_KEY });
const med = await client.media.upload({ file, filename: "invoice.pdf" });

Java:

EnterpriseAIClient client = EnterpriseAIClient.builder()
    .baseUrl("<control-plane-url>")
    .apiKey(System.getenv("EAI_API_KEY"))
    .build();
Media med = client.media().upload(Path.of("contrat.pdf"));

Go:

c, _ := eai.NewClient(eai.WithBaseURL("<control-plane-url>"), eai.WithAPIKey(os.Getenv("EAI_API_KEY")))
med, _ := c.Media.Upload(ctx, "contrat.pdf")

8. Configuration​

  • Environment variables: EAI_S3_ENDPOINT, EAI_S3_BUCKET, EAI_S3_REGION, EAI_DATABASE_URL
  • Storage profiles: EAI_AUDIT_WORM_STORAGE_PROFILE, EAI_STORAGE_CA_BUNDLE, EAI_STORAGE_REQUEST_TIMEOUT_SECONDS, EAI_STORAGE_PRESIGN_MAX_SECONDS, EAI_STORAGE_CAPABILITY_MAX_AGE_SECONDS, EAI_EGRESS_PRIVATE_ALLOWLIST (on premises endpoints), EAI_CRED_<NAME> (credentials of platform profiles); see configuration
  • Helm values keys: controlPlane.config, controlPlane.secrets

9. Security controls​

  • No public bucket
  • Short lived presigned URLs
  • No secret in a profile: credential_ref to OpenBao (tenant) or env://EAI_CRED_<NAME> (platform), refused with 422 storage_secret_in_profile
  • TLS required outside dev and test (422 storage_tls_required); outbound calls through the egress guard only
  • Capabilities never assumed: a usage is refused with 409 storage_capability_missing when the capability is not verified, unknown or too old; WORM never simulated (retention read back, 502 storage_worm_not_honoured)
  • Policy decision before each action (storage.profile.*, storage.presign.*)
  • Every request carries the tenant resolved from the credential; PostgreSQL RLS
  • Every mutation and policy decision writes a hash-chained audit event

10. Observability​

  • Prometheus metrics: eai_object_bytes_stored, eai_object_requests_total
  • Traces: OpenTelemetry spans with GenAI semantic conventions (gen_ai.operation.name, gen_ai.agent.id, gen_ai.tool.name), attributes eai.tenant_id, eai.run_id, traceparent propagation.
  • Audit actions: media.upload, media.delete
  • Profile audit actions: storage.profile.create, storage.profile.update, storage.profile.delete, storage.profile.verify, storage.presign.issue, storage.capability.refused

11. FinOps units​

  • Stored GB-month, requests, egress.
  • Emitted as Usage Event (POST /v1/finops/usage) with tenant, project, agent and run.

12. Failure modes and resilience​

  • Resumable multipart upload
  • Versioning enabled
  • Storage unreachable: 502 storage_unreachable; credential refused: 502 storage_auth_failed; backend refusal: 502 storage_backend_error (no secret in the details)
  • Failed verification: profile failed with last_error, usages refused until a successful verification
  • Normalized errors (error envelope, message_i18n en/fr)

13. Standalone adoption (client of a client)​

  • enterprise_ai.multimodal + S3 compatible storage.
  • SDK: enterprise_ai.privacy / dlp / knowledge_bases / memory / retention / lineage
  • Minimal deployment: Helm chart with controlPlane.enabled=true and PostgreSQL; disabled external components use their in-process fallback.

14. Acceptance tests​

  1. Object of a tenant inaccessible to another
  2. Cross-tenant access refused
  3. Error returned in fr with Accept-Language: fr
  4. Upload, download, metadata, delete and sha256 integrity per adapter (SeaweedFS, Azurite, fake-gcs-server): services/control-plane/tests/test_storage_*.py
  5. Capability discovery true and false, refusal of a missing capability, expired and out of scope presigned URLs refused
  6. WORM audit export refused without a verified Object Lock, qualified and not deletable on an Object Lock bucket