Analytics export to ClickHouse
The worker job analytics_export (python -m eai.worker, elected leader) copies the new rows of eai.usage_events and eai.telemetry_spans (all tenants, system context) to eai_finops.usage_events and eai_telemetry.spans. It runs only when EAI_CLICKHOUSE_URL is set; otherwise it ends with skipped.
Settings
| Variable | Default | Role |
|---|---|---|
EAI_WORKER_ANALYTICS_EXPORT_SECONDS | 60 | job interval |
EAI_ANALYTICS_EXPORT_BATCH | 5000 | rows per insert |
EAI_ANALYTICS_EXPORT_MAX_BATCHES | 20 | inserts per source and per run |
EAI_ANALYTICS_EXPORT_LAG_SECONDS | 5 | younger rows wait for the next run (in flight transactions) |
EAI_CLICKHOUSE_USER / EAI_CLICKHOUSE_PASSWORD | user eai-writer (INSERT only) | |
EAI_CLICKHOUSE_TABLE | eai_telemetry.spans | optional rename of the spans table (strict database.table) |
DDL
The sql/clickhouse/ files are applied in numeric order (001_usage_events.sql, 002_daily_rollups.sql, 003_telemetry_spans.sql); an applied file is never edited, a change adds the next number. On PostgreSQL, V094__analytics_export.sql creates eai.analytics_export_watermarks.
Idempotency
Each source is read in the order of its monotonic key (updated_at, id for usage, created_at, id for spans). Each batch carries an insert_deduplication_token derived from its bounds: a replayed batch is not inserted twice. The tables are ReplacingMergeTree(version): a re-exported row (reconciled usage event) replaces the previous one. The watermark advances only after a successful insert; on failure it stays unchanged, the bounded error (no credentials) is kept in last_error and the next run replays the same batch (at least once).
Backfill
UPDATE eai.analytics_export_watermarks SET last_ts = NULL, last_id = NULL WHERE source = 'usage_events';
The source restarts from the beginning; duplicates are merged by the ReplacingMergeTree (query with FINAL for an exact result before merges).