Marquez (marquez)
Lineage of datasets, jobs and runs collected with OpenLineage: the platform configures the API that receives the lineage events of the RAG ingestion and the agent pipelines, its PostgreSQL database and the lineage web interface.
Generated from catalog/services/marquez.yaml: do not edit by hand.
Summary
| Version | 0.51.1 |
| Use | install and configure, configure and connect |
| Layer | functional (AI catalog) |
| Category | Data platforms |
| Installation methods | podman |
| External connection | no |
| Sandbox only | no |
Licence
| SPDX identifier | Apache-2.0 |
| Licence policy | core licence (MIT or Apache-2.0) |
| Licence exception | - |
| Platform component | openlineage |
Packages and artifacts
A package the project of the service publishes (its image, its binary archive, its Python or OS package) carries the licence of the service; a third-party package carries the licence written on its row, read from the official upstream metadata.
| Kind | Package | Version | Licence | Details |
|---|---|---|---|---|
| container image | docker.io/marquezproject/marquez | 0.51.1 | Apache-2.0 | image key api, used by api, digest sha256:0721c976cff17d8b14f7949d85d6dac9c7ea37cb9fe857caa19833730fcb1a50 |
| container image | docker.io/marquezproject/marquez-web | 0.51.1 | Apache-2.0 | image key web, used by web, digest sha256:7312112ca0e611b0a37b00d3da6a677485ae4a97622b23d3ae24d849a927868b |
Images are pinned by tag in the descriptor, and by digest when the descriptor declares one (the offline bundle then refuses a pulled image that does not carry it); the digest is recorded in the mirror of the Manager when the offline bundle is imported, and every pull or load is checked against it.
Supported systems and architectures
| Method | Operating systems | Architectures |
|---|---|---|
podman | RHEL family (dnf), Debian family (apt), Arch Linux (pacman) | x86_64, aarch64 (not restricted by the descriptor) |
Certified host systems: RHEL 8/9/10, Rocky Linux 8/9/10, AlmaLinux 8/9/10, Oracle Linux 8/9, CentOS Stream 9/10, Ubuntu 22.04/24.04, Debian 12, Arch Linux (rolling). The host preflight accepts these systems (other releases of a known family give a warning, end of life releases are refused) and the x86_64 and aarch64 architectures.
Prerequisites
Host
- at least 2048 MB of memory (warning,
memory).
Network and ports
| Component | Port name | Port | Opened in the host firewall |
|---|---|---|---|
api | api | 5000/tcp | yes |
api | admin | 5001/tcp | no |
web | ui | 3000/tcp | yes |
Services and connections needed
database:postgresql.
Details and resolution: see Dependencies.
Installation methods and modes
| Method | Default | What the method uses |
|---|---|---|
podman | yes | the container images of the descriptor, run by Podman under systemd |
Components
| Component | Kind | Installation methods | Ports |
|---|---|---|---|
Lineage API (api) | master | podman | 5000/tcp, 5001/tcp |
Lineage interface (web) | master | podman | 3000/tcp |
Single server (standalone) (default)
One API and one interface.
What the mode deploys:
| Component | Placement |
|---|---|
api | min 1, max 1 |
web | min 1, max 1 |
Replicated (replicated)
Several stateless API instances behind the load balancer, sharing the database.
What the mode deploys:
| Component | Placement |
|---|---|
api | min 2, recommended 2 |
web | min 1, recommended 1 |
Install this service
The install wizard of the Navin Manager plans, validates and runs the installation; you need the services:install permission on the cluster.
-
Open the wizard. In Clusters, choose the row action Install services of the target cluster. The wizard opens on its first step, Service.
-
Choose the service. On the Service step, select Marquez. The card shows the licence
Apache-2.0and the dependencies the installation needs. -
Choose the hosts and the mode. On the Hosts step, tick the hosts, then choose the Mode:
standalone(default),replicated. Each mode deploys the components listed in Installation methods and modes. With several components, place each one on its hosts in Component placement (Suggest placement proposes a placement that respects the cardinalities). -
Bind the dependencies. On the Bindings step, choose for each dependency (
database) Install with the platform or Use an existing connection; an optional dependency can stay on No binding. -
Configure. On the Configure step, review the parameters (see Configuration parameters). Passwords left empty with a generated default are generated and sealed by the Manager.
-
Review and install. On the Install step the Manager validates the plan and shows the installation order; when it says "The plan is valid. Ready to install.", choose Install. A highly available mode is installed only when its HA checks pass.
-
Verify. The operation ends when the health checks of every instance pass (see Health checks); the service then shows as running in Services.
Connect an existing service
| Schemes | https, http |
| Default port | 5000 |
| Authentication | none |
| Health check | /api/v1/namespaces |
| Authentication method | Fields | Description |
|---|---|---|
none | - | Marquez has no authentication of its own: protect it by the network or an authenticating gateway. |
Official documentation: https://github.com/MarquezProject/marquez
| Field | Type | Default | Description | Notes |
|---|---|---|---|---|
Namespace (namespace) | string | navin | OpenLineage namespace of the platform jobs. | - |
High availability
| Cluster | replicas (replication) |
| High availability | external (external) |
| Failover | load balancer (load-balancer) |
| Replication | shared store (shared-store) |
| Quorum | none (none) |
| Discovery | none (none) |
| Entry point | load balancer (load-balancer) |
| Official documentation | https://marquezproject.ai/docs/deployment/ |
Prerequisites
| Prerequisite | Must be highly available | Modes | Description |
|---|---|---|---|
database | yes | modes of level ha or cluster | Highly available PostgreSQL holding the lineage metadata. |
Topology per mode
| Modes | Highest catalog level | Distribution | Quorum component | Fault domains | Description |
|---|---|---|---|---|---|
standalone | standalone (standalone) | single (single) | - | 1 | One API and one interface. |
replicated | ha (ha) | active-active (active-active) | - | 2 | Several stateless API instances sharing PostgreSQL behind the load balancer. |
Notes
- Running several API instances is not described by the upstream project: validate it on your workload. The web interface stays a single instance unless its count is raised.
Configuration parameters
General
| Parameter | Type | Default | Allowed values | Required | Secret | Restart | Description |
|---|---|---|---|---|---|---|---|
API port (api_port) | port | 5000 | - | no | no | yes | Port of the lineage API (/api/v1/lineage receives the OpenLineage events). |
Admin port (admin_port) | port | 5001 | - | no | no | yes | Port of /healthcheck and /metrics. |
Interface port (web_port) | port | 3000 | - | no | no | yes | Port of the lineage web interface. |
TLS (tls_enabled) | bool | false | - | no | no | yes | Serve the lineage API and the web interface in HTTPS with a certificate of the Manager CA. The API gets a Dropwizard https connector whose PKCS12 key store the container builds at each start from the certificate (openssl of the image, random password); the interface is served in HTTPS by a front of the same routes that calls the API in HTTPS and verifies it with the Manager CA. The admin port (/healthcheck, /metrics) stays HTTP, published on the loopback of the host only. |
Default namespace (default_namespace) | string | navin | - | no | no | no | OpenLineage namespace of the platform jobs. |
Dependencies
For each active dependency the Manager takes the first of: the binding chosen in the wizard; a service of the same installation that provides it; a service already installed in the cluster; the only connection of the expected type. Otherwise an optional dependency stays unbound, and a required one is added to the installation in its default mode (the column below names the catalog service added).
| Binding | Provided by | Connection type | Endpoint | Optional | When nothing is bound | Description |
|---|---|---|---|---|---|---|
database | postgresql | postgresql | sql | no | postgresql is added to the installation | Database of the lineage metadata (namespaces, datasets, jobs, runs). |
Endpoints
| Endpoint | Scheme | Component | Port | Path | TLS parameter | Binding parameters | Binding secrets | Provisioning command |
|---|---|---|---|---|---|---|---|---|
api | http | api | 5000 (api_port) | /api/v1 | tls_enabled | - | - | - |
web | http | web | 3000 (web_port) | - | tls_enabled | - | - | - |
Client configuration files
Files rendered for the client configuration download:
marquez.env(client.env.j2)
Health checks
Liveness (per instance)
| Component | Check | Details |
|---|---|---|
api | HTTP http://127.0.0.1:{{ config.admin_port }}/healthcheck | Expected 200; 60 retries; interval 3 s; timeout 5 s |
web | HTTP {{ 'https' if config.tls_enabled else 'http' }}://{{ host.ip or '127.0.0.1' }}:{{ config.web_port }}/ | Expected 200; 30 retries; interval 2 s; timeout 5 s |
Readiness (service check)
| Check | Details |
|---|---|
HTTP {{ 'https' if config.tls_enabled else 'http' }}://{{ host.ip or '127.0.0.1' }}:{{ config.api_port }}/api/v1/namespaces | Expected 200; Expected namespaces |
Preflight checks
| Check | Type | Meaning | Severity | Components |
|---|---|---|---|---|
memory | memory_mb {"min": 2048} | at least 2048 MB of memory | warning | api |
api_port_free | port_free {"param": "api_port"} | port 5000 free on the host | blocking | api |
Uninstall
Paths removed when the data is purged: {{ paths.log }}.
Quick links
- Marquez lineage:
{{ 'https' if config.tls_enabled else 'http' }}://{{ host.fqdn }}:{{ config.web_port }}/
Upstream documentation
- Homepage: https://marquezproject.ai
- Source: https://github.com/MarquezProject/marquez
- High availability: https://marquezproject.ai/docs/deployment/
- Authentication: https://github.com/MarquezProject/marquez