Elasticsearch of the client

Elasticsearch is never installed by Navin: its licences (AGPL-3.0, SSPL-1.0 or Elastic-2.0) are outside the policy of the platform (ADR-0101) and OpenSearch replaces it. A bank that already runs an Elasticsearch can still send it the logs and the traces of the platform: the Manager reaches it through a connection, as an existing service. Contract: navin-manager/CONTRACT.md (section 10, Elasticsearch of the client).

What connects to it

Service	Binding	What is written
otel-collector	elasticsearch (optional)	OTLP logs in the data stream logs-eai.otel-default (OpenTelemetry mapping), next to OpenSearch when both are bound
jaeger	elasticsearch (optional)	Spans in the indices eai-jaeger-span-* and eai-jaeger-service-*; Elasticsearch takes precedence over OpenSearch when both are bound

Versions

Elasticsearch 8.12 or a later 8.x: the OpenTelemetry mapping of the collector uses the require_data_stream parameter of the bulk API, added in 8.12, and the collector 0.130.0 states the API of Elasticsearch 7.17 and 8.x. The Manager does not know the version of a connection before the install: the preflight asks to confirm it (warning, never blocking). Checked with Elasticsearch 8.19.24.

Declaring the connection

1. Administration, Connections, new connection of type Elasticsearch: one URL per node (https://es-1.bank.example:9200), the user, its password, and the certificate of the authority of the cluster (PEM) when a public authority did not issue it. Leave the certificate check on.
2. Test it: the Manager calls GET / with this user and shows the version of the cluster; a wrong password is refused (auth_failed).
3. In the install wizard of otel-collector or jaeger, choose this connection for the binding elasticsearch.

Rights of the user

The minimal role, created by the Elasticsearch team of the bank (API _security/role or Kibana):

{
  "cluster": ["monitor", "manage_index_templates"],
  "indices": [
    {"names": ["logs-eai.otel-*"], "privileges": ["create_doc", "auto_configure"]},
    {"names": ["eai-jaeger-*"], "privileges": ["create_index", "index", "read", "view_index_metadata"]}
  ]
}

manage_index_templates is for Jaeger, which creates its index templates at start; monitor lets both clients read the state of the cluster.

What the Manager writes

• The configuration of the collector or of Jaeger, a sensitive file, holds the user and the password; the password is never on the command line.
• The certificate of the authority of the connection is written next to it (external-ca.pem, 0644) and the certificate check follows the option of the connection.

What was verified

navin-manager/server/tests/test_elasticsearch_connection_real.py runs Elasticsearch 8.19.24 with security on (HTTPS with a test authority, users of the file realm), the collector 0.130.0 and Jaeger 2.22.0 of the catalog, with the configuration rendered from a connection, then:

• runs the connection test of the Manager: version read with the right password, refusal with a wrong one;
• sends a log to the collector and finds it in the data stream logs-eai.otel-default;
• sends a trace to the collector, which forwards it to Jaeger: it is stored in eai-jaeger-span-* and read back by the query API of Jaeger;
• writes with a user limited to the role above.

test_catalog_obs.py checks the renders (several nodes, certificate of the connection, check turned off, plain HTTP cluster without security, binary and Podman methods) and test_platform_no_agpl.py that no Elasticsearch service, image or download enters the catalog.

Limits

• User and password authentication (native, file, LDAP or Active Directory realm of Elasticsearch): no API key or client certificate for this connection, no Elastic Cloud by cloud ID (give the URL of the endpoint).
• Elasticsearch 9 was not tried.
• The retention (data stream lifecycle, ILM) and the indices stay managed by the Elasticsearch team of the bank.
• OpenSearch Dashboards and OpenSearch Security Analytics do not read this Elasticsearch: the logs and traces written there are read with the tools of the bank (Kibana) and the interface of Jaeger.