Upgrade the Navin Platform

This page upgrades the Navin Platform to a new release. The SQL migrations are forward only and compatible with the previous release (expand then contract), so the old pods keep working while the new ones start. It is done by the platform administrator, in a change window for production.

Before you begin

• Read the release notes of the target version (What's new in release 1.0, Release notes 1.0.0): migrations range, changed settings, known issues.
• Check that the API contracts stay compatible: no breaking change within contracts/v1.
• Check that the current installation is healthy (Verify) and that no migration is changed or unknown (navin-platform status, python -m eai.migrate --status).

OS packages

1. Back up the database. The upgrade applies the new migrations at the first start of the API.

   sudo -u postgres pg_dump --format=custom --file=/var/backups/navin_platform-<date>.dump navin_platform

   For an external server, run the same pg_dump with the administrator login and PGSSLMODE=verify-full.

   Expected result: pg_restore --list /var/backups/navin_platform-<date>.dump prints the table of contents.

2. Install the new package. An upgrade keeps the settings, the data and the database; the venv is rebuilt from the wheelhouse of the new package and the running services are restarted.

   sudo apt update && sudo apt install --only-upgrade navin-platform    # Ubuntu
   sudo dnf upgrade navin-platform                                       # RHEL family
   sudo pacman -Syu                                                      # Arch Linux

   With the offline archive: sudo ./navin-install standalone from the archive of the new version.

   Expected result: navin-platform version prints the new version.

   If it fails: journalctl -u navin-platform-api -n 100 tells why the API does not start.

3. Check the migrations.

   sudo navin-platform-db check

   Expected result: exit status 0. The worker refuses to start while a migration is pending.

4. Compare the settings. /etc/navin-platform/platform.env.example is refreshed by the upgrade; compare it with your platform.env, then run sudo navin-platform check.

Kubernetes

1. Back up. Take an on demand Velero backup before a production upgrade:

   velero backup create pre-upgrade-<version> --from-schedule eai-platform-daily

   CloudNativePG also keeps a continuous WAL archive in s3://eai-backups/cnpg (point in time recovery).

2. Publish the new image tags. The CI pushes the signed images to Harbor and updates deploy/gitops/<env>/image-tags.yaml.

3. Synchronize. dev and staging synchronize automatically. Production is manual, outside the freeze window of the AppProject (from Friday 18:00, for 62 hours):

   argocd app sync -l eai.io/environment=prod,eai.io/layer=apps
   argocd app wait -l eai.io/environment=prod,eai.io/layer=apps --health --timeout 1800

   The PreSync / pre-upgrade hook runs python -m eai.migrate before the Deployments change. Without Argo CD: helm upgrade enterprise-ai deploy/helm/enterprise-ai -n enterprise-ai -f <values files>.

4. Check. /readyz, the service monitoring tab of the Navin Manager, the Live metrics portal screen, GET /v1/audit/verify, the k6 smoke tests of deploy/loadtest.

5. Operators. Change chart_versions in deploy/terraform/modules/platform-bootstrap/variables.tf, then tofu plan and tofu apply, one operator at a time.

Roll back

Installation	Command	Database
Kubernetes	argocd app rollback enterprise-ai-prod <revision>	Kept when the migrations are additive; otherwise CloudNativePG point in time recovery
OS packages	Install the previous package version (apt install navin-platform=<version>, dnf downgrade navin-platform)	Restore the dump of step 1 when the new release applied migrations: migrate --status reports the newer versions as unknown

Isolated site

Build, verify and sign the bundle on the connected side, then verify it on site and compare the manifest release/manifests/<version>.json with the installed one: see Air-gapped installation.

Troubleshooting

Symptom	Cause	Fix
migrate --check reports changed	A released migration file was edited	Restore the released file; never edit an applied migration
migrate --status reports unknown	The database is newer than the code (downgrade)	Install the newer release again, or restore the backup
Pods of the old and new release both run	Rolling update in progress	Normal: the migrations are compatible with the previous release