Skip to main content

LiteLLM config

The LiteLLM config screen lets you read the configuration the Control Plane generates from the models registry for the LiteLLM proxy, and push it to the proxy. Portal path: /llm/litellm. It is used by the AI platform administrators after a change of providers or models, and by the security officers and auditors who check what the proxy serves. The screen is built in portal/src/screens/llm.tsx (LiteLLM).

What the screen shows​

  • The header button Sync to LiteLLM.
  • After a synchronization, the JSON answer of the sync: mode, the models added and removed, and the number of models. When EAI_LITELLM_URL is not set, the mode is in_process: the configuration is generated and nothing is pushed.
  • The section Generated configuration ("Secrets are referenced, never included."): the JSON returned by GET /v1/litellm/config, with its model_list. Only models with the status approved, not disabled and whose provider is enabled are included; credentials are os.environ/<NAME> references resolved by the LiteLLM deployment, never secret values.
  • When the list is empty: "The generated configuration is empty. Register approved models, then synchronize." with a Sync to LiteLLM button.
  • For a platform_admin only, the section Operator consoles with the link Open the console of LiteLLM (model gateway), read from GET /v1/platform-manager/console, or the reason why it is not configured.

Actions​

Button or actionWhat it doesAPI callRoles
Sync to LiteLLM (header or empty state)Generates the configuration and pushes the model list to the proxy. Toast LiteLLM configuration synchronized, then reloads the configuration.POST /v1/litellm/sync, then GET /v1/litellm/configtenant_admin, platform_admin, ai_platform_engineer (the button is shown to everyone, the server decides)
Retry (error panel)Reads the configuration again.GET /v1/litellm/configtenant_admin, platform_admin, ai_platform_engineer, security_officer, auditor
Open the consoleOpens the LiteLLM console in a new tab (https addresses without credentials only).GET /v1/platform-manager/consoleplatform_admin

Roles​

  • Reading GET /v1/litellm/config: ADMIN plus security_officer and auditor in services/control-plane/src/eai/domains/llm/__init__.py, that is tenant_admin, platform_admin, ai_platform_engineer, security_officer, auditor. Other roles see an error panel.
  • Synchronization POST /v1/litellm/sync: tenant_admin, platform_admin, ai_platform_engineer (ROLES.llmAdmins of portal/src/lib/kit.ts). The portal does not hide the button: other roles get forbidden.
  • Operator consoles: shown only when the user holds platform_admin.

Typical tasks​

Publish a new model to the proxy​

  1. Check the model. In the Models registry, the model must be approved, enabled, and its provider enabled, because only those enter the configuration. Expected result: the model row shows approved. If it fails: change its status, or enable its provider in Providers.
  2. Synchronize. Click Sync to LiteLLM. Expected result: toast LiteLLM configuration synchronized, the JSON answer lists the model under added. If it fails: "The model provider returned an error." (upstream_error, 502) means the proxy did not answer; check the proxy and EAI_LITELLM_URL.
  3. Read the result. Check Generated configuration, so that you confirm the entry and its credential reference. Expected result: the model is in model_list, its key is an os.environ/ reference. If it fails: the list is empty when no model is approved.

Check the proxy configuration during an audit​

  1. Open the screen. As auditor or security_officer, open LLM management > LiteLLM config. Expected result: the JSON of Generated configuration. If it fails: an error panel with forbidden means your role cannot read it.
  2. Look for secrets. Read the litellm_params of each entry, so that you verify no secret value is exposed. Expected result: credentials are references only. If it fails: report it as a security incident.

Troubleshooting​

SymptomCauseFix
"The generated configuration is empty"No model is approved, enabled and served by an enabled provider.Approve a model in the Models registry, then Sync to LiteLLM.
Sync answer with mode: in_process and "EAI_LITELLM_URL not set: config generated, nothing pushed"The proxy address is not configured.Set EAI_LITELLM_URL in the Control Plane settings, then synchronize again.
"The model provider returned an error." (upstream_error, 502)The LiteLLM proxy did not answer the push.Check that the proxy runs and is reachable, then retry.
forbidden (403) on Sync to LiteLLMThe role cannot synchronize.Ask a tenant_admin or an ai_platform_engineer.
Operator consoles says the console is not configuredNo console address is known.Set the setting named in the message or install the service with the Navin Manager.