Skip to main content
Run the supported Docker Compose topology on a private Linux VM in your cloud account. Applications reach only a TLS reverse proxy or tunnel; administrators and Prometheus use the private loopback gateway port.

​
Architecture

  • Gateway, Postgres, Redis, and public Nginx proxy on one host.
  • Public TLS endpoint forwards to 127.0.0.1:4082.
  • Private operator access reaches 127.0.0.1:4000 through SSH, VPN, or a private network.
  • Postgres and Redis have no host ports in the baseline Compose file.

​
Install

Edit deploy/gateway.yaml:
  1. Set public_base_url to the external HTTPS origin.
  2. Give the environment a unique shared_state.key_prefix.
  3. Keep only the providers and model routes you intend to operate.
  4. Review model capabilities, unsupported parameters, and pricing snapshots.
  5. Keep concrete provider deployments internal and expose stable aliases.
Populate deploy/compose/.env from your secret manager. Do not commit it. The optional monitoring overlay also uses LLM_GATEWAY_NOTIFICATION_PROVIDER (slack or google_chat) and LLM_GATEWAY_NOTIFICATION_WEBHOOK_FILE, which must point to a mode-0600 host file containing one approved HTTPS incoming-webhook URL. The file is mounted read-only rather than copied into the container environment. Local Compose does not remap bind-mounted file ownership: keep the parent directory root-only and own the mode-0600 file as numeric UID/GID 10001:10001 so the non-root gateway and relay can read it. Generate independent random bootstrap values, for example with openssl rand -hex 32. If you remove a provider from deploy/gateway.yaml, also remove its required environment entry from the gateway service in deploy/compose/gateway.yaml. Open http://127.0.0.1:4000/admin/ui over the operator tunnel and sign in as gateway-admin using GATEWAY_BOOTSTRAP_ADMIN_PASSWORD. The bootstrap admin API key is for scripts such as the stage gate; do not paste it into the console. The supplied loopback-only deployment sets admin_session_cookie_secure: false so cookies work through that HTTP SSH tunnel. Set it to true when the private operator origin itself uses HTTPS; never expose the HTTP operator origin beyond loopback or a trusted private tunnel.
To include private Prometheus and Alertmanager services, select the receiver and enable the matching alerts.slack_webhook_url_file or alerts.google_chat_webhook_url_file field in the gateway YAML, then add -f deploy/compose/observability.yaml to the command. Their host ports default to loopback-only 4092 and 4093.

​
Expose safely

Configure your TLS proxy, load balancer, or tunnel to forward only to 127.0.0.1:4082. Never point it to port 4000. Verify:
For an internal pilot where every application is already authorized on the same Tailscale tailnet, sudo python3 scripts/configure_tailscale_https.py --apply creates and verifies a tailnet-only HTTPS/WSS origin on the node’s stable MagicDNS name. Use the reported HTTPS origin as public_base_url. This is not public ingress and must not use Tailscale Funnel; external customer traffic requires a managed TLS endpoint and a separate ingress review. From the operator network:
For a remote VM, create a local operator tunnel instead of opening port 4000:

​
Validate before use

Run the disposable-key stage gate with the public and private origins:
Require status: passed, key_revoked: true, and operator_surface: verified_private. Then create one key per application/environment and run a real task before migrating traffic. No certified VM size is published yet. Measure your models, payloads, streaming duration, and concurrency with representative load before production use.