Metrics and alerts

Scrape the private operator origin:

scrape_configs:
  - job_name: llm-gateway
    scrape_interval: 15s
    static_configs:
      - targets: [127.0.0.1:4000]
    metrics_path: /metrics

Metrics

Metric	Type	Monitor
`gateway_http_requests_total`	Counter	Traffic, status, error rate by normalized route
`gateway_http_request_duration_seconds`	Histogram	End-to-end latency including provider time
`gateway_realtime_active_sessions`	Gauge	Active realtime sessions by alias and provider
`gateway_realtime_sessions_total`	Counter	Completed realtime sessions by alias, provider, and status
`gateway_realtime_session_duration_seconds_sum`	Counter	Aggregate realtime connected seconds
`gateway_realtime_frames_total` / `gateway_realtime_bytes_total`	Counter	Aggregate frame/byte volume by direction, never content
`gateway_request_logs_dropped_total`	Counter	Lost request-usage metadata
`gateway_structured_logs_dropped_total`	Counter	Structured stdout records lost after queue overflow or sink failure
`gateway_ready`	Gauge	Aggregate data-plane serving readiness
`gateway_component_ready`	Gauge	Engine, model catalog, persistence, shared state, admin audit, logger, alerts, reconciliation, retention, signals

HTTP route labels use registered route templates, including for failures raised before endpoint dispatch. Unknown paths are reported as unmatched; client-controlled path segments are never emitted as metric labels. Non-standard HTTP methods share the OTHER label, keeping method cardinality bounded. GET /ready includes serving_ready and component flags. Catalog lag or activation failure makes status degraded and model_catalog_ready false but does not drain a replica that retains a safe runtime; gateway_ready remains 1. Alert on the model-catalog component so operators repair propagation independently of traffic routing. /ready and /metrics share a one-second dependency-probe cache by default, and concurrent refreshes coalesce. Configure observability.readiness_cache_ttl_seconds; the value is also the maximum extra failure or recovery detection delay, and 0 disables caching.

Calibrated signals and digests

With Postgres persistence, optional signals rules evaluate tenant/application/environment metadata for sustained 5xx rate, 429 rate, missing usage, and p95 latency. Minimum samples plus consecutive breach and recovery windows reduce noise. Latency combines an absolute threshold with previous-window regression when a baseline exists. Firing, reminder, resolved, and completed-week digest notifications use the same durable Slack or Google Chat destination as budget alerts. Use GET /admin/signals/issues and GET /admin/signals/digests on the private operator origin. POST /admin/signals/run is an audited admin-only drill. A failed signals worker degrades signals_ready without interrupting inference. Signals use request metadata only; prompt, response, audio, and realtime-frame content are not inspected.

Included Prometheus rules

deploy/observability/gateway-alerts.yaml contains alerts for target loss, aggregate/component readiness, sustained 5xx rate, high p95 inference latency, request-log drops, and structured-log drops. Structured stdout is drained through a bounded process-local background queue so collector backpressure cannot block inference; alert on gateway_structured_logs_dropped_total to detect queue overflow or sink failures. Load the file into Prometheus and route warning and critical labels through Alertmanager. Thresholds are starting points; tune them after observing your providers and workload. The optional deploy/compose/observability.yaml overlay supplies private loopback Prometheus and Alertmanager services plus a shared Docker secret for the notification relay and gateway budget outbox. Select slack or google_chat with LLM_GATEWAY_NOTIFICATION_PROVIDER, keep its webhook in a mode-0600 host file referenced by LLM_GATEWAY_NOTIFICATION_WEBHOOK_FILE, and enable the matching alerts.*_webhook_url_file field in the environment gateway YAML. Run scripts/stage_observability_smoke.py, then confirm both firing and resolved synthetic messages at the receiver. Separately exhaust a disposable budget and require its durable alert row to become delivered. For local Compose, keep the host secret directory root-only and assign the mode-0600 webhook file to numeric UID/GID 10001:10001. Compose bind-mounts the file without remapping ownership; a root-owned file is unreadable to the non-root gateway and relay and startup fails closed.

Recommended dashboards

Request rate and 2xx/4xx/429/5xx split.
p50/p95/p99 by inference route.
Readiness by component.
Request-log drop increase.
Structured-log drop increase.
Usage, total tokens, estimated cost, and error rate by application/environment/model/provider.
Budget utilization, stale reservations, and alert delivery failures.
Soft-quarantined budget reservations and repeated reconciliation error types from structured logs.

Metrics reset when a process restarts. Prometheus remains the time-series authority. Usage and budgets come from Postgres APIs.

Overview

Get Started

Administration

API Reference

Operations

Metrics

Calibrated signals and digests

Included Prometheus rules

Recommended dashboards

​Metrics

​Calibrated signals and digests

​Included Prometheus rules

​Recommended dashboards

Metrics

Calibrated signals and digests

Included Prometheus rules

Recommended dashboards