> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.aiplanet.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics and alerts

> Prometheus metrics, readiness probes, structured logs, and budget alerts.

Scrape the private operator origin:

```yaml theme={null}
scrape_configs:
  - job_name: llm-gateway
    scrape_interval: 15s
    static_configs:
      - targets: [127.0.0.1:4000]
    metrics_path: /metrics
```

## Metrics

| Metric                                                           | Type      | Monitor                                                                                                           |
| ---------------------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------- |
| `gateway_http_requests_total`                                    | Counter   | Traffic, status, error rate by normalized route                                                                   |
| `gateway_http_request_duration_seconds`                          | Histogram | End-to-end latency including provider time                                                                        |
| `gateway_realtime_active_sessions`                               | Gauge     | Active realtime sessions by alias and provider                                                                    |
| `gateway_realtime_sessions_total`                                | Counter   | Completed realtime sessions by alias, provider, and status                                                        |
| `gateway_realtime_session_duration_seconds_sum`                  | Counter   | Aggregate realtime connected seconds                                                                              |
| `gateway_realtime_frames_total` / `gateway_realtime_bytes_total` | Counter   | Aggregate frame/byte volume by direction, never content                                                           |
| `gateway_request_logs_dropped_total`                             | Counter   | Lost request-usage metadata                                                                                       |
| `gateway_structured_logs_dropped_total`                          | Counter   | Structured stdout records lost after queue overflow or sink failure                                               |
| `gateway_ready`                                                  | Gauge     | Aggregate data-plane serving readiness                                                                            |
| `gateway_component_ready`                                        | Gauge     | Engine, model catalog, persistence, shared state, admin audit, logger, alerts, reconciliation, retention, signals |

HTTP route labels use registered route templates, including for failures raised before endpoint dispatch. Unknown paths are reported as `unmatched`; client-controlled path segments are never emitted as metric labels. Non-standard HTTP methods share the `OTHER` label, keeping method cardinality bounded.

`GET /ready` includes `serving_ready` and component flags. Catalog lag or activation failure makes `status` degraded and `model_catalog_ready` false but does not drain a replica that retains a safe runtime; `gateway_ready` remains 1. Alert on the model-catalog component so operators repair propagation independently of traffic routing.

`/ready` and `/metrics` share a one-second dependency-probe cache by default, and concurrent refreshes coalesce. Configure `observability.readiness_cache_ttl_seconds`; the value is also the maximum extra failure or recovery detection delay, and `0` disables caching.

## Calibrated signals and digests

With Postgres persistence, optional `signals` rules evaluate tenant/application/environment metadata for sustained 5xx rate, 429 rate, missing usage, and p95 latency. Minimum samples plus consecutive breach and recovery windows reduce noise. Latency combines an absolute threshold with previous-window regression when a baseline exists. Firing, reminder, resolved, and completed-week digest notifications use the same durable Slack or Google Chat destination as budget alerts.

Use `GET /admin/signals/issues` and `GET /admin/signals/digests` on the private operator origin. `POST /admin/signals/run` is an audited admin-only drill. A failed signals worker degrades `signals_ready` without interrupting inference. Signals use request metadata only; prompt, response, audio, and realtime-frame content are not inspected.

## Included Prometheus rules

`deploy/observability/gateway-alerts.yaml` contains alerts for target loss, aggregate/component readiness, sustained 5xx rate, high p95 inference latency, request-log drops, and structured-log drops. Structured stdout is drained through a bounded process-local background queue so collector backpressure cannot block inference; alert on `gateway_structured_logs_dropped_total` to detect queue overflow or sink failures.

Load the file into Prometheus and route `warning` and `critical` labels through Alertmanager. Thresholds are starting points; tune them after observing your providers and workload.

The optional `deploy/compose/observability.yaml` overlay supplies private
loopback Prometheus and Alertmanager services plus a shared Docker secret for
the notification relay and gateway budget outbox. Select `slack` or
`google_chat` with `LLM_GATEWAY_NOTIFICATION_PROVIDER`, keep its webhook in a
mode-`0600` host file referenced by
`LLM_GATEWAY_NOTIFICATION_WEBHOOK_FILE`, and enable the matching
`alerts.*_webhook_url_file` field in the environment gateway YAML. Run
`scripts/stage_observability_smoke.py`, then confirm both firing and resolved
synthetic messages at the receiver. Separately exhaust a disposable budget and
require its durable alert row to become delivered.

For local Compose, keep the host secret directory root-only and assign the
mode-`0600` webhook file to numeric UID/GID `10001:10001`. Compose bind-mounts
the file without remapping ownership; a root-owned file is unreadable to the
non-root gateway and relay and startup fails closed.

## Recommended dashboards

* Request rate and 2xx/4xx/429/5xx split.
* p50/p95/p99 by inference route.
* Readiness by component.
* Request-log drop increase.
* Structured-log drop increase.
* Usage, total tokens, estimated cost, and error rate by application/environment/model/provider.
* Budget utilization, stale reservations, and alert delivery failures.
* Soft-quarantined budget reservations and repeated reconciliation error types from structured logs.

Metrics reset when a process restarts. Prometheus remains the time-series authority. Usage and budgets come from Postgres APIs.