Skip to main content
Applications use stable aliases. Operators map those aliases to concrete provider deployments. There are two administration layers:
  • Deployment configuration owns providers, credential environment variables, and the baseline model catalog.
  • The private console and Admin API can add, update, delete, and alias concrete models that use an already configured provider. These changes are versioned in Postgres and survive restart.
The repository audit and smoke scripts help operators select and validate routes, but they are not another configuration layer. scripts/audit_model_catalog.py does not add models or providers; UI mutations call the same Admin API used by automation.

​
Discover available aliases

/v1/model/info reports capabilities, safe routing shape, metadata, reviewed pricing, and client-safe audio formats, voice aliases, defaults, and input limits for models allowed to the key. It does not expose provider credentials or provider-native voice/format identifiers.

​
Typical aliases

These are deployment conventions, not globally guaranteed model mappings. Always discover the aliases enabled in your environment.

​
Self-managed provider configuration

Providers reference credentials through environment variables:
Concrete deployments define capabilities and reviewed prices. Keep them internal:
Replace the example prices with current reviewed provider prices before issuing budgeted keys. Every weighted or fallback route reachable by a budgeted alias must have pricing.

​
Speech models and provider mappings

Transcription and English-translation models require the corresponding capability, public input formats, byte/duration limits, and reviewed input_audio_second pricing. Translation currently uses the native OpenAI-compatible adapter; other provider types are rejected during catalog validation. OpenAI transcription routes use the pinned LiteLLM adapter, while Azure Speech and ElevenLabs use reviewed native adapters behind the same OpenAI-shaped gateway endpoint. Speech synthesis models require text_to_speech: true, public voice and output-format aliases, a common default voice, and reviewed input_character pricing. OpenAI built-in names can use identity mapping. Provider-specific or custom voice IDs should always use an explicit internal mapping:
The one ELEVENLABS_API_KEY secret can serve both models. The transcription adapter streams the bounded upload to ElevenLabs Scribe v2 without retaining raw audio. It accepts language, temperature, json, verbose_json, and word timestamps; prompt and segment timestamps are rejected before provider spend. Use limits and pricing reviewed for the selected account rather than copying the placeholders. voices and output_formats are the only names accepted from applications. voice_ids and output_format_ids are selected after routing and are never returned by the client API. Every target in a weighted/fallback TTS route must expose a common public voice and format. ElevenLabs and Azure Speech models fail catalog validation unless every public alias has an internal mapping.

​
Realtime profiles

Realtime aliases preserve one explicit provider-native event protocol. Configure the provider secret in deployment YAML, then add the concrete model and alias through YAML, the Admin API, or Models → Add model / Add alias.

​
OpenAI Realtime

Replace the model identifier and price with values validated for the provider account. Every weighted/fallback target must use the same openai-realtime-v1 protocol; ElevenLabs and Deepgram native events cannot be placed behind this alias. The console exposes the capability, protocol, and typed price. It does not accept provider secrets. Production uses Redis-backed expiring per-key admission leases. Budgeted sessions reserve a bounded duration slice and renew ahead of consumption instead of holding the configured maximum; key revocation, admission loss, budget exhaustion, or persistence loss closes the active session fail-closed.

​
ElevenLabs realtime STT

The ElevenLabs profile relays Scribe v2 Realtime audio-to-transcript events. It is separate from bounded Scribe v2, realtime TTS stream-input, and ElevenAgents conversations:
Allowed input formats are pcm_8000, pcm_16000, pcm_22050, pcm_24000, pcm_44100, pcm_48000, and ulaw_8000; configure only formats validated for the application. The client may select a configured format, language, timestamps, language detection, manual/VAD commit, and no-verbatim mode. Provider tokens, logging/retention controls, keyterms, and VAD thresholds remain deployment/provider-policy concerns rather than client-controlled query fields. The gateway does not retain frames, but provider-side handling follows the ElevenLabs account agreement and settings.

​
ElevenLabs realtime TTS

Realtime TTS streams partial text in and base64 audio events out. Keep its alias separate from bounded /v1/audio/speech, Scribe realtime STT, and ElevenAgents:
Every target must provide complete private mappings for the same public voice and output-format aliases. Realtime TTS reserves the configured maximum text characters at admission, rejects text beyond that cumulative bound, and settles to the accepted character count. Clients cannot override upstream model, provider voice ID, voice settings, provider logging, or credentials.

​
ElevenLabs Agents

Conversational agents use their own signed-URL protocol. The private deployment model is the provider agent ID; the public alias exposes neither that ID nor the short-lived signed token:
The gateway calls ElevenLabs’ signed-URL endpoint with the provider credential, validates that the result remains on the configured provider origin and is bound to the configured agent, then opens the WebSocket without exposing either credential. Client query parameters cannot select an agent, branch, environment, or token. Agent prompt, LLM, voice, dynamic-variable defaults, retention, and provider logging stay deployment/account owned. Use separate aliases and keys for agents with different tenant or data-handling boundaries.

​
Azure Speech STT and TTS

Azure Speech is a native region-scoped adapter. Add the provider to deployment YAML first; the Admin API and console never accept the subscription key or region. One environment-backed Speech resource key can serve both STT and TTS models in that region:
Replace the region, voice, formats, limits, and prices with values reviewed for your resource. The current Azure short-STT adapter deliberately accepts only valid 16 kHz, 16-bit, mono PCM WAV input and requires a model limit of at most 60 seconds. It supports json and verbose_json, defaults the language to en-US, and rejects prompt, temperature, and timestamp parameters before provider spend. Azure TTS supports configured public voice/format mappings and speed from 0.5 through 2.0. The Admin API and Models → Add model console form accept these fields. In the console, public formats and voices belong under Audio contract; provider mappings remain visible only on the private operator surface. Typed pricing fields cover input/output audio seconds, input characters, and realtime session seconds. Provider credentials remain YAML/secret-manager owned in all cases. Self-managed deployments may define a fixed headers map on a provider for upstream tenant or routing headers. Those values are applied by both gateway engines and are never accepted from application requests or returned by the public model APIs. Virtual aliases support weighted targets and configured fallbacks. Run real chat, streaming, tools, vision, Responses, embeddings, transcription, translation, or speech checks for each declared capability before making it available to applications.

​
Add a model and alias in the console

  1. Open /admin/ui, authenticate, and select Models.
  2. Add a concrete model using a configured provider and its LiteLLM model identifier.
  3. Declare only capabilities that have been tested. For audio, enter the public formats/voices, private provider mappings, required limits, and the matching typed price meter. Mark its validation state.
  4. Add any unique alias name and choose one or more concrete targets with positive weights.
  5. Create an API key and select that public alias.
  6. Call /v1/models and /v1/model/info, verify the alias’s client-safe audio contract, then send an inference request using the alias.
Names may contain letters, digits, ., _, :, and -, up to 128 characters. Concrete models are normally internal; aliases are normally public. Disabled or internal entries cannot be granted to application keys.

​
Add a model and alias with the API

First read the active revision:
Create a priced concrete model using that revision:
Use the returned revision to create any application-facing alias:
Create the application key with allowed_models: ["invoice-agent-model"]. A key scope must match an active public model or alias; a misspelling is rejected instead of creating a key that can never work.

​
Authority and replica behavior

After the first API mutation, the Postgres catalog is authoritative over the YAML models and virtual_models sections. /admin/reload and process restart reapply that database catalog. /admin/model-catalog/reset explicitly returns to the current deployment YAML baseline. If the stored graph cannot produce a ready provider engine, GET /admin/model-catalog reports source: database_pending and preserves a compatible graph for inspection. The process retains its last safe runtime, /ready returns 200 with status: degraded, serving_ready: true, and model_catalog_ready: false, and Prometheus reports the model-catalog component as down. Repair or reset the pending revision while the prior runtime continues serving. The response includes schema_version and supported_schema_version. When a stored catalog was written by a newer incompatible gateway, older replicas report model_catalog_schema_unsupported, keep serving their safe runtime, and reject catalog edits or reload so they cannot overwrite fields they do not understand. Upgrade the replica or use revision-based reset when discarding the stored override is intentional. A catalog mutation or reset advances a Postgres generation. Every replica polls, readiness-checks, and atomically activates that generation, then publishes its acknowledgement. GET /admin/model-catalog includes generation, local generation, poll/stale timing, and every known replica’s status. Configure a unique stable LLM_GATEWAY_INSTANCE_ID on each process and monitor component health. A lagging or failed process reports degradation but continues serving its last known-good runtime.