LLM Gateway

One OpenAI-compatible endpoint for Amazon Bedrock models and the models you host yourself in the cluster, running in your own AWS account: per-team keys and budgets, an input guardrail, a log of every call in your S3 bucket, and an optional usage dashboard. Built on the open-source LiteLLM proxy.

What gets deployed

The LLM Gateway stack deploys up to three SeaGit applications into one cluster of an environment, managed together from the Stack tab:

litellm: the gateway. It serves /v1/chat/completions, /v1/models and the LiteLLM Admin UI over HTTPS on your environment’s domain. It calls Bedrock with its own IAM role, which may invoke only the models you chose. No AWS keys are stored.
db (only with “Create new Postgres”): Postgres 17 on an encrypted EBS volume, kept off spot nodes, with a nightly backup to the stack’s S3 bucket. LiteLLM keeps keys, teams, budgets and spend here.
llm-dashboard (optional): the usage dashboard, which reads the gateway’s logs with read-only access.

Each component appears in the Stack tab with an open link (↗) once it is running.

Requirements

A paid plan.
A started cluster on the native (sgt-k8s) executor, in an environment with a domain. The gateway and dashboard are exposed on that domain.
The EBS CSI add-on on the cluster, when you create a new database.
Model access enabled in the Amazon Bedrock console for the models you choose. Amazon Nova models need no request; Anthropic, Meta and Mistral models do.

Deploy

Open Templates, choose LLM Gateway, pick the environment (and the cluster, if it has more than one), and answer:

Question	What it does
Models	The Amazon Bedrock chat models the gateway serves. Only models your organization approves are listed.
Default model	Also served under the model name "default", so applications can switch models without a code change.
Self-hosted models (optional)	OpenAI-compatible model servers in the same cluster (llama.cpp, vLLM, Ollama), served next to the Bedrock models. One per line: an alias, the server’s base URL, and its own model name if it needs one. See Self-hosted models below.
API key for the self-hosted models (optional)	Sent as a bearer token to every self-hosted server, for one started with an API key. Stored encrypted; never shown again.
Database	"Create new Postgres" adds a Postgres database to the stack, backed up nightly to the stack’s S3 bucket. "Use existing Postgres" takes a postgresql:// URL that must be reachable from the cluster; it is stored encrypted and never shown again.
Database storage / Keep backups for	Only for a new database: the volume size (GiB) and how many days of nightly backups to keep.
Include usage dashboard	Adds the usage dashboard (cost, usage, blocked requests, traces). On by default.

A deploy usually takes 1–3 minutes. When it finishes, the Stack tab shows:

Output	Meaning
Gateway URL	The OpenAI-compatible endpoint, e.g. https://<name>.<your domain>.
Admin UI	<Gateway URL>/ui. Sign in as admin with the master key.
Master key	Full admin access to the gateway. Hidden until you reveal it; every reveal is audited.
Dashboard / Dashboard key	The usage dashboard and its sign-in key.
Database host	Only for a new database: its in-cluster address.
Call logs / Database backups	Where the per-call log and the database backups are written in your S3 bucket.

Call the gateway

Any OpenAI-compatible client works. Point it at the Gateway URL and use a key. GET /v1/models lists the model names: each chosen model under a short name (for example amazon-nova-micro), plus default.

curl -s https://<gateway-url>/v1/chat/completions \
  -H "Authorization: Bearer $LLM_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Summarise our travel policy"}],
    "metadata": {"team": "support", "feature": "faq", "trace_id": "run-42"}
  }'

The optional metadata fields (team, feature, env, session, trace_id) end up in the call log and the dashboard. Use the same trace_id for every call of one agent run to see it as a timeline.

Self-hosted models

Serve a model you run yourself (a fine-tuned GGUF on llama.cpp, a vLLM or Ollama server) through the same endpoint, keys, guardrail, call log and tracing as the Bedrock models. Deploy the model server as a SeaGit application in the same cluster, then list it under Self-hosted models, one per line:

seagit-forms http://forms-application.apps.svc.cluster.local:8081/v1
qwen         http://vllm-application.ml.svc.cluster.local:8000/v1  Qwen/Qwen2.5-7B-Instruct

Alias: the model name clients send (lowercase letters, digits, . _ -). It must be unique and must not be default or the short name of a chosen Bedrock model.
Base URL: the server’s OpenAI-compatible in-cluster address, usually ending in /v1: http://<deployment>-application.<space>.svc.cluster.local:<port>. Only in-cluster service addresses are accepted, so the traffic and the API key never leave the cluster.
Upstream model (optional): the name the server itself serves, which vLLM and Ollama need. Without it the alias is sent.

Self-hosted models appear in GET /v1/models and are called like any other model. The default model stays a Bedrock model, and Bedrock IAM permissions do not apply to self-hosted ones: any key allowed to use the alias can call it. Self-hosted calls have no price, so they show no cost and do not count against spend budgets; they still appear in the call log, the dashboard and the MLflow traces.

If an alias is used twice or clashes with a Bedrock model, the gateway does not start, and its log names the line, rather than silently dropping a model.

Keys, teams and budgets

Give each team or application its own virtual key rather than the master key. Create keys in the Admin UI (Virtual Keys) or through the API:

curl -s https://<gateway-url>/key/generate \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"key_alias": "team-search", "models": ["default"], "max_budget": 50}'

A key can call only the models listed for it. Any other model is refused with 401.
max_budget (USD) stops the key once its recorded spend reaches the budget. Spend is recorded per key and per team.
Requests without a key, or with an unknown key, are refused with 401.

Keys, teams and spend live in the stack’s database. They survive Stop/Start and redeploys.

Who can reach it

Anyone on the internet (with a key) — the default.
Allowed IPs only — one IPv4 address or range per line. Change the list later in the gateway app’s Routing settings. Apps on another cluster need that cluster’s outbound IPs here.
Inside the cluster only — no public address; only apps on the gateway’s cluster can call it.

Every call needs a key in all three modes.

Embedding models and RAG stacks

Embedding models serves Bedrock embedding models (Titan Text Embeddings v2, Cohere Embed v3) at /v1/embeddings. A self-hosted embedding server is a Self-hosted models line ending in the word embedding.

A RAG stack can send its questions — and optionally its embeddings — through this gateway. Each such stack gets its own key, limited to the models picked for it, registered from inside the gateway’s cluster and revoked when the RAG stack is deleted. The master key stays in the gateway’s namespace.

Guardrail

Every request is checked before it reaches the model:

Prompt attacks (for example “ignore all previous instructions”) are blocked with 400. The model is never called and nothing is charged.
Personal data (email addresses, US social security numbers, card numbers) is replaced with a placeholder such as {EMAIL} before the model sees it.

The guardrail matches patterns. It catches common cases, not every possible attack or format. Each finding is logged, so you can see what it caught in the dashboard.

Call log

Every call (answered, blocked or failed) is written as one JSON line to your S3 bucket under gateway-logs/dt=YYYY-MM-DD/, within about 10 seconds. Prompts and answers are never logged, only:

Field	Meaning
ts, request_id	When the call finished and its id.
model, upstream_model	The model name the client asked for and the model that answered (Bedrock or self-hosted).
status, stop_reason	success, blocked or error; and how the answer ended (stop, length, guardrail_intervened).
input_tokens, output_tokens, cost_usd	Usage and estimated cost.
latency_ms, guardrail_ms	End-to-end time and guardrail time.
key_alias	Which virtual key made the call.
team, feature, env, session, trace_id	Copied from the request’s metadata, when you send it.
guardrail_findings	What the guardrail blocked or masked.

The files are plain NDJSON, so Athena, DuckDB or any log tool can read them directly.

Usage dashboard

Usage: calls over time by outcome, and per model and per key (latency, truncated answers).
Cost: spend over time by model, and by key, team and model.
Blocked: guardrail findings over time, and the blocked requests.
Tracing: one trace_id as a timeline, and the latest calls.

Costs are estimates from public Bedrock prices. Your AWS bill is authoritative.

When the gateway sends traces to an MLflow stack, each one holds the model, the prompt, the response, token counts and timing. By default, Personal data in traces redacts emails, phone numbers, card numbers and AWS keys before a trace is stored, with the same rules as the RAG stack. Choose Store prompts and answers as they are to keep them whole. If a call cannot be redacted, it is not traced. The call log above never stores prompts or answers either way.

Database and backups

New database: backed up right after deploy and then nightly to backups/ in the stack’s bucket, kept for the number of days you chose. Deleting the stack takes a final backup first; if that backup fails, the delete stops and offers Delete anyway.
Existing database: SeaGit only connects to it. Its backups, upgrades and availability are yours. The gateway waits for the database at start-up and is not marked ready until it can reach it.

Stop, start and delete

Stop scales every component to zero. The database volume, keys and logs are kept. Start brings it back.
Delete removes the components, IAM roles and DNS records. Tick Also delete my backups to also delete the stack’s S3 bucket, including the call logs and database backups; otherwise the bucket and its contents are kept.

Current limits

Amazon Bedrock models, plus OpenAI-compatible servers you run in the same cluster. Other hosted model providers are not set up by the stack.
The Admin UI is protected by the master key. Single sign-on is not set up by the stack yet.
LiteLLM’s audit log of admin actions (who created a key, changed a budget) is a LiteLLM Enterprise feature and is not included. The call log records every model call.
One gateway replica.

Troubleshooting

A component stays pending: the cluster may be out of capacity. A new database needs a non-spot node with about 512 MiB free. See Troubleshooting.
The gateway never becomes ready: it cannot reach its database. For an existing database, check the URL and that the cluster can reach it on port 5432.
401 for one model only: that key isn’t allowed to use the model.
A model returns an access error: enable model access for it in the Amazon Bedrock console.