LLM Gateway and Proxy on Kubernetes

Overview

An LLM gateway is a single proxy that every application calls instead of talking to model providers directly. It standardizes authentication, routes requests to the right model, enforces rate limits and budgets, and applies safety guardrails. Use this architecture when several teams consume LLMs and you need central control over cost, security, and provider choice without rewriting each application.

Components

• Envoy ingress: terminates TLS and load-balances traffic into the gateway pods.
• LLM gateway service: a stateless service that exposes one OpenAI-compatible API and translates requests to OpenAI, Anthropic, Azure OpenAI, or self-hosted models.
• Redis: tracks rate-limit counters, token budgets, and a semantic cache of recent prompt/response pairs to cut cost.
• PostgreSQL: stores API keys, per-team policies, model routing rules, and usage records for billing.
• Keycloak: issues and validates OAuth tokens that map callers to teams and quotas.
• Prometheus + OpenTelemetry Collector: capture latency, token counts, error rates, and traces for every call.

Data Flow

A client sends a chat request with a bearer token. The gateway authenticates the caller, checks the team's budget and rate limit in Redis, and selects a model from the routing rules (for example, a cheap model for short prompts and a frontier model for complex ones). It applies input guardrails, forwards the request to the chosen provider, applies output guardrails, records token usage in PostgreSQL, and returns the response. A semantic cache short-circuits identical or near-identical prompts.

Scaling and Resilience

The gateway is stateless, so it scales horizontally with the Horizontal Pod Autoscaler based on request rate and CPU. Redis runs in a clustered, replicated configuration so counters survive node loss. Configure provider fallbacks: if the primary model returns errors or times out, retry on a secondary provider. Use circuit breakers per provider so one slow vendor does not exhaust the connection pool. Run across multiple node pools and zones.

Security

Never embed provider keys in client apps; the gateway holds them in a secrets manager and rotates them. Enforce least-privilege quotas per team. Scan prompts for injection and exfiltration attempts, and filter outputs for sensitive data leakage. Log every request and response for audit, with PII redaction. Apply network policies so only approved namespaces can reach the gateway.

Trade-offs and Alternatives

A self-hosted gateway gives full control and avoids per-seat SaaS fees, but you own the uptime and upgrades. Managed gateways (cloud-native AI gateways or SaaS proxies) reduce operational load at the cost of flexibility and data residency. A service mesh can provide some routing and observability, but it lacks token accounting and model-specific guardrails. Keep the gateway thin: heavy business logic belongs in the calling services, not the proxy.