Designing an LLM Gateway like LiteLLM or Portkey

When a company has three teams calling one model provider, nobody needs an LLM gateway. When it has three hundred teams calling five providers plus a few self-hosted open-weight models, things change fast. Every team pastes its own provider keys into its own secrets store and writes its own retry loop, usually a bad one. Nobody can say how much the company spent on LLMs last week, or who spent it. When a provider has a bad hour, half the company's AI features break at once. And when security asks whether confidential data is going to a provider that is not approved for it, the honest answer is "we do not know."

An LLM gateway (also called an AI gateway; LiteLLM and Portkey are well-known examples) fixes this. It sits between every internal caller and every model backend. Callers get one API. The company gets one control point: routing, failover, caching, keys, budgets, logging and policy live in one place instead of three hundred.

In this post I walk through how I would design one for a large company. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives:

• A unified API and provider routing
• Fallbacks, retries and circuit breaking
• Semantic caching
• Cost metering, budgets and per-tenant keys

Scoping the Problem and Clarifying Assumptions

I am building the layer in front of model providers, not the providers. How a GPU fleet batches requests and manages its KV cache is a different system, covered in Designing an LLM Inference Platform. To the gateway, a self-hosted model pool is just one more provider with an endpoint, a capacity limit and a failure rate.

What I'm including:

• One API for chat completions, tool calling and embeddings, with streaming
• Several hosted providers in multiple regions, plus internal self-hosted models
• Routing by cost, latency, capability and data residency, including canaries
• Retries, fallbacks and circuit breaking across providers
• Exact and semantic response caching
• Virtual keys, token-based rate limits, budgets, chargeback and audit logs

What I'm NOT building here:

• Inference server internals (linked above)
• Agent orchestration, RAG and tool execution. These are callers: they decide what to send, the gateway decides where it goes.
• Safety classifier internals. I show where the policy hook sits.

Key assumptions:

• About 500 internal apps and agents owned by about 150 teams
• Tens of thousands of requests per second at peak; I plan for 30,000
• Five hosted providers and three self-hosted model families, across three regions
• Every app is already tagged with the highest data class it sends (public, internal, confidential, restricted), and some providers are approved only for some classes and regions

How this differs from a classic API gateway: a normal gateway fronts services the company owns, with similar cost per request and failure modes it controls. Here the backends are mostly external, expensive and inconsistent. Two requests to the same endpoint can differ in cost a thousandfold. Providers fail with rate limits more often than crashes. Each one has its own schema for messages, tools and streaming. And the most important routing input is often what data is in the request, not where it is going. (For general gateway patterns, see Load Balancers, Reverse Proxies, API Gateways, and Service Meshes.)

Functional and Non-Functional Requirements

Functional Requirements

1. Unified API: one canonical request and response format, whatever the backend, including tools and streaming.
2. Route: map a logical model alias to concrete provider deployments, chosen by rules.
3. Fail over: retry and fall back on retryable errors, within a deadline and retry budget.
4. Cache: serve exact and, where enabled, semantically similar repeat requests.
5. Keys: issue virtual keys to apps; never expose provider keys.
6. Limit and budget: token-based rate limits and spend budgets per key, team and org.
7. Meter, log and apply policy: record tokens, cost, route and outcome for every request; run PII redaction and other hooks per app.

Non-Functional Requirements

Back-of-the-Envelope Capacity Estimation

These are planning assumptions, not measurements.

Traffic: 30,000 requests per second at peak, about 10,000 on average, so 10,000 × 86,400 ≈ 860 million requests per day.

Request shape: agents and RAG push prompts up, so I assume 2,000 input and 300 output tokens on average. At peak that is 60 million input and 9 million output tokens per second flowing through the gateway.

Concurrency. Most calls stream. If a request stays open 8 seconds on average:

concurrent streams at peak = 30,000 req/s x 8 s = 240,000 open streams

The gateway's CPU work per request is small (parse, translate, a few Redis calls, forward bytes). What it holds is connections. If a benchmark shows one pod handles 5,000 concurrent streams and 1,500 new requests per second, concurrency binds: 240,000 / 5,000 ≈ 48 pods. I would run about 100 across three regions so that losing a region still leaves headroom.

Usage events: one per request, about 1 KB, so roughly 860 GB per day. Easy for Kafka and a warehouse.

Payload logs are the expensive part. 2,300 tokens is about 9 KB of text, so logging everything is roughly 7.7 TB per day, much of it confidential. Payload logging is therefore opt-in per app, sampled, redacted and short-lived.

Semantic cache: if 20% of traffic enables it, that is 6,000 embedding calls per second at peak, a real cost the hit rate must pay for. At about 3 KB per 768-dimension float32 vector, 100 million entries is about 300 GB before index overhead, sharded by tenant.

High-Level Architecture and Core Components

Component Walkthrough

Gateway edge. A stateless, non-blocking HTTP service (Go, Rust, or an Envoy filter chain) that terminates TLS, parses the canonical request and owns the stream back to the caller. It must hold hundreds of thousands of open streams cheaply.

Key service. Resolves a virtual key to its app, team, allowed aliases, data class, limits and budget. Each pod holds this in memory with push invalidation, so the hot path never waits on a database.

Policy hooks. Pre-request hooks (PII redaction, blocked topics, prompt-injection heuristics for agents) and post-response checks, each with a timeout and a declared failure mode. Restricted-data apps fail closed.

Cache. An exact cache in Redis and an optional semantic cache (third deep-dive).

Router and fallback engine. Resolves the alias, filters and scores targets, and runs attempts with retries and circuit breakers (first two deep-dives).

Provider adapters. One per provider API, translating requests, streams, errors and usage to and from canonical form.

Usage pipeline. Each finished or cancelled request emits a usage event to Kafka, which feeds spend counters, the warehouse, alerts and the audit archive.

Config registry. Routes, aliases, keys, budgets and policies, versioned and reviewed like code, pushed to pods as snapshots.

Request Flow

1. An agent sends POST /v1/chat/completions with model: "chat-default", its virtual key and stream: true.
2. The gateway resolves the key: the app is confidential, may use chat-default, and has used 70% of its monthly budget.
3. The limiter reserves estimated tokens against the key's per-minute buckets, and policy redacts two email addresses.
4. The exact cache misses. Semantic caching is off for this route.
5. The router expands chat-default to three targets, drops one in a region not approved for confidential data, and picks the healthiest of the rest.
6. The adapter calls the provider with the real key and streams translated chunks back as they arrive.
7. At the end, the gateway reads reported usage, computes cost, reconciles the reservation and emits a usage event.

Scaling It Out

• Stateless pods, regional state. Rate-limit counters and the exact cache live in a Redis cluster per region. Spend totals are aggregated from the event stream.
• Regional affinity. Apps call the gateway in their own region, and it prefers providers in that region, which keeps both latency and data residency right.
• Config as snapshots. Each pod holds a full, versioned snapshot of routes, keys and budgets. If the registry is down, pods keep serving on the last good snapshot. The data plane must not depend on the control plane.
• Separate pools for bulk work. Large batch jobs get their own gateway pool, so a nightly spike cannot take connection slots from interactive features.
• Degrade, do not refuse. If Redis is slow, the limiter falls back to per-pod local limits sized to a share of the global limit.

Deep-Dive: A Unified API and Provider Routing

One Canonical Schema

I would make the canonical API OpenAI-compatible, not because it is the best design, but because it is the de facto standard: most SDKs and agent frameworks already speak it, so adopting the gateway is a base-URL change. I extend it with an x-gateway metadata block (trace id, cache control, tags) rather than inventing a new schema.

The hard part is everything around "messages in, text out":

• Message structure. Some providers take the system prompt as a top-level field, others as a message. Some require strictly alternating turns, or represent content as typed blocks rather than strings.
• Tool calling. Everyone uses a name, description and JSON Schema, but tool-call output, tool-result messages and parallel-call ids are shaped differently. The adapter keeps a stable canonical tool-call id, so an agent loop does not break if turn 3 is served by a different provider than turn 2.
• Streaming. All stream over SSE, but some send flat delta chunks ending in a [DONE] sentinel, while others send typed start, delta and stop events. Tool-call arguments arrive as partial JSON fragments. Each adapter runs a small state machine per stream, and handles usage that arrives only on the final event.
• Parameters. Context windows, output caps, structured-output support and accepted parameters differ per model. I keep a capability record per deployment, and the adapter rejects parameters a target cannot honour, with a per-app opt-in to drop them instead. Silently dropping a parameter a caller relies on is a bug.

Other quirks: errors inside an HTTP 200 body or a final stream event, refusals that look like normal completions with a special finish reason, and rate-limit headers that differ in meaning. Adapters are tested against recorded golden responses, and a synthetic canary calls every provider every minute.

Aliases, Not Model Names

Apps call aliases such as chat-default, chat-fast or embed-v3, never a dated provider model id. The platform team owns the mapping, so migrating everyone to a new model is a config change and a canary, not 300 pull requests. An app that needs a pinned version gets an alias with a single target.

alias: chat-default
capabilities_required: [tools, streaming]
targets:
  - id: provA-eu-large
    provider: provider_a
    region: eu-west
    weight: 90
    data_classes: [public, internal, confidential]
  - id: provB-eu-large
    provider: provider_b
    region: eu-west
    weight: 10              # canary
    data_classes: [public, internal]
  - id: selfhosted-large
    provider: internal
    region: eu-west
    weight: 0               # reached only by rules or fallback
    data_classes: [public, internal, confidential, restricted]
fallback_chain: [provA-eu-large, provB-eu-large, selfhosted-large]
timeouts: {connect_ms: 1000, first_token_ms: 8000, total_ms: 120000}

Routing: Filter, Then Score

Hard filters first, because they must never be traded off:

• Data residency and class. The app's data class must be allowed for the target's provider and region. Restricted data stays on self-hosted models. Enforcing this in the router means one implementation, auditable, instead of 500.
• Capability. Tools, images, structured output, and a context window that fits the estimated token count.
• Circuit state. Targets with an open circuit are removed.

Then scoring, by the alias's policy:

• Weighted (default), used for canaries and gradual migrations.
• Cost-first for batch and offline jobs.
• Latency-first for interactive features, on a rolling percentile of time to first token rather than the last sample.
• Quota-aware. Each provider account has a tokens-per-minute limit. The router tracks how much it has used this minute and spreads load before hitting it, instead of after the 429s start.

Weighted routing needs stickiness: I hash the conversation or session id so a conversation stays on one target. Switching provider mid-conversation changes tone and tool-call habits, and it throws away the provider's prompt cache.

What I would build first: the canonical schema, adapters for the two most-used providers, aliases, weighted routing and residency filters. Latency-first and quota-aware scoring come once there is enough health data to trust.

Deep-Dive: Fallbacks, Retries, and Circuit Breaking

A single provider fails more often than a product can tolerate, which is the main reason teams adopt a gateway. But naive retries turn a partial outage into a full one, and naive fallbacks quietly double spend.

Classify Before You Retry

Deadlines and Retry Budgets

Every request carries a deadline, and every retry must fit inside it. I cap retries twice: at most three attempts per request, and, per target, retries may add at most about 10% to its load. The second cap matters most. When a provider is struggling, a retry budget stops the gateway from multiplying its traffic exactly when it can least cope.

def execute(request, chain, deadline):
    attempts = 0
    for target in chain:
        if not breaker.allows(target) or not capability_ok(target, request):
            continue
        if attempts > 0 and not retry_budget.try_spend(target):
            continue                     # never amplify load on a struggling target
        if time_left(deadline) < target.min_useful_ms:
            break
        attempts += 1
        try:
            return call(target, request, timeout=time_left(deadline))
        except ProviderError as e:
            kind = classify(e)
            health.record(target, kind)
            if kind in ("invalid_request", "auth", "content_filter"):
                raise                    # not retryable
            if attempts >= MAX_ATTEMPTS:
                break
    raise AllTargetsFailed(request.id)

Fallback Order

I order fallbacks by how much they change the answer:

1. Same model, different deployment (another region or account). Identical quality.
2. Equivalent model at another provider. Similar, not identical; prompts tuned for one model may do worse on another.
3. Smaller or self-hosted model. Only for aliases whose owners say a degraded answer beats none.

Every response carries an x-gateway-served-by header, and the usage event records the full attempt path, so teams can see when they are running on a fallback.

Mid-Stream Failures

If the provider dies after 200 tokens have reached the user, I cannot transparently retry: a fresh attempt would produce a different answer. My default is to end the stream with a clean canonical error event, so the client can say "the answer was interrupted" and offer a retry. Continuing on another target by sending the partial answer as an assistant prefix is possible with some providers, but the seam can show, so it is opt-in. Silently restarting and streaming a second, different answer is never acceptable.

Circuit Breakers and Health Scores

Each target (provider, region, account, model) has a rolling health score from error rate, timeouts and time to first token against its normal range. The breaker is:

• Closed: traffic flows.
• Open: for example, over 20% retryable errors across at least 50 requests in 30 seconds. The router skips the target, so callers go straight to fallbacks without paying a timeout first.
• Half-open: after a cooldown, a trickle of probes. Success closes it; failure reopens it with a longer cooldown.

Each pod computes health from its own traffic, which reacts in seconds, and shares it per region through Redis so low-traffic pods benefit too. I keep 429s out of the breaker's error rate: "this account is full" is a routing signal for the quota tracker, not evidence that the provider is broken.

Hedged Requests

For latency-critical aliases I can hedge: if the first token has not arrived by the target's p95, send the same request to a second target, keep whichever streams first, and cancel the other (the idea from "The Tail at Scale"). For LLMs this is costly: the loser has usually already been billed for its input tokens, so each hedge roughly doubles input cost for that request. Hedging is off by default, enabled per alias, capped as a share of traffic, and charged to the team that turned it on.

What I would build first: error classification, deadlines, ordered fallback chains and per-pod breakers. Shared health, per-target retry budgets and hedging come next; mid-stream resume last, if ever.

Deep-Dive: Semantic Caching

Two Caches, Very Different Risk

The exact cache hashes the normalised request and returns a stored answer on a match. It is cheap and never answers a different question. Evaluation suites, CI pipelines and retried batch jobs send identical requests often.

The semantic cache embeds the prompt, finds the nearest earlier prompt and returns its answer if it is close enough. It catches "how do I reset my password" versus "how can I reset my password". It can also return the answer to a different question: "cancel my order" and "cancel my subscription" sit close in embedding space and need different answers.

So the exact cache is on by default for deterministic requests, and the semantic cache is opt-in per route.

What Goes Into the Key

A hit must only happen when the stored answer is valid for this caller and this request:

def cache_key(req, tenant):
    canonical = {
        "tenant": tenant.cache_namespace,             # never shared by default
        "resolved_model": req.resolved_target_model,  # a model upgrade invalidates
        "system": req.system_prompt,
        "tools": req.tools,
        "params": {k: req.params.get(k) for k in
                   ("temperature", "top_p", "max_tokens", "response_format", "seed")},
        "messages": normalise(req.messages),          # whitespace, stable ordering
    }
    return sha256(json.dumps(canonical, sort_keys=True).encode()).hexdigest()

Three details matter. A request with temperature above zero asked for variety, so I cache it only with an explicit x-gateway-cache: allow. The resolved model is in the key so that moving an alias to a new model stops serving the old model's answers. And I never semantically cache responses that contain tool calls: replaying one makes an agent execute a tool again on stale reasoning.

Semantic Lookup, Thresholds and Guards

I embed only what varies, normally the last user message plus a short summary of recent turns. Everything else (tenant, resolved model, system prompt hash, tool schema hash, key parameters) is an exact metadata filter. Only the question itself may be "close". The vector index (HNSW) is partitioned by tenant namespace, so a search cannot cross tenants by construction, rather than by a filter someone might forget.

There is no universal similarity threshold. I set it per route from labelled data: sample real query pairs across similarity scores, label whether the cached answer is right for the new question, and pick the threshold that keeps false hits below what the route tolerates. A support FAQ bot can accept a few. A finance assistant should not use semantic caching at all. Two guards help: reject matches whose numbers, dates, ids or product names differ ("refund order 1234" versus "refund order 5678"), and reject matches whose conversation lengths differ a lot, because context changes what "and for Canada?" means. I log every semantic hit with its score; users retrying right after a hit suggests the threshold is too loose.

TTLs and Privacy

• TTLs per route: days for stable reference answers, minutes or no caching for anything about prices, inventory or live data. For RAG routes, a hash of the retrieved document ids goes into the metadata filter, so new content misses the cache.
• Invalidation mostly happens for free: changing the model, system prompt or tools changes the key. Owners can also purge their namespace.
• Privacy: cached answers inherit the request's data class. Confidential routes get encrypted entries and short TTLs; restricted routes are not cached. Entries carry user and app ids so deletion requests reach the cache too. Cross-tenant sharing (say, one cache for a public FAQ bot) requires explicit opt-in from the owner and the data class.

Gateway Caching vs Provider Prompt Caching

These are often confused. Provider prompt caching reuses the provider's computed state for a repeated prefix (system prompt, tool list, document). The model still generates a fresh answer, so it is never wrong; it cuts input cost and time to first token. Some providers apply it automatically, others need explicit cache markers. The gateway's job is to make it work: keep stable content at the start, insert markers where needed, and keep a conversation on the same provider and region, since those caches are local. Gateway response caching skips the model entirely, saving the full cost, but only for repeated questions, and the semantic version can be wrong.

Almost every agent and RAG request has a long stable prefix, while far fewer repeat a question. So I expect prefix-friendly routing to save more company-wide than semantic caching. I would build that and the exact cache first, then add semantic caching to specific routes (support bots, internal FAQ assistants) where measured repeat rates pay for the embedding calls.

Deep-Dive: Cost Metering, Budgets, and Per-Tenant Keys

Virtual Keys and Provider Keys

Apps never see a provider key. Each app gets virtual keys from the gateway, each carrying its app, team, cost center, allowed aliases, maximum data class, rate limits, budget, logging policy and environment (so a dev key cannot spend production budget). The gateway stores only a hash of each key, like a password store, and resolves it from the in-memory snapshot. Revocation is a config push that lands in seconds.

Provider keys live in a secrets vault. The company may hold several accounts per provider, for higher limits or separate regions, and the router picks an account as part of picking a target. Pods load provider keys into memory at startup and on rotation, and never log them.

Rotation without downtime: add the new provider key, let pods pick it up, watch traffic move, then revoke the old one. For virtual keys, an app can hold two active keys while it redeploys, and keys can carry an expiry that warns the owners beforehand.

Counting Tokens

Each provider tokenises differently. I keep two counts: an estimate at admission (a local tokenizer for the model family where available, otherwise a calibrated characters-per-token ratio), used for rate limits and context-length routing; and the provider-reported usage, which is the billing truth, split into input, cached input, output and reasoning tokens because each is priced differently. If a stream is cancelled before usage is reported, I record the estimate, flag it, and reconcile later against the provider's usage exports. Cost comes from a versioned price table per provider, model, region and token type, so historical spend uses the price that applied at the time.

The Usage Event

{
  "event_id": "evt_01J...",
  "virtual_key_id": "vk_7f3a",
  "app": "support-copilot",
  "cost_center": "CC-4471",
  "alias": "chat-default",
  "served_by": "provA-eu-large",
  "attempts": [{"target": "provA-eu-large", "outcome": "ok"}],
  "cache": "miss",
  "usage": {"input": 2140, "cached_input": 1792, "output": 318, "reasoning": 0},
  "usage_source": "provider",
  "cost_usd": 0.00421,
  "price_version": "2026-10-01",
  "ttft_ms": 612,
  "status": "ok"
}

Events go to Kafka keyed by virtual key id. Consumers are idempotent on event_id, because a pod may resend after a restart. A spend aggregator maintains running totals per key, team and org in a budget ledger and pushes current spend back into the config snapshot. A warehouse loader feeds chargeback and dashboards. An audit archive stores events, and payloads where policy allows, in append-only storage.

Budgets and Token Rate Limits

Budgets are about money over weeks; rate limits are about load over seconds. I keep them separate.

• Soft limit (say 80%): notify the team and show it on their dashboard. Traffic is unaffected.
• Hard limit: depends on the key. Dev and experimental keys are blocked with a clear error naming the budget. For customer-facing production keys, breaking a feature over an internal budget line is usually worse, so the owners choose in advance: downgrade to a cheaper alias, allow an overrun with an alert to the finance owner, or block.

Aggregated spend lags by seconds, which is fine for monthly budgets but too slow for an agent stuck in a loop. So each key also has a fast spend bucket in Redis (dollars per hour) enforced on the hot path, which catches runaways long before the monthly limit.

Rate limits are in tokens, not requests. Each key has buckets for requests, input tokens and output tokens per minute, plus concurrency. The gateway reserves estimated input plus max_tokens at admission and reconciles to actual usage at the end; the bucket mechanics are the same as in Designing a Distributed Rate Limiter. There is a second layer too: per provider account. Every team shares the company's provider quotas, so without it one team's batch job could exhaust an account and cause 429s for every interactive feature. When an account nears its limit, interactive traffic gets priority.

Chargeback and Audit

Chargeback is a query over the warehouse:

SELECT cost_center, team,
       SUM(cost_usd)                                           AS spend_usd,
       SUM(CASE WHEN cache = 'hit' THEN est_saved_usd ELSE 0 END) AS cache_saved_usd,
       AVG(CASE WHEN served_by <> primary_target THEN 1.0 ELSE 0 END) AS fallback_rate
FROM llm_usage_events
WHERE ts >= DATE '2026-10-01' AND ts < DATE '2026-11-01'
GROUP BY cost_center, team
ORDER BY spend_usd DESC;

Two practices make it trusted. I reconcile against provider invoices monthly; gaps usually mean a price-table error or estimated usage from cancelled streams. And I attribute shared costs explicitly: embedding calls for semantic caching and hedging waste go to the teams that enabled them, and the gateway's own infrastructure is a visible platform line.

The audit record answers "who sent what class of data to which provider and region, when, and under which policy", including which hooks ran and what they did ("redacted 2 emails"). Admin actions (new keys, route changes, budget raises) are audited with their approver. Security can read these logs directly.

What I would build first: hashed virtual keys, provider keys in the vault, per-key token limits, the usage pipeline and a spend dashboard. Budgets, fast spend buckets and invoice reconciliation follow once the event data is trusted.

Conclusion: Bringing It All Together

An LLM gateway looks like a reverse proxy, and in code it mostly is one. What makes it hard is that its backends are external, priced per token, inconsistent and unreliable in ways the company does not control. The gateway is where the company imposes order on all of that.

If I had to summarise the design in four decisions:

1. Route on aliases, apply residency and capability filters first, then score by cost, latency and health. Data policy is enforced once, and model migrations become config changes.
2. Classify every error before acting, and spend retries from a budget. Fall back to the same model elsewhere before a different model, and never silently restart a stream.
3. Cache exactly by default and semantically only by choice, with tenant, resolved model, system prompt and parameters in every key, and route so that provider prompt caching actually hits.
4. Never let a provider key leave the gateway, and meter from provider-reported usage. Reserve, reconcile, aggregate from events, and check the total against the invoice.

Related reading: Designing an LLM Inference Platform, Designing a Distributed Rate Limiter, Load Balancers, Reverse Proxies, API Gateways, and Service Meshes, and Designing a Distributed Cache.