Designing an LLM Inference Platform: Architecture, GPU Scheduling, and Token Streaming

When I design an LLM inference platform like the one behind ChatGPT, Claude or Gemini, the first thing I notice is that almost none of my usual web-scale instincts apply. A normal request costs a few milliseconds of CPU and a database read. An LLM request holds a slice of an accelerator worth tens of thousands of dollars for several seconds, its cost grows with every token it reads and writes, and the user watches the answer arrive one token at a time. If the first token is slow, the product feels broken. If the GPUs sit idle between requests, the business loses money.

So the design problem is really two problems pulling in opposite directions: latency for each user and utilisation of very expensive hardware. Most of the interesting decisions sit where those two meet.

In this post I walk through how I would build this platform from scratch. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives:

• GPU scheduling, continuous batching and the KV cache
• Token streaming
• Multi-tenant quotas
• Model routing with prompt caching

Scoping the Problem and Clarifying Assumptions

I am building the serving side: the system that takes a prompt and returns generated tokens. Training and fine-tuning are separate platforms with very different shapes (long batch jobs, checkpoints, data pipelines) and are out of scope here.

What I'm including:

• A chat product (web and mobile) and a public developer API, served by the same fleet
• Several model sizes: a large flagship model, a mid-size model and a small fast model
• Streaming responses, conversation history and system prompts
• Per-user and per-organisation quotas, with paid tiers that get priority
• Usage metering for billing by input and output tokens

What I'm NOT building here:

• Model training, RLHF or evaluation pipelines
• Retrieval (RAG) and tool execution. I treat these as callers of the platform: they build a prompt and send it in.
• Content moderation internals. I show where the safety check sits, but its design is its own system.

Key assumptions:

• Models are decoder-only transformers, which covers every major chat model today
• I run on NVIDIA H100-class GPUs with 80 GB of memory, in nodes of 8 connected by NVLink
• The flagship model is about 70B parameters, served in FP8, so its weights take about 70 GB
• Conversations average around 1,500 input tokens once history and the system prompt are included, and about 400 output tokens

How this differs from classic serving: in a recommendation or ads system, an inference call is a single forward pass that returns a score in milliseconds. LLM generation is autoregressive. The model runs once per output token, and each step depends on the one before it. A 400-token answer is 400 sequential forward passes, and the request keeps state on the GPU the whole time. That state, the KV cache, turns out to drive almost every decision in the design.

Functional and Non-Functional Requirements

Functional Requirements

1. Generate: accept a list of messages plus parameters (model, max tokens, temperature) and return generated text.
2. Stream: return tokens as they are produced, not after the full answer is done.
3. Cancel: stop generation when the user presses stop or disconnects, and free the GPU immediately.
4. Quota: enforce per-tier limits on requests and tokens per minute, and on concurrent requests.
5. Meter: record input, cached and output tokens per request, for billing and analytics.
6. Route: send each request to the right model and region, with fallback when a pool is unhealthy.

Non-Functional Requirements

I track TTFT and ITL separately because they have different causes. TTFT is dominated by queueing plus prefill (processing the whole prompt). ITL is dominated by decode (one step per token) and by how many other requests share the GPU.

Back-of-the-Envelope Capacity Estimation

These numbers are my planning assumptions, not measurements. The point is the method: estimate tokens per second, then divide by measured throughput per node.

Traffic:

• 10 million daily active users, 20 requests each per day: 200 million requests per day
• 200M / 86,400 s ≈ 2,300 requests per second on average, about 7,000 at peak (3x)

Tokens:

• Output: 2,300 × 400 ≈ 0.9 million output tokens per second on average, about 2.8 million at peak
• Input: 2,300 × 1,500 ≈ 3.5 million input tokens per second on average

KV cache per token (70B-class model, 80 layers, grouped-query attention with 8 KV heads of dimension 128, FP16 cache):

bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes
                = 2 x 80 x 8 x 128 x 2
                = 327,680 bytes, about 320 KB per token

A single 8K-token conversation therefore holds about 2.6 GB of KV cache. On an 8-GPU node with 640 GB of memory, about 70 GB goes to weights and the remainder is mostly KV cache. That caps how many conversations can be decoding at once on that node. Memory, not compute, is usually the first limit on concurrency.

Fleet size: I benchmark a node to find the output-token throughput it sustains while still meeting the ITL target. Suppose it is around 15,000 output tokens per second for this model at a healthy batch size. Then peak decode needs 2.8M / 15K ≈ 190 nodes, about 1,500 GPUs, for the flagship model alone, before regional redundancy. Prefill adds more, though prefill is compute-bound and processes tokens far faster than decode. This is why the small and mid-size models matter: every request I can serve well on a cheaper model frees flagship capacity.

High-Level Architecture and Core Components

Component Walkthrough

API Gateway. Terminates TLS, authenticates API keys and user sessions, validates the request shape and counts input tokens with the model's tokenizer. Counting tokens early matters: quota checks, routing and admission all need to know the request's size before it reaches a GPU.

Quota Service. Decides whether the request is allowed right now under its tenant's request, token and concurrency limits. Backed by Redis for low-latency counters. It is covered in the third deep-dive.

Safety check. A fast classifier on input (and later on output) that runs in parallel with routing so it adds little latency. It can block or flag. I treat it as a dependency with its own SLO.

Request Router. Chooses the model pool (from the requested model, tier and current health) and then a specific replica, preferring one that already holds this prompt's prefix in cache. It is covered in the fourth deep-dive.

Inference servers. Each replica is a model loaded across one or more GPUs (for example, a 70B model split across 2 or 4 GPUs with tensor parallelism) running an engine such as vLLM, TensorRT-LLM or SGLang. Inside, a scheduler does continuous batching over a paged KV cache. This is the heart of the system and the first deep-dive.

Stream relay. Carries tokens from the inference server back through the gateway to the client over Server-Sent Events, and propagates cancellation in the other direction.

Metering stream and billing. Every finished or cancelled request emits a usage event (tenant, model, input tokens, cached tokens, output tokens, latency). Events go to Kafka, then to a warehouse for billing, dashboards and capacity planning.

Model registry and weights store. Model versions, their configs and their weights live in object storage. New replicas pull weights on start. Loading tens of gigabytes is slow, so I cache weights on local NVMe and pre-warm replicas before traffic shifts.

Request Flow

1. The client sends POST /v1/chat with messages and stream: true.
2. The gateway authenticates, tokenizes (1,500 tokens) and asks the quota service for admission.
3. The router picks the flagship pool in the nearest healthy region, then the replica with the best prefix-cache match and room in its queue.
4. The replica's scheduler adds the request to the running batch at the next step. Prefill runs, and the first token streams back.
5. Decode continues one token per step until a stop token or max_tokens. Each token is relayed to the client.
6. On completion or cancellation, the scheduler frees the KV blocks and emits a usage event.

Scaling It Out

• Pools per model and per workload. Interactive chat and offline batch jobs share models but not queues. Batch work fills the gaps at night and is preempted first under load.
• Multi-region. GPUs are scarce, so I cannot always serve from the nearest region. The global balancer routes on latency and free capacity. A user in Mumbai may be served from Singapore at peak. The gap of tens of milliseconds is small next to generation time.
• Autoscaling on the right signal. CPU usage means nothing here. I scale on queue depth, KV cache usage and TTFT. Because cold starts take minutes (fetching weights, loading them onto GPUs, warm-up), I scale ahead of the daily curve and keep a warm buffer.

Deep-Dive: GPU Scheduling, Continuous Batching, and the KV Cache

Why Batching Is Everything

During decode, each step reads the entire model's weights from GPU memory to produce one token per sequence. At a batch size of 1, the GPU spends nearly all its time moving weights and very little computing. Decode is memory-bandwidth-bound. If I batch 64 sequences, the same weight read serves 64 tokens. Throughput rises almost linearly while each sequence's step time grows only a little. This is where serving cost comes from: cost per token falls roughly with batch size, until the KV cache runs out of memory or ITL breaks its target.

Static Batching Fails for Generation

The naive approach groups N requests, runs them together and returns when all are done. Generation lengths vary a lot: one answer is 20 tokens, another is 2,000. With static batching, the short requests finish and their slots sit empty until the longest one ends. The GPU runs mostly empty, and new requests wait for the whole batch.

Continuous (Iteration-Level) Batching

The fix, introduced by the Orca paper and now standard in every serious engine, is to schedule per step rather than per request. After every forward pass, the scheduler:

1. Removes sequences that just finished, freeing their slots and KV memory
2. Admits waiting requests into the free slots
3. Runs the next step on whatever is in the batch now

A request joins the moment there is room and leaves the moment it is done. Utilisation stays high, and new requests do not wait behind long generations.

# Simplified scheduler loop (one iteration per model forward pass)
while True:
    for seq in running:
        if seq.finished or seq.cancelled:
            block_manager.free(seq)
    running = [s for s in running if not (s.finished or s.cancelled)]

    # Admit by priority, as long as KV blocks and the token budget allow
    budget = MAX_TOKENS_PER_STEP - len(running)       # one decode token each
    while waiting and budget > 0:
        req = waiting.peek_highest_priority()
        chunk = min(req.remaining_prompt, budget)      # chunked prefill
        if not block_manager.can_allocate(req, chunk):
            break                                      # memory-bound: stop admitting
        block_manager.allocate(req, chunk)
        req.schedule_prefill(chunk)
        budget -= chunk
        if req.prefill_done:
            running.append(waiting.pop(req))

    outputs = model.forward(running + prefilling)      # one fused step
    stream_tokens(outputs)

Prefill vs Decode: Two Different Workloads

Prefill processes the whole prompt in one pass and is compute-bound, with large matrix multiplications. Decode produces one token at a time and is memory-bound. Mixing them carelessly causes a familiar problem: a 30,000-token prompt arrives, its prefill takes a large share of a second, and every decoding user in that batch sees a stall in their stream.

I use two techniques here:

• Chunked prefill. Split long prompts into chunks (say 512 or 2,048 tokens) and mix one chunk with the ongoing decode steps. Each step's work stays bounded, so ITL stays smooth, at a small cost to that request's TTFT.
• Disaggregated prefill and decode. At large scale, run separate pools: prefill nodes tuned for compute, decode nodes tuned for memory, with the KV cache sent between them over fast interconnect. Systems like DistServe and Splitwise showed this can raise goodput under strict latency SLOs. It adds real complexity, because moving gigabytes of KV cache per request is not free. I would start with chunked prefill and move to disaggregation only once TTFT and ITL targets keep fighting each other.

The KV Cache and Paged Attention

To avoid recomputing attention over the whole history at every step, the model stores each token's key and value vectors: the KV cache. At about 320 KB per token for my flagship model, it is the dominant use of memory.

Early servers reserved one contiguous block per request, sized for the maximum length. A request that might reach 8K tokens but stops at 300 wasted over 95% of its reservation, and fragmentation wasted more. PagedAttention (the idea behind vLLM) borrows virtual memory from operating systems. The KV cache is split into fixed-size blocks (for example 16 tokens each). Each sequence has a block table mapping logical positions to physical blocks, and blocks are allocated on demand as the sequence grows. Waste drops to under one block per sequence, so many more sequences fit in a batch, and throughput rises accordingly.

Paging also enables two features I rely on later:

• Sharing: sequences with the same prefix (the same system prompt) can point at the same physical blocks, copy-on-write.
• Preemption: when memory runs out mid-generation, the scheduler can evict a low-priority sequence's blocks to CPU memory (swap) or drop them and recompute later, instead of failing the request.

Memory Pressure and Preemption Policy

KV usage grows with every generated token, so a node that admitted too many requests can run out of memory mid-decode. My policy:

1. Admission control first. Admit only when the projected memory (prompt plus expected output) fits, with headroom.
2. When pressure hits anyway, preempt the lowest-priority, most recently admitted sequence. Recently admitted ones have done the least work, so they are cheapest to redo.
3. Swap or recompute: for short sequences, recomputing prefill is cheaper than moving blocks over PCIe. For long ones, swap to CPU memory.
4. Never preempt the paid interactive tier for batch work. Priority classes are enforced inside the scheduler, not just at the gateway.

Other Levers

• Quantisation: FP8 weights and an FP8 KV cache roughly halve memory against FP16, which means larger batches. I gate every quantised build on an evaluation suite, because quality loss shows up unevenly across tasks.
• Speculative decoding: a small draft model proposes several tokens and the large model verifies them in one pass. Verification is exact, so output quality is unchanged. It lowers ITL when the draft is often right, but it uses extra compute, so it helps most at low and medium load.
• Parallelism choice: tensor parallelism within a node (over NVLink) for models that do not fit on one GPU. I add pipeline parallelism across nodes only when a model does not fit in one node, because pipeline bubbles cost throughput.

Deep-Dive: Token Streaming and Long-Lived Connections

Why SSE

Generation is one-way after the request is sent: the server pushes tokens and the client reads them. Server-Sent Events fits this exactly. It is plain HTTP, works through proxies and CDNs, has built-in event ids for resuming, and is trivial to consume from browsers and SDKs. WebSockets add two-way framing that I do not need for basic chat. I would use them, or WebRTC, only for voice and real-time multimodal sessions.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

id: 1
data: {"delta": "The", "index": 0}

id: 2
data: {"delta": " KV", "index": 0}

id: 3
data: {"delta": " cache", "index": 0}

event: done
data: {"usage": {"input_tokens": 1512, "cached_tokens": 1280, "output_tokens": 412}}

Long-Lived Connections at Scale

At 7,000 new requests per second, each streaming for 5 to 30 seconds, I hold on the order of 100,000 concurrent open streams at peak. Each one is cheap on its own (an idle socket and a small buffer), but together they shape the edge tier:

• Non-blocking I/O everywhere. The gateway and relay use event-loop servers (Envoy, Go or Rust services), not thread-per-request designs.
• Idle timeouts tuned for streams. Load balancers often kill connections after 60 seconds of no data. Long reasoning steps can be silent that long, so I send SSE comment heartbeats (: ping) every 15 seconds.
• Turn off proxy buffering. One buffering proxy that waits for the response to complete turns a streaming product into a batch one. I set X-Accel-Buffering: no and test every hop end to end.
• Connection draining on deploys. A gateway pod that is shutting down stops accepting new streams but keeps existing ones until they finish, with a hard limit.

Cancellation Is a Cost Feature

When a user closes the tab or presses stop, every further token is wasted GPU time. The gateway detects the closed connection, sends a cancel to the replica, and the scheduler removes the sequence at the next step and frees its KV blocks. Without this path, abandoned requests keep generating to max_tokens. On a chat product where people often stop long answers, that wastes a lot of capacity. Billing counts only the tokens actually generated.

Resuming Dropped Streams

Mobile clients switch networks often. Instead of regenerating the whole answer, the relay writes each event to a short-lived Redis stream keyed by request id (expiring after a few minutes). If the client reconnects with Last-Event-ID, the relay replays from that point and continues live. Generation itself never stopped. This keeps a network blip from costing a full regeneration.

Backpressure

If a client reads slowly, tokens pile up. The relay buffers per stream up to a limit. Past that limit I would rather drop the client connection, keep generating into the resume buffer, and let the client reconnect than block the inference server. The GPU loop must never wait on a client socket.

Deep-Dive: Multi-Tenant Quotas, Token-Based Rate Limits, and Priority Tiers

Why Requests per Second Is the Wrong Unit

One request might be "hi" (10 tokens) and the next a 100,000-token document summary. Their cost differs by four orders of magnitude, so a plain request-rate limit is useless for protecting GPUs. I limit on four dimensions per tenant:

Charging Tokens Before You Know the Count

Input tokens are known at the gateway. Output tokens are not known until generation ends. My approach:

1. At admission, reserve input_tokens + max_tokens (or a typical output estimate for this tenant) from the token bucket.
2. When the request finishes, reconcile: refund the unused part of the reservation, or charge the shortfall.

The reservation stops a tenant from launching 1,000 requests at once on a budget meant for 10. The reconciliation keeps billing and limits accurate.

-- Redis Lua: atomically reserve tokens from a tenant's bucket (refill on read)
local key, now = KEYS[1], tonumber(ARGV[1])
local rate, cap, cost = tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
local b = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) * rate)
if tokens < cost then
  return {0, tokens}             -- reject, caller returns 429 with retry-after
end
redis.call('HMSET', key, 'tokens', tokens - cost, 'ts', now)
redis.call('EXPIRE', key, 120)
return {1, tokens - cost}

Buckets are sharded by tenant id across a Redis cluster. For very large enterprise tenants whose traffic would make one key hot, I split the bucket into N sub-buckets and give each gateway region a slice of the budget, rebalanced every few seconds.

Priority Tiers

Quotas answer "is this tenant within its limit". Priority answers "who goes first when the fleet is full". I keep them separate:

• Tiers map to queues: enterprise and paid interactive, then free interactive, then batch. The scheduler pulls by weighted fair queuing, not strict priority, so free users slow down under load but are never starved forever.
• Fairness within a tier: deficit round-robin across tenants, so one heavy tenant in the paid tier cannot fill the queue ahead of everyone else.
• Provisioned throughput: some enterprise customers buy reserved capacity. I model this as dedicated replicas, or a guaranteed share of a pool's token throughput that the scheduler protects.

Graceful Degradation Under Overload

When demand exceeds capacity, which happens at every launch, I degrade in a fixed order instead of failing at random:

1. Pause the batch tier
2. Lower max_tokens defaults for free users and route them to the mid-size model
3. Shed new free-tier requests with a clear 429 and retry-after
4. Only then add queueing delay for paid tiers

Load shedding happens at the gateway, before prefill. Rejecting a request after it has used GPU time is the worst outcome.

Deep-Dive: Model Routing, Prompt Caching, and Cost per Token

Prompt (Prefix) Caching

Most prompts share a long, stable prefix: the same system prompt, the same tool definitions, the same retrieved document, or the earlier turns of the same conversation. Recomputing prefill for that prefix on every turn is pure waste. With a paged KV cache, a replica can keep the KV blocks for recent prefixes (indexed by a hash of each block's tokens, chained so a hit covers the longest matching prefix) and reuse them. A cache hit skips that part of prefill, which cuts TTFT and compute together. This is why API providers price cached input tokens well below fresh ones.

The catch is that the cache is local to a replica. If turn 2 of a conversation lands on a different replica than turn 1, the cache misses.

Cache-Aware Routing

So the router is not a plain round-robin load balancer. For each request it:

1. Computes the hash of the prompt's prefix blocks
2. Uses consistent hashing (or a prefix-to-replica index) to find the replicas likely to hold the longest prefix
3. Chooses among them by load, so a popular system prompt does not overload one replica
4. Falls back to the least-loaded replica if the cache-affine ones are too busy

This is a classic trade-off between affinity and balance. Pure affinity creates hot spots. Pure load balancing throws away cache hits. I use bounded-load consistent hashing: go to the affine replica unless its load is more than, say, 1.25 times the pool average.

For long conversations, I also pin the session to its replica with a soft affinity key (the conversation id), so multi-turn chats keep hitting their own cached history.

Choosing the Model

Not every request needs the flagship model. Model selection is the single biggest lever on cost per token:

• Explicit: the API caller names a model, so I route to that pool.
• Product-level routing: in the chat product, a lightweight classifier (or rules on length, attachments and the user's chosen mode) sends simple requests to a smaller model and escalates hard ones.
• Degradation routing: under overload, the free tier moves to the mid-size model (as in the previous section).

I log the router's decision with every request and evaluate it offline. A router that saves 40% of cost but quietly lowers answer quality is a regression, not a win.

Deploying New Models Safely

Model versions change behaviour, not just speed. I deploy them like risky schema changes:

• Shadow traffic first: copy real prompts to the new version and compare latency and offline quality scores, without returning its output.
• Canary by percentage, per tier, watching TTFT, ITL, error rate, refusal rate and user feedback signals.
• Pinned versions for API customers (model-2026-10-01) so their products do not change under them, with a published deprecation window.

Cost per Token: The Unit Economics

The number I put on the dashboard is cost per million output tokens, per model and per tier:

cost per 1M output tokens = (GPU-hours x price per GPU-hour) / (output tokens served / 1M)

What moves it, roughly in order of impact:

1. Batch size at the ITL target. Paged KV cache, FP8 and good admission control all raise this.
2. Prefix cache hit rate. It saves prefill compute directly.
3. Model mix. It decides how much traffic reaches the smaller pools.
4. Wasted tokens. Uncancelled abandoned streams, retries and overlong max_tokens defaults.
5. Idle capacity. The gap between the diurnal peak and trough. The batch tier and spot-style internal jobs fill the trough.

If I can explain each week's change in this one number, I understand the platform.

Conclusion: Bringing It All Together

An LLM inference platform looks like an API on the outside and like an operating system on the inside. The gateway, quota service and router are familiar distributed-systems parts, with one change: the unit of cost is the token, not the request. Every limit, priority decision and bill is expressed in tokens.

Inside each replica, the important ideas are borrowed from systems engineering:

• Continuous batching is per-iteration scheduling.
• Paged attention is virtual memory for the KV cache.
• Prefix caching is a shared-page cache with copy-on-write.
• Preemption and swap are what keep the system up under memory pressure.

If I had to summarise the design in four decisions:

1. Batch continuously and page the KV cache. GPU memory is the binding limit, so manage it like an OS manages RAM.
2. Stream over SSE and treat cancellation as a first-class path. It protects both the user experience and the GPU budget.
3. Meter in tokens with reserve-then-reconcile, and separate quota from priority. That is what makes a shared fleet fair.
4. Route for cache affinity and model fit. It cuts TTFT and cost in the same move.

Related reading: REST, GraphQL, gRPC, WebSockets, SSE, and WebRTC Compared, Designing a Distributed Rate Limiter, Load Balancers, Reverse Proxies, API Gateways, and Service Meshes, and LLM Memory Compaction.