# Designing an LLM Fine-Tuning Platform ## Blog Details - **Author**: Naveen R. - **Date**: November 3, 2026 - **Tags**: LLM, fine-tuning, distributed systems, machine learning infrastructure, LoRA - **Read Time**: 20 mins When I design an LLM fine-tuning platform, the kind of service where a customer uploads a few thousand examples and gets back "their" version of an open-weight model, I start from an uncomfortable fact: almost every part of it is a batch system, yet customers judge it like a product. They expect a job to start soon, survive hardware failures they never hear about, and produce a model that is better at their task and not quietly worse at everything else. Meanwhile every job burns GPU-hours, and a single bad dataset can waste a day of a 16-GPU slice. So the design is about three things: **turning messy customer data into trustworthy training input**, **keeping expensive GPUs doing useful work despite failures and preemption**, and **making the output cheap to serve and safe to ship**. Cheap serving pushes me toward parameter-efficient adapters. Safe shipping puts an evaluation gate between training and production. In this post I walk through how I would build this platform from scratch. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives: - Dataset ingestion, validation, versioning and dedup - Distributed training with data and tensor parallelism, checkpointing and preemption - LoRA adapters and multi-adapter serving - Evaluation gates and safe rollout ## Scoping the Problem and Clarifying Assumptions I am building a **multi-tenant fine-tuning service**: customers bring data, pick a base model and a recipe, and receive a fine-tuned model they can call through an API. Think of the managed fine-tuning offerings from model providers and clouds, or an internal platform shared by many product teams. **What I'm including:** - Dataset upload, validation, versioning and deduplication - Supervised fine-tuning (SFT) on chat-formatted examples, plus preference tuning (for example DPO) as a second recipe - LoRA as the default method, with full fine-tuning available for large customers - A shared GPU cluster with quotas, priorities and preemptible capacity - Checkpointing, resumption and failure handling - An evaluation gate and a staged rollout into a multi-adapter serving tier **What I'm NOT building here:** - Pre-training from scratch. That is a months-long, single-tenant run with very different economics. - The base-model inference engine itself (continuous batching, KV cache, token streaming). I build on top of it and cover only what adapters add. - A general eval and observability product. I cover the gate inside the training-to-deploy pipeline and link out for the rest. **Key assumptions:** - Base models are open-weight decoder-only transformers in two sizes: an **8B-class** model and a **70B-class** model - Training runs on H100-class GPUs with 80 GB of memory, 8 per node over NVLink, and a fast inter-node network (InfiniBand or equivalent) - Most customers have between 1,000 and 100,000 examples, averaging around 1,000 tokens each - Roughly 85% of jobs target the 8B model and use LoRA **How this differs from the classic system:** a classic ML training platform (say, for a recommendation model) trains one model per team on data the team already owns and swaps it in behind a fixed interface. Here the data arrives from strangers in arbitrary shape, the base model is enormous and frozen, and thousands of tenants each want their own variant on serving hardware I cannot dedicate per tenant. The interesting constraints come from that multiplicity: **thousands of small models riding on a few big ones**. ## Functional and Non-Functional Requirements ### Functional Requirements 1. **Upload datasets**: accept JSONL files of chat examples (or preference pairs) up to tens of gigabytes, resumably. 2. **Validate and version**: reject malformed data with row-level errors, deduplicate, and produce an immutable dataset version. 3. **Create jobs**: start a training job from (base model, dataset version, recipe, hyperparameters), with cost and time estimates up front. 4. **Run and report**: show progress, training and validation loss, and events (started, preempted, resumed, failed). 5. **Evaluate and gate**: score every candidate against held-out data and regression suites, and block bad ones. 6. **Deploy**: make a passed model callable by a stable name, with staged rollout and one-click rollback. 7. **Lineage**: for any deployed model, answer "which data, code, base model and hyperparameters produced this?". ### Non-Functional Requirements | Requirement | Target | Why | |-------------|--------|-----| | Job start latency | under 15 minutes p90 for small jobs at normal load | Customers iterate on data and want feedback the same hour | | Goodput | over 90% of allocated GPU time does useful training | Lost work after a failure is pure cost | | Durability | no completed job lost, at most one checkpoint interval of work lost on failure | Jobs can run for many hours | | Isolation | no tenant can read another's data, weights or eval results | Datasets are often proprietary | | Reproducibility | a job can be rerun from its recorded inputs | Debugging and audit | | Serving cost | thousands of fine-tuned models on a shared base fleet | Dedicated replicas per tenant do not scale | I separate **goodput** from utilisation on purpose: a GPU redoing work lost in a crash is busy but not productive. ### Back-of-the-Envelope Capacity Estimation These are my planning assumptions, not measurements. The method is what matters: tokens per job, FLOPs per token, effective FLOP rate per GPU. **Jobs:** - I assume **2,000 training jobs per day**, about 1.4 per minute. The control plane is trivial at this rate; the GPUs are not. - Average job: 20,000 examples × 1,000 tokens × 3 epochs ≈ **60 million training tokens** **Compute per job.** The rule of thumb for transformer training is about 6 × parameters FLOPs per token. LoRA skips the frozen base's weight gradients, so it costs somewhat less, but I use 6N as a conservative upper bound: ```text 8B job: 6 x 8e9 x 6e7 tokens = 2.9e18 FLOPs 70B job: 6 x 7e10 x 6e7 tokens = 2.5e19 FLOPs H100 dense BF16 peak is roughly 1e15 FLOP/s. I assume 40% model FLOPs utilisation: 4e14 FLOP/s per GPU. 8B job: 2.9e18 / 4e14 = about 7,200 GPU-seconds = 2 GPU-hours 70B job: 2.5e19 / 4e14 = about 63,000 GPU-seconds = 17.5 GPU-hours ``` **Fleet:** - 1,700 small jobs × 2 GPU-hours = 3,400 GPU-hours - 300 large jobs × 17.5 GPU-hours = 5,250 GPU-hours - Total ≈ **8,650 GPU-hours per day**, or about 360 GPUs busy around the clock - Evals add roughly 10%, and packing is never perfect. Planning for 70% scheduled utilisation, I need about **570 GPUs, roughly 72 nodes**, plus a preemptible pool for peaks **Memory, which decides parallelism.** Full fine-tuning with Adam in mixed precision needs about 16 bytes per parameter (BF16 weights and gradients, FP32 master weights and two optimizer moments). For the 70B model that is about **1.1 TB** before activations, more than one 8-GPU node holds. LoRA keeps the frozen base in BF16 (about 140 GB) and trains a tiny adapter, so it fits on one node. **That one fact is why LoRA is the default.** **Adapter size.** LoRA rank 16 on all attention and MLP projections of the 8B model adds about 40 million parameters, around **80 MB in BF16** (about 400 MB for the 70B model). A full 70B checkpoint with optimizer state is over 1 TB. Megabytes against terabytes drives both the checkpointing and serving designs. Datasets are small by comparison: about 80 MB raw per job, or 160 GB per day. ## High-Level Architecture and Core Components ![The fine-tuning platform end to end: clients call the fine-tuning API, a job service records jobs and lineage in a SQL database and sends datasets through a validation pipeline into versioned shards in object storage; the GPU scheduler gang-launches training on the cluster, which reads shards and writes checkpoints; candidates pass through an eval gate into the adapter registry, and the multi-adapter serving tier hot-loads them.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-llm-fine-tuning-platform/01-high-level-architecture.png) ### Component Walkthrough **Fine-tuning API.** Authenticates the tenant, accepts uploads (via pre-signed multipart URLs, so bytes skip my API servers) and job requests, and returns cost and time estimates from the token-count arithmetic above. **Job service and metadata DB.** The job service owns the state machine: `QUEUED → VALIDATING → SCHEDULED → RUNNING → EVALUATING → SUCCEEDED | FAILED | CANCELLED`, with `PREEMPTED` looping back to `SCHEDULED`. State, dataset versions, checkpoints and lineage live in PostgreSQL: small, but the source of truth for billing and audit. **Dataset pipeline.** Workers that validate, scan, deduplicate, tokenize and shard uploads into immutable dataset versions. Covered in the first deep-dive. **GPU job scheduler.** Places jobs with **gang scheduling** (all GPUs a job needs, or none), tenant quotas, priority tiers and preemption. Holding 15 of a job's 16 GPUs idle while waiting for the last one is the classic way to waste a cluster. **Training cluster.** Nodes run a pinned training image with base weights cached on local NVMe, stream dataset shards and train with the recipe's parallelism plan. Covered in the second deep-dive. **Eval gate.** Runs held-out and regression evaluations on the final checkpoint and decides whether it can be deployed. Covered in the fourth deep-dive. **Adapter registry and serving tier.** The registry maps a stable model name to an adapter and its base model. The serving tier runs base models with many adapters loaded. Covered in the third deep-dive. ### Request Flow 1. The client uploads `support-chats.jsonl` to a pre-signed URL and calls `POST /v1/datasets` to register it. 2. The dataset pipeline validates, deduplicates and writes `ds_acme_support@v3`. The client sees a report: 18,412 valid rows, 1,104 dropped as duplicates, 37 rejected with reasons. 3. The client calls `POST /v1/fine-tunes` with the base model, dataset version and recipe. The job service checks quota and enqueues the job. 4. The scheduler gang-allocates 8 GPUs on one node. The trainer starts, logs loss every few steps and writes checkpoints. 5. On completion, the eval gate scores the candidate. It passes, and the adapter is registered as `ft:acme-support:v3`. 6. The serving tier starts routing shadow, then canary, then all traffic for that name to the new adapter. ### Scaling It Out ![The scaled-out control and compute plane: jobs enter per-tier queues, a gang scheduler with quotas, informed by a node health checker that cordons bad nodes, places large and priority jobs on a reserved GPU pool, small resumable jobs on a preemptible pool and evaluations on a separate eval pool; the training pools write checkpoints to a regional store, and pool metrics feed monitoring.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-llm-fine-tuning-platform/02-scalable-architecture.png) - **Pools by risk, not by tenant.** A reserved pool runs large and priority jobs. A preemptible pool (spot capacity, or idle inference GPUs at night) runs small LoRA jobs that checkpoint often and resume cheaply. A separate eval pool keeps evaluation from queueing behind training. - **Topology-aware placement.** Tensor parallelism never crosses a node. Multi-node jobs go under the same network switch where possible, because all-reduce speed sets step time. - **Health as a scheduling input.** At a few hundred GPUs, some are always degraded (flaky NVLink, ECC errors, a slow NIC). A health checker runs burn-in tests between jobs and cordons bad nodes, because synchronous training waits for the slowest rank. - **Regional storage.** Checkpoints and shards stay in the GPUs' region; moving terabyte checkpoints across regions is slow and expensive. ## Deep-Dive: Dataset Ingestion, Validation, Versioning, and Dedup ![The dataset pipeline: a multipart upload lands in raw object storage, a schema and token validator checks each row, a PII and policy scan filters content, an exact and near-duplicate stage queries and updates a MinHash LSH index, and the survivors are written as immutable tokenized shards with a manifest committed to the dataset versions table; errors and duplicate statistics feed a validation report.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-llm-fine-tuning-platform/03-dataset-ingestion.png) ### Why Data Gets Its Own Pipeline Most failed fine-tunes I would expect are data failures, not infrastructure failures: wrong CSV quoting, every assistant turn truncated at 256 characters, or the same 200 examples repeated 50 times. Training happily runs on all of that and produces a worse model. So the cheapest checks run **before** a GPU is allocated, and their output is shown to the customer. ### Upload and Validation Uploads go straight to object storage. A validator streams each file and checks every row: - **Schema**: valid JSON, a `messages` list with allowed roles, at least one assistant turn to learn from. Preference rows need a prompt plus `chosen` and `rejected`. - **Tokens**: tokenize with the base model's tokenizer and reject rows longer than the context length. Counts also feed the cost estimate. - **Distribution sanity**: warnings for oddities such as a median assistant length of 3 tokens or one response repeated thousands of times. Errors are reported per row, and the dataset fails fast if too many rows are invalid. A dataset that is 30% broken usually means the export was wrong, and training on the other 70% hides the problem. ### Safety and PII Scanning A scanning stage runs pattern detectors and a classifier for PII (emails, card numbers, credentials) and policy violations. Per tenant settings it redacts, drops the row or fails the dataset. Some categories are blocked regardless of settings, because a model fine-tuned to remove safety behaviour is a risk to the provider, not just the customer. ### Deduplication Duplicates cause overfitting and skew evaluation, because a "held-out" row that also appears in training is not held out. I run two passes: 1. **Exact dedup.** Normalise (Unicode NFC, collapsed whitespace) and hash each full conversation. 2. **Near dedup.** Customer data is full of templated near-copies, like the same support macro with a different order number. I use **MinHash with locality-sensitive hashing**: each example becomes a set of word n-grams, a 128-value MinHash signature approximates Jaccard similarity, and LSH banding finds candidate pairs without comparing everything to everything. Pairs above a threshold (say 0.8) collapse to one. ```python # Near-duplicate detection with MinHash + LSH banding (simplified) NUM_PERM, BANDS = 128, 32 # 32 bands x 4 rows ROWS = NUM_PERM // BANDS def signature(text): shingles = {" ".join(w) for w in ngrams(normalize(text).split(), 5)} return [min(h(i, s) for s in shingles) for i in range(NUM_PERM)] def dedup(examples, lsh_index, threshold=0.8): kept = [] for ex in examples: sig = signature(ex.text) band_keys = [hash(tuple(sig[b*ROWS:(b+1)*ROWS])) + b for b in range(BANDS)] candidates = lsh_index.lookup(band_keys) # same bucket in any band if any(jaccard_est(sig, c.sig) >= threshold for c in candidates): continue # near-duplicate: drop lsh_index.insert(band_keys, ex.id, sig) kept.append(ex) return kept ``` Dedup also runs **across** the training set and the customer's eval set, which is where it matters most. Any training row that near-matches an eval row is removed and counted in the report. That single check prevents the most common way a fine-tune looks great in testing and disappoints in production. ### Versioning Every accepted dataset becomes an **immutable, content-addressed version**: tokenized, shuffled shards plus a manifest listing every shard by hash, with the tokenizer and pipeline config that produced them: ```json { "dataset": "ds_acme_support", "version": 3, "parent_version": 2, "source_upload_sha256": "9f2c...e1", "tokenizer": "base-8b-tokenizer@2026-07", "pipeline": {"dedup": "minhash-0.8", "pii": "redact", "max_tokens": 8192}, "stats": {"rows_in": 19553, "rows_kept": 18412, "dup_dropped": 1104, "invalid": 37}, "shards": [ {"path": "s3://ft-data/acme/ds_acme_support/v3/00000.bin", "sha256": "a41b...07", "tokens": 6291456} ], "manifest_sha256": "c07d...9a" } ``` A job references `ds_acme_support@v3` by manifest hash, never "latest", which makes it reproducible and traceable. Tokenizing once at ingestion also takes CPU work off the GPU nodes. ### Failure Modes and What I Would Do First - **Huge uploads.** Validation is split by byte range across workers, then merged. - **Tokenizer drift.** The manifest records the tokenizer, and the job service refuses a dataset built for a different base tokenizer. - **Deletion requests.** Deletion is a tombstone plus a garbage collector that removes shards once no running job references them. Adapters trained on that data keep a lineage record saying so. **First version:** schema and token validation, exact dedup, immutable versions. **Later:** near dedup across train and eval, distribution warnings, and suggested fixes in the report. ## Deep-Dive: Distributed Training: Data and Tensor Parallelism, Checkpointing, and Preemption ![One multi-node training job: the gang scheduler reserves every GPU at once and a rendezvous store assigns ranks to two 8-GPU nodes, which exchange gradients with all-reduce; a preemption handler signals all ranks through the store to save on notice, each node snapshots asynchronously to local NVMe before an upload with a commit marker to durable checkpoint storage, and heartbeats go to monitoring.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-llm-fine-tuning-platform/04-distributed-training.png) ### Choosing the Parallelism Plan The platform picks the plan from the base model, method and job size; customers never configure it. - **Data parallelism (DP).** Every GPU holds a full copy, processes a different slice of each batch, and gradients are averaged with all-reduce. For LoRA on the 8B model this is enough: the frozen base (16 GB in BF16) fits on each GPU, and the all-reduce moves only the adapter's gradients. - **Sharded data parallelism (ZeRO / FSDP).** Shard weights, gradients and optimizer state across GPUs and all-gather each layer just in time. PyTorch FSDP and DeepSpeed ZeRO stage 3 both implement this, and it is how full fine-tuning fits without changing model code. - **Tensor parallelism (TP).** Split each large matrix multiplication across GPUs (the Megatron-LM approach). It communicates inside every layer, so I keep it within a node over NVLink. - **Pipeline parallelism (PP).** Different layers on different nodes. It adds bubbles and complexity, so I use it only when sharding alone runs short. My default plans: | Job | Plan | Footprint | |-----|------|-----------| | LoRA on 8B | DP over 1 to 8 GPUs | 1 node or less | | LoRA on 70B | TP 8 within a node (or FSDP), DP across nodes if the dataset is large | 1 to 2 nodes | | Full fine-tune on 8B | FSDP over 8 GPUs | 1 node | | Full fine-tune on 70B | FSDP across nodes, TP 8 within each node | 2 to 4 nodes | My rule: **use the least parallelism that fits in memory, then add data parallelism for speed.** Every extra dimension adds communication, failure surface and debugging cost. ### Gang Scheduling and Rendezvous The scheduler launches a job only when its full set of nodes is free, and backfills short or preemptible jobs into the gaps while a large job waits. Ranks then find each other through a **rendezvous store** (elastic launchers such as torchrun use one): each process registers, gets a rank, and training starts only when all have joined. If a node fails during rendezvous, the group restarts rather than running degraded. ### Checkpointing In synchronous training, if one GPU of 32 dies, every rank stops. At a few hundred GPUs, something fails every day, so checkpoints turn "start over" into "lose a few minutes". To resume exactly, a checkpoint holds the trainable weights, optimizer state, scheduler state and step, each rank's data loader position (so no example is skipped or repeated) and RNG state. For LoRA this is a few hundred megabytes. For a full 70B fine-tune it is over a terabyte, and writing it straight to object storage would stall training for minutes. So I checkpoint in two tiers: 1. **Fast local snapshot.** Each rank copies its shard to host memory and training continues; a background thread writes it to local NVMe. 2. **Durable upload.** A background process uploads to the regional object store and, once every rank's shard is there, writes a **commit marker**. Resume only uses committed checkpoints, which avoids half the ranks resuming from step 4,000 and half from 3,500. The classic Young approximation sets the interval: √(2 × checkpoint cost × mean time between failures). With an assumed 30 seconds of blocking cost and 8 hours between failures, that is √(2 × 30 × 28,800) ≈ 1,300 seconds, about **every 20 minutes**. Asynchronous saving lowers the cost, so I can checkpoint more often. ### Preemption Preemption happens for two reasons: the cloud reclaims spot capacity (with a short warning, about two minutes on AWS), or my own scheduler evicts a low-priority job to make room for a higher-priority one. Both use the same path: ```python # Trainer-side preemption handling (simplified) preempt_requested = threading.Event() signal.signal(signal.SIGTERM, lambda *_: preempt_requested.set()) for step in range(start_step, total_steps): loss = train_step(next(loader)) # All ranks must agree, or some will save while others keep going. stop = all_reduce_max(1 if preempt_requested.is_set() else 0) if stop or step % ckpt_every == 0: save_checkpoint(step, model, optimizer, scheduler, loader.state(), rng_state()) if stop: wait_for_commit_marker(step) # durable before we exit report_status("PREEMPTED", step) sys.exit(0) ``` The detail that matters is **collective agreement**: a signal may reach one node before another, so ranks agree on stopping at a step boundary through a tiny all-reduce. The rescheduled job loads the newest committed checkpoint, possibly on different nodes. FSDP checkpoints use a format that can be resharded on load, so a 32-GPU job can resume on 16 GPUs. ### Detecting the Failures That Do Not Crash - **Hangs.** A rank stuck in a collective stalls everyone. A watchdog restarts the job if the step counter stops advancing for several minutes. - **Stragglers.** Per-rank step times identify consistently slow nodes to cordon. - **Loss spikes and NaNs.** The trainer skips a non-finite step; a run of them rolls back to the last checkpoint and fails the job with a clear message rather than burning hours. **First version:** single-node jobs only (LoRA up to 70B fits), DP or TP within the node, LoRA-sized checkpoints every few minutes, restart on failure. **Later:** multi-node FSDP for full fine-tunes, asynchronous two-tier checkpoints, resharding on resume and a preemptible pool. ## Deep-Dive: LoRA Adapters and Multi-Adapter Serving ![Serving many adapters on shared base models: a request for a tenant's fine-tuned model reaches the inference gateway, an adapter-aware router resolves the name through the adapter registry and sends the request to a replica that already has that adapter warm, spilling to another replica when needed; each replica keeps hot adapters in GPU slots, backed by a host RAM cache that fetches adapter weights from object storage on a miss.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-llm-fine-tuning-platform/05-multi-adapter-serving.png) ### Why LoRA Is the Default Output **LoRA** (low-rank adaptation) freezes the base weights and learns, for each chosen weight matrix W, a small update ΔW = B·A, where A and B have a low inner rank r (often 8 to 64). At inference, the layer computes: ```text h = W x + (alpha / r) * B (A x) W: d_out x d_in (frozen, shared by every tenant) A: r x d_in (per adapter) B: d_out x r (per adapter) ``` Training needs far less optimizer memory and checkpoints are tiny. **For serving, it is the whole point:** a fine-tuned "model" is an 80 MB file riding on a base model already loaded for everyone, instead of 10,000 separate deployments idle most of the day. Customers who need a full fine-tune get dedicated replicas and pay for them. (QLoRA, which trains against a 4-bit base, is a cheaper training recipe, but I evaluate the adapter on the production base precision, since behaviour can shift between quantisations.) The base-model serving engine (continuous batching, paged KV cache, streaming, cache-aware routing) is covered in [Designing an LLM Inference Platform](/blog/system-design-llm-inference-platform). Here I cover what adapters add. ### The Adapter Registry The registry is a small, strongly consistent table, and it is the contract between training and serving: ```sql CREATE TABLE adapter ( adapter_id TEXT PRIMARY KEY, -- content hash of weights + config tenant_id TEXT NOT NULL, base_model TEXT NOT NULL, -- e.g. 'base-8b@2026-07' rank INT NOT NULL, target_modules TEXT[] NOT NULL, artifact_uri TEXT NOT NULL, training_job_id TEXT NOT NULL REFERENCES training_job(id), gate_status TEXT NOT NULL, -- PENDING | PASSED | FAILED | OVERRIDDEN created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE model_alias ( -- what callers actually use tenant_id TEXT NOT NULL, name TEXT NOT NULL, -- e.g. 'ft:acme-support' version INT NOT NULL, adapter_id TEXT NOT NULL REFERENCES adapter(adapter_id), traffic_pct INT NOT NULL CHECK (traffic_pct BETWEEN 0 AND 100), PRIMARY KEY (tenant_id, name, version) ); ``` Two choices matter. **Adapters are immutable and content-addressed, and callers use aliases**, so promotion and rollback change `model_alias`, never weights. And every adapter records its **exact base model version**, because it is only valid on that base; a base upgrade means re-validating every adapter on it, and the registry lists them. ### Hot-Loading and the Adapter Cache Hierarchy I expect adapter popularity to be very skewed: a few hundred carry most traffic, thousands are called a few times a day. So adapter weights live in a three-level cache: 1. **GPU slots.** A fixed reservation per replica, for example 64 slots of the largest supported rank. 2. **Host RAM.** Say 1,000 adapters (about 80 GB). Host to GPU over PCIe takes milliseconds. 3. **Object storage.** The source of truth. A cold fetch takes hundreds of milliseconds to seconds. Eviction is LRU with a frequency bias, so one rare tenant's burst does not flush popular adapters. Slot memory comes out of the KV cache budget: more slots mean fewer cold loads but fewer concurrent sequences. ### Batching Requests for Different Adapters Together One batch per adapter would make batches tiny and waste the shared base. Instead I batch **different adapters in the same forward pass**: - The base computation `W x` is identical for every request, so it runs once for the whole batch at full efficiency. - The adapter term `B (A x)` differs per request. Specialised kernels gather each request's A and B and compute these small products for the whole mixed batch in one launch. The Punica work introduced a kernel for this (segmented gather matrix-vector multiplication), and S-LoRA extended the idea with unified paging of adapter weights and KV cache so thousands of adapters can share one server. Engines such as vLLM support multi-LoRA batching in this style. Mixed batches are somewhat slower per step than base-only ones, and the overhead grows with distinct adapters and rank, so I cap both per pool. Very high-traffic tenants can get a **merged** model (B·A folded into W) on dedicated replicas, removing the overhead but no longer sharing the base. ### Adapter-Aware Routing The router resolves the alias (applying rollout traffic splits), then prefers replicas with the adapter in a GPU slot, then in host RAM, then the least-loaded replica. As with prefix-cache routing, pure affinity overloads the replica holding a popular adapter, so hot adapters are replicated across several replicas past a traffic threshold. ### Failure Modes - **Cold-start storms.** Promoting an alias pre-warms the adapter on a few replicas before the first burst arrives. - **Wrong base.** Replicas check the adapter's base version on load and refuse mismatches. - **Tenant isolation.** A tenant can only resolve its own aliases, and adapter weights are encrypted at rest with per-tenant keys. **First version:** registry plus aliases, host and GPU caches, multi-adapter batching from the serving engine, simple affinity routing. **Later:** popularity-based replication, pre-warming on promotion, merged models for the heaviest tenants. ## Deep-Dive: Evaluation Gates and Safe Rollout ![The gate between training and production: a candidate adapter is scored by held-out task evals, a base capability regression suite and safety evals, and a gate service compares results with stored baselines and records them; a passing candidate goes to shadow traffic, then a canary by percentage watched by online metrics, and is finally promoted to the production alias.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-llm-fine-tuning-platform/06-eval-gate-rollout.png) ### Why the Gate Belongs in the Pipeline Falling training loss says the model fits its data. It does not say the model is better at the task, has not forgotten general skills, or still refuses what it should; fine-tuning can erode safety behaviour even on harmless-looking data. So a job is finished when the gate has a verdict, not when training ends. Eval datasets, LLM-as-judge scorers and production quality tracking are their own system, covered in [Designing an LLM Evaluation and Observability Platform](/blog/system-design-llm-eval-observability). Here I focus on the gate: what runs on every candidate, what blocks deployment, and how a passing candidate reaches traffic. ### What the Gate Runs Evals run on the eval pool with the production serving engine and base precision, so I evaluate what I will ship: 1. **Held-out task evals.** Customer data held back at ingestion (deduplicated against training) or their own eval file, scored with task metrics: field accuracy, code pass rate, or a calibrated judge. The candidate is compared with **the current production version** of the alias, or the base model for a first version. 2. **Base capability regression suite.** Fixed general benchmarks. The question is not whether the candidate is good at them but whether it got noticeably **worse** than the base, the signature of overfitting or too high a learning rate. 3. **Safety evals.** Refusals on a harmful-request set and over-refusal on a benign set, with platform-set thresholds. 4. **Format checks.** JSON parse rate where the use case needs it, and output length against the training data. ### Thresholds and Decisions Thresholds are relative to a baseline and evaluated on a confidence interval, because a few hundred held-out examples are noisy: ```yaml # Gate policy for one alias (platform defaults merged with tenant overrides) alias: ft:acme-support baseline: production # or "base" for a first version checks: - name: heldout_task_score must: candidate_lower_ci >= baseline - 0.01 - name: json_parse_rate must: candidate >= 0.98 - name: base_capability_suite must: drop_vs_base <= 0.03 # per benchmark, platform default - name: harmful_refusal_rate must: candidate >= base - 0.01 overridable: false # tenants cannot waive safety - name: benign_over_refusal must: candidate <= base + 0.02 on_fail: block override: requires_reason_and_owner ``` Scores and the verdict are stored against the adapter id, so "why was this deployed?" has an answer months later. A tenant can override a failed quality check with a recorded reason, but not a safety check. ### Safe Rollout After the Gate Held-out data never matches production exactly, so the rollout continues through the alias table: 1. **Shadow.** Mirror sampled live requests to the candidate, discard its outputs, and compare length, format validity, refusal rate and latency with production. 2. **Canary.** Set `traffic_pct` to 5%, 25%, then 50%, holding each step, keyed by a hash of the session id so a conversation does not flip versions mid-way. 3. **Promote.** The old version stays warm for a while, so rollback is an alias update that takes seconds. Hard signals (errors, parse failures, refusal spikes) roll back automatically during canary. Softer quality signals go to a human. ### Failure Modes - **Contaminated held-out sets** make the gate pass everything. Cross-set dedup at ingestion is the fix. - **A noisy gate** gets overridden by habit. Confidence intervals and per-check explanations keep it credible. - **Base model upgrades** re-gate every adapter on the old base as a bulk job through the same gate and rollout, with tenants notified first. **First version:** held-out task score against the base model, the safety suite, and manual promotion by changing the alias. **Later:** confidence-interval thresholds, comparison with the current production version, automated shadow and canary with rollback. ## Conclusion: Bringing It All Together A fine-tuning platform looks like a job queue on the outside and like a careful data and release pipeline on the inside. The GPUs are the expensive part, but most of the risk sits on either side of them: data that should never have been trained on, and models that should never have been shipped. The pieces line up with familiar systems ideas: - Dataset versions are immutable, content-addressed artifacts, like build outputs. - Gang scheduling and checkpoints are batch-computing basics applied to synchronous jobs that fail as a unit. - Adapters are small pages of state over a shared base, cached and batched like memory pages. - The eval gate and alias rollout are a deployment pipeline with a quality check in the middle. If I had to summarise the design in four decisions: 1. **Validate, dedup and version data before a GPU is touched.** Cheap checks up front save expensive training runs, and immutable versions make every model traceable. 2. **Use the least parallelism that fits, and checkpoint for the failures you will certainly have.** Collective agreement on preemption and commit markers on checkpoints are what make resumption safe. 3. **Make LoRA the default output and serve many adapters on one base.** It turns thousands of tenant models into a cache problem instead of a fleet problem. 4. **Put an eval gate between training and the alias, then roll out with shadow and canary.** A fine-tune is not done when the loss drops. It is done when it has earned its traffic. Related reading: [Designing an LLM Inference Platform](/blog/system-design-llm-inference-platform), [Designing an LLM Evaluation and Observability Platform](/blog/system-design-llm-eval-observability), [Designing a Distributed Job Scheduler](/blog/system-design-distributed-scheduler), and [Designing a Recommendation Engine](/blog/system-design-recommendation-engine).