Designing an LLM Guardrails and Safety Layer

When a company ships its first LLM feature, safety usually lives inside that feature: a regex for card numbers, a call to a moderation endpoint, a line in the system prompt asking the model to behave. By the tenth feature, that approach has fallen apart. Every team has rebuilt the same checks with different thresholds, nobody can say which products scan retrieved documents for injected instructions, and when a new attack pattern appears on a Friday evening there is no single place to turn on a fix. Safety has to become shared infrastructure, the way authentication and rate limiting did.

That is the system I want to design here: a guardrails service that every LLM product calls, or that runs inline in the LLM gateway, to check what goes into a model and what comes out. It looks for prompt injection and jailbreaks, personal data, toxic or disallowed content, claims the sources do not support, and leaks of secrets or system prompts. It applies a different policy per product and tenant, inside a latency budget tight enough that teams have no reason to route around it.

This is defensive engineering, so I describe attack categories at the level a designer needs, not as recipes. I cover scoping, requirements with capacity estimates, the architecture, and four deep-dives:

• Input checks for prompt injection, jailbreaks and PII
• Output checks for toxicity, grounding and data leakage, including streamed output
• The policy engine and the latency budget
• Red-teaming, monitoring and incident response

Scoping the Problem and Clarifying Assumptions

I am building the runtime safety layer: the service that inspects LLM traffic and returns a verdict. It sits between products and models, whether the model is self-hosted or an external provider.

What I'm including:

• Checks on user messages, retrieved documents and tool outputs before they reach the model
• Checks on model responses, complete or streamed, before they reach a user or a tool
• Per-product and per-tenant policies as versioned configuration
• Redaction and reversible tokenisation of personal data
• The feedback loop: verdict logs, human review, red-team suites and fast rollout

What I'm NOT building here:

• Model alignment or safety training. The model's own refusals help, but the guardrail treats the model as untrusted.
• The LLM gateway itself (auth, quotas, routing, provider fallback). The guardrail plugs into it.
• The agent runtime and its tool sandbox. I explain why least-privilege tools matter, but running tools safely is the agent platform's job.
• Classic web security (WAF, DDoS, bot detection), which I assume sits in front of everything.

Key assumptions:

• About 40 LLM products: chat assistants, support bots, RAG search, coding tools and agents that call internal APIs
• Peak traffic of about 40,000 LLM calls per second, most of them streaming
• Most calls go through a central gateway, but some teams call the guardrail directly, so it must work in both modes
• Product teams own their policies. A central safety team owns the shared detectors, the baseline and the incident process.

How this differs from a classic content filter: comment moderation checks one piece of text from one author, often asynchronously, and can take seconds. Here the prompt is assembled from sources with different trust levels, some of which try to give the model instructions. The output arrives token by token while the user watches. And in an agent, an output is not just text but a tool call that might send an email or change a record. The guardrail is part content filter, part data-loss prevention, part security boundary, and it must run in tens of milliseconds.

Functional and Non-Functional Requirements

Functional Requirements

1. Check input: given the prompt with each segment labelled by source, return allow, block or allow-with-redactions, with reasons.
2. Check output: given a response or a window of a stream, return release, redact, block or regenerate.
3. Redact and restore: replace personal data with placeholders before the model sees it, and restore real values in the output where policy allows.
4. Apply policy: resolve which checks, thresholds and actions apply for a product, tenant and endpoint.
5. Log verdicts: emit a structured event per decision, without storing raw sensitive text by default.
6. Change fast: push a new rule, threshold or kill switch to the whole fleet within minutes.

Non-Functional Requirements

Latency shapes the whole design. If the guardrail adds 300 ms to every call, teams will find reasons to skip it, and a skipped guardrail protects nothing.

Back-of-the-Envelope Capacity Estimation

These are planning assumptions, not measurements. The method is what matters: count the tokens that actually need scanning, then divide by benchmarked throughput per accelerator.

Traffic: 40,000 calls per second at peak. I assume an average assembled input of 2,000 tokens (system prompt, history, retrieved chunks, new user turn) and an average output of 400 tokens.

Input tokens to scan:

• Naively: 40,000 × 2,000 = 80 million tokens per second. Too much to classify on every call.
• Most of it does not need rescanning. System prompts and tool definitions are trusted and static, so I skip them by hash. History was scanned on earlier turns, so I cache a verdict per segment hash. I assume only about 25% of input tokens are new and untrusted per call.
• That leaves about 20 million tokens per second.

Output tokens to scan: 40,000 × 400 = 16 million tokens per second. Stream windows overlap slightly, so each token is scanned about 1.3 times: about 21 million tokens per second.

Classifier fleet: suppose a small encoder classifier (around a hundred million parameters) sustains about 200,000 tokens per second per GPU at my batch sizes. I would benchmark this. Then input needs 20M / 200K ≈ 100 GPUs and output about 105 GPUs. Several classifier heads share one encoder pass, so I do not multiply by the number of checks. With 50% headroom and regional redundancy, I plan for 300 to 400 small GPUs, modest next to the generation fleet they protect.

LLM judge: if 2% of calls escalate, that is 800 calls per second at perhaps 2,500 tokens each. A real but bounded pool, and much of it can run asynchronously.

Verdict events: about 3 decisions per call gives 120,000 events per second. At about 1 KB of metadata each, that is roughly 10 TB per day before compression.

High-Level Architecture and Core Components

Component Walkthrough

LLM gateway. The single path from products to models. It identifies the product and tenant and calls the guardrail twice: after assembling the prompt and while the response streams back. Putting the guardrail here means teams get it by default instead of by remembering.

Guardrail service. Stateless. It resolves the policy, runs detectors in parallel, combines results and returns a verdict. Rules and parsing run on CPU; model-based checks call GPU pools. It is deployed in cells so one bad policy or hot tenant cannot take down every product.

Policy registry. Versioned policies per product, tenant and endpoint, plus global rules owned by the safety team. Policies are compiled into execution plans and pushed to every instance, so a check never waits on a config lookup.

Rules and PII detectors. Deterministic checks in a few milliseconds: secret and identifier patterns with checksum validation, deny lists, structural checks on tool calls, and entity detection for personal data.

Small classifiers. Encoder models fine-tuned for injection, jailbreak, toxicity, self-harm and topic. Open safety models in the Llama Guard style are a reasonable start for content categories, but I expect to fine-tune on my own traffic, because what counts as allowed depends on the product.

LLM judge pool. A mid-size model with a rubric, for cases the classifiers are unsure about and for checks that need reasoning, such as whether a claim is supported. Slow and costly, so it sees a small fraction of traffic.

Verdict cache. Results keyed by segment hash and policy version. Popular retrieved chunks and earlier turns hit it constantly.

Verdict events. Every decision goes onto a stream with the policy version, checks run, scores, latency and action. Dashboards, alerts, review sampling and offline evaluation read from it.

Request Flow

1. A support bot assembles a prompt: system prompt, three retrieved help articles, the conversation and the customer's new message. Each segment carries a source label.
2. The guardrail resolves the support bot's policy, skips trusted and cached segments, and runs PII, injection and topic checks on the rest in parallel.
3. The message contains a card number. The PII detector replaces it with a placeholder. Verdict: allow-with-redactions.
4. The gateway sends the redacted prompt to the model and starts receiving the stream.
5. Output goes to the guardrail in windows, each checked for disallowed content, leaks and unsupported claims, then released.
6. At the end, the guardrail emits its verdict event.

Scaling It Out

• Co-locate with the gateway. Same region, ideally the same zone. A cross-region hop would eat the budget.
• Cells, not one fleet. Each cell serves a fixed set of products, so a regex that backtracks badly or a tenant sending huge documents affects one cell.
• Separate GPU pools per tier. Classifiers run on cheap accelerators at high batch sizes. Judges get their own pool, so an escalation spike does not slow inline checks.
• Policy as a pushed snapshot. Instances hold the full compiled policy set in memory. If the registry is down, they keep serving the last good snapshot.
• Async path. Checks that only feed monitoring or retroactive action go through a queue and never touch user latency.

Deep-Dive: Input Checks: Prompt Injection, Jailbreaks, and PII

Two Kinds of Injection

I separate threats by where the hostile text comes from:

• Direct injection and jailbreaks. The user is the attacker, trying to make the model ignore its instructions, reveal its system prompt or produce forbidden content. Common families include role-play framing, false claims of authority, splitting a request across turns, and obfuscating the forbidden part so filters miss it.
• Indirect injection. The user may be innocent. Hostile text arrives through content the model reads on their behalf: a web page, an email, a shared document, a tool's output. To the model it is all tokens, and it cannot reliably tell data from instructions.

Indirect injection is the bigger risk for a company because it scales. One poisoned document in a shared drive reaches every user whose assistant retrieves it, and in an agent it can turn into actions. The OWASP Top 10 for LLM Applications lists prompt injection first for this reason.

Trust Boundaries Come Before Detection

The first step is not detection but labelling. The gateway sends the prompt as segments with a source and trust level, not one flat string:

{
  "policy_ref": "support-bot/prod",
  "tenant": "acme",
  "segments": [
    {"id": "s1", "role": "system",    "trust": "trusted",   "hash": "9f2c...", "text": null},
    {"id": "s2", "role": "retrieved", "trust": "untrusted", "source": "help-centre/4411", "text": "..."},
    {"id": "s3", "role": "tool",      "trust": "untrusted", "source": "orders-api",       "text": "..."},
    {"id": "s4", "role": "user",      "trust": "user",      "text": "..."}
  ]
}

Trusted segments are referenced by hash and never rescanned. Untrusted segments get the strictest checks. The labels also reach the model: the gateway wraps untrusted content in clear delimiters, and the trusted system prompt says that text inside them is data, never instructions. Some teams also transform untrusted text, for example by marking every line, so the boundary is hard to forge from inside. None of this makes the model immune, but it lowers the success rate and gives the classifiers context.

Normalise, Then Detect in Layers

Attackers rely on the gap between what a filter sees and what the model reads. So I normalise first: Unicode normalisation, stripping invisible characters, decoding common encodings that hide text, removing markup. Checks run on the normalised form; the model gets the original.

Then three layers, cheapest first:

1. Rules and signatures (under 2 ms). Patterns from past incidents, safety-team deny lists, and structural oddities such as a retrieved document containing a fake chat-turn boundary. Precise but brittle, and the fastest thing to update during an incident.
2. Small classifiers (5 to 15 ms). An encoder scores each untrusted segment for injection and jailbreak likelihood, trained on red-team data, labelled production samples and public datasets, and calibrated so a 0.9 threshold means the same thing across releases.
3. LLM judge (200 ms or more, rare). Only for the gray zone between two thresholds, on policies that pay for it.

The combiner weighs results by trust level. The same phrase means one thing in a user message (asking about a topic) and another in a retrieved document (trying to issue commands). For untrusted content I prefer dropping the suspicious segment to blocking the request: the model is told a source was withheld and answers from the rest.

Why Detection Alone Is Not Enough

Some injections will get through. Language has no reliable boundary between data and instruction, and attackers can iterate against any classifier. So the rest of the system must limit the damage:

• Least-privilege tools. An assistant that summarises email should not hold a send-email tool, and tools act with the user's permissions, never the service's.
• Taint tracking. Once untrusted content enters the context, the session is marked tainted, and high-risk tools (send, delete, pay, share externally) require user confirmation.
• Tool calls are outputs. Their arguments are validated against schema and policy before execution: no outbound URLs off the allowlist, no recipients outside the user's organisation.
• No secrets in prompts. Credentials live in the tool layer, so there is nothing in the context to extract.

PII: Detect, Then Redact or Tokenise

Detection combines patterns with checksums (card numbers validated with the Luhn check, national ID formats, IBANs), named-entity recognition for names and addresses, and context rules. Microsoft Presidio follows this recogniser-plus-context design and is a reasonable base. The action is a policy choice:

Tokenisation replaces jane.doe@acme.com with <EMAIL_1> and stores the mapping in a vault keyed by request id, encrypted per tenant and expiring with the session. The model reasons over the placeholder, and the guardrail restores the real value in the output, only for the user who supplied it. Personal data stays out of external providers and their logs, at a small cost when the model genuinely needs the value.

First vs later: first, trust labelling, normalisation, PII tokenisation and one injection classifier on untrusted segments. Later, the judge tier, per-product classifiers and taint tracking wired into the agent runtime.

Deep-Dive: Output Checks: Toxicity, Grounding, and Data Leakage

Grounding Checks

Moderation for toxicity, self-harm, violence and off-limits product topics (a banking bot giving medical advice) uses the same classifier stack as input, with output thresholds per product. Grounding is different. It splits the answer into claims and asks whether each cited passage supports it, in tiers:

1. Citation validity (always on). Every citation must point to a source that was in the context, and the passage must overlap meaningfully with the claim. This catches invented citations.
2. Entailment classifier (default for RAG). A small cross-encoder trained for natural language inference labels each claim-passage pair as supported, contradicted or unrelated.
3. LLM judge (high-stakes products). For legal, medical or financial answers.

Grounding scores are noisy, so I rarely block on them. Usually I strip the unsupported sentence, append a notice that parts could not be verified, or flag it for the product team's quality metrics. Making the model cite well in the first place is covered in the AI answer engine post; the guardrail is the independent check afterwards.

Leakage Detection

• Secrets. The pattern-and-entropy detectors used for code repositories: key formats, private key headers, connection strings, high-entropy strings next to words like "token".
• System prompt. At compile time I fingerprint each product's system prompt as overlapping n-grams. High overlap in an output window means the model is reciting its instructions. I also embed a random canary string in each system prompt; it has no meaning, so seeing it in output is an unambiguous leak.
• Cross-tenant data. The guardrail knows the request's tenant. An identifier belonging to another tenant in the output is blocked and pages someone, because it usually means a retrieval or caching bug upstream.

Checking a Stream Without Killing Latency

Waiting for the full answer makes streaming pointless; checking each token alone gives classifiers no context and multiplies cost. The compromise is a sliding window with hold-back. The gateway buffers 48 to 64 tokens, ending on a sentence boundary when one is close, and checks the window together with the last 16 tokens of the previous one so a phrase split across the boundary is seen whole. The window is released only after it passes. The first window is smaller, about 16 tokens, to keep time to first visible token low. Users see short bursts instead of single tokens. I expect that to be fine at reading speed, but I would confirm it with a user test per product.

async def guarded_stream(model_stream, policy, ctx):
    window, carry = [], []
    async for token in model_stream:
        window.append(token)
        if len(window) < policy.window_size(first=ctx.released == 0):
            continue
        if not ends_sentence(window) and len(window) < policy.max_window:
            continue
        verdict = await check_output(carry + window, policy, ctx)  # parallel checks inside
        if verdict.action == "release":
            yield detokenise(window, ctx)        # restore PII placeholders for this user
        elif verdict.action == "redact":
            yield detokenise(verdict.redacted, ctx)
        else:                                    # block or regenerate
            await model_stream.cancel()          # stop paying for tokens we will discard
            yield policy.violation_message(verdict)
            return
        ctx.released += len(window)
        carry, window = window[-16:], []
    if window:                                   # flush the tail
        verdict = await check_output(carry + window, policy, ctx, final=True)
        yield detokenise(window, ctx) if verdict.ok else policy.violation_message(verdict)

Whole-answer checks run only on the final flush. If they fail after the user has read the text, the options are a notice or a retraction in the UI, and each product's policy makes that trade explicit.

What to Do on a Violation

On a mid-stream block I always cancel generation, or the provider keeps producing tokens nobody sees and the company pays for them.

First vs later: first, moderation on every window, secret scanning and the canary check. They are cheap and catch the incidents that make the news. Later, entailment grounding and cross-tenant checks, which need product-specific data.

Deep-Dive: The Policy Engine and the Latency Budget

Policies Are Configuration, Not Code

Each product declares what it needs, and the safety team sets floors it cannot go below:

policy: support-bot/prod
version: 42
extends: company-baseline        # global floors: secrets, cross-tenant data, severe content
input:
  injection: { segments: [retrieved, tool], threshold: 0.85, action: drop_segment }
  jailbreak: { segments: [user], threshold: 0.9, action: block, escalate_band: [0.6, 0.9] }
  pii: { card_number: block, email: tokenise, phone: tokenise }
output:
  moderation: { categories: [hate, self_harm, sexual, violence], threshold: 0.8, action: block }
  topics:     { deny: [medical_advice, legal_advice], action: regenerate_once }
  grounding:  { mode: async, action: flag }
  leakage:    { system_prompt: block, secrets: redact, cross_tenant: block_and_page }
streaming: { window_tokens: 48, first_window_tokens: 16 }
failure_mode:
  default: fail_open
  overrides: { pii: fail_closed, leakage.cross_tenant: fail_closed }
latency_budget_ms: { input_p99: 40, window_p99: 30 }

The extends chain is resolved at compile time, producing an execution plan: which detectors run on which segment types, at what thresholds, in which tier. The compiler rejects any plan that lowers a baseline threshold, and the plan is tested against the red-team suite before it is pushed. Tenants can only tighten within limits the product allows, such as stricter PII rules for a healthcare customer.

Inline vs Async

I sort checks by what happens if they are late. Inline checks protect against harm that cannot be undone once a user or tool sees the content: PII before an external call, injection in agents, leakage, severe content. Async checks feed metrics or retroactive action: whole-answer grounding for most products, judge audits of samples, low-severity flags. Moving a check to async is often the right answer to "this is too slow", but never for checks whose whole value is stopping content before it arrives.

Parallel Checks with Early Exit

All inline checks run concurrently. When any returns a high-severity block, the planner cancels the rest and returns, because a toxicity score is irrelevant once a secret has been found. When the deadline passes, unfinished checks resolve to their fail mode.

async def run_plan(plan, segments, deadline_ms):
    loop_time = asyncio.get_running_loop().time
    tasks = {asyncio.create_task(c.run(segments)): c for c in plan.inline_checks}
    results, deadline = [], loop_time() + deadline_ms / 1000
    while tasks:
        done, _ = await asyncio.wait(tasks, timeout=max(0, deadline - loop_time()),
                                     return_when=asyncio.FIRST_COMPLETED)
        if not done:                                # budget exhausted
            for t, check in tasks.items():
                t.cancel()
                results.append(check.on_timeout())  # applies fail_open / fail_closed
            break
        for t in done:
            tasks.pop(t)
            r = t.result()
            results.append(r)
            if r.severity == "high" and r.action.startswith("block"):
                for other in tasks:
                    other.cancel()
                return combine(results)             # early exit
    return combine(results)

Model Tiering and Shared Encoders

Each category is a cascade of rules, then a classifier, then a judge, and only uncertain cases move up. I track the escalation rate per check as a cost metric; if it creeps up, traffic has shifted or the classifier needs retraining. I also share encoder passes: one encoder embeds the text and lightweight heads score injection, toxicity and topic, dividing GPU cost roughly by the number of heads at the price of retraining them together.

Fail-Open vs Fail-Closed

When a check times out or its backend is down, the policy decides. Fail-closed protects against irreversible harm: PII to external providers, cross-tenant leakage, high-risk tool calls. The request fails with a clear error and retry hint. Fail-open suits checks where a short gap is acceptable, such as low-severity moderation. The request proceeds and the verdict records skipped: timeout for async re-checking. I alert on the fail-open rate: a guardrail silently failing open 5% of the time has a 5% hole in it.

Caching Verdicts

The key is hash(normalised_text) + check_id + model_version + threshold_version, so a new classifier or threshold invalidates old verdicts automatically. Popular retrieved chunks, earlier turns and common short messages hit it. I never share entries across tenants for tenant-dependent checks, and I keep TTLs short (minutes to hours) so a newly discovered attack is not hidden behind an old "allow".

The p99 Budget

The budget I publish for the inline input path. Parallel stages overlap, so the total is the critical path, not the sum.

The batching window is the main lever: longer windows fill bigger batches and cut cost but add queueing delay. I cap it and autoscale on queue time, not GPU utilisation.

Deep-Dive: Red-Teaming, Monitoring, and Incident Response

Adversarial Test Suites in CI

Every policy change, classifier release and threshold tweak passes a test gate with three parts:

• Attack sets by category: jailbreak families, indirect injection in documents and tool outputs, obfuscation styles, leakage probes, PII formats per country. They come from internal red-teaming, past incidents and public research, and are access-controlled because they are sensitive.
• Benign look-alikes that must pass: engineers asking how injection works, a novel with a violent scene, a support ticket full of legitimate account numbers. These keep false positives honest.
• Product golden sets of consented, redacted real traffic, so a change cannot quietly break one team's main use case.

The gate compares the candidate with production per category and blocks on any drop in detection or rise in false positives beyond a margin. A classifier that is better overall but worse on one country's ID format is a regression for those customers. I also run automated red-teaming, where an attacker model generates variations of known categories against a staging guardrail. Successful variants join the attack set after review, so the suite does not fill with near-duplicates.

Tracking False Positives and False Negatives

False negatives cause incidents. False positives cause teams to lower thresholds or switch checks off, which causes incidents later. So I measure both. For false positives, I sample blocked and redacted requests for review and, where it makes sense, give users a "blocked incorrectly" signal; a jump in appeals on one product is often the first sign of a bad threshold. For false negatives, I sample allowed traffic weighted towards borderline scores, audit it with a stronger judge and humans, and count misses found by other means. Reviewers work in a queue with redacted views, strict access control and protections for people viewing harmful content (blur by default, rotation, exposure limits). Their labels feed retraining and the test sets.

Dashboards and Alerts

-- Block and fail-open rates per product and check over the last hour
SELECT product, check_id,
       avg(CASE WHEN action LIKE 'block%' THEN 1.0 ELSE 0 END)   AS block_rate,
       avg(CASE WHEN skipped = 'timeout' THEN 1.0 ELSE 0 END)    AS fail_open_rate,
       approx_percentile(latency_ms, 0.99)                       AS p99_ms
FROM verdict_events
WHERE event_time >= now() - INTERVAL '1' HOUR
GROUP BY product, check_id;

I alert on change against a baseline (the same hour last week), not absolute levels, because normal block rates differ hugely across products. A jump in injection blocks on one product often means a campaign. A drop to zero usually means a broken check, not peace. I also alert on fail-open rate, per-stage p99 and judge escalation rate. Any canary or cross-tenant leak pages immediately.

Kill Switches and Fast Rollout

Normal changes roll out in stages: shadow mode (evaluate and log without enforcing), one cell, one region, everywhere, with automatic rollback if block rates or latency leave expected bounds. Shadow mode shows what a rule would have blocked on real traffic before it blocks anything.

Incidents need a faster path that is still safe:

• Emergency rules: a pre-approved type (signature, deny pattern, tighter threshold for one category) that skips the full suite but still runs the benign set as a smoke test, and propagates in under 30 seconds. They expire automatically so they cannot pile up as unreviewed debt.
• Capability downgrades: force confirmation on all tool calls for a product, or stop retrieval from one source, without turning the product off.
• Product kill switch: disable an LLM feature for a product or tenant and serve a static fallback.

Each switch is a registry flag with an owner and an audit log, and each is exercised in game days. A kill switch nobody has tried is a guess.

Incident Playbook

1. Contain with the narrowest effective switch: an emergency rule, then a capability downgrade, with the kill switch as a last resort.
2. Scope by querying verdict events and gateway logs for matching traffic before containment. This is why events carry segment hashes and policy versions.
3. Notify: data exposure triggers the privacy and legal process, and product owners and affected tenants are told what happened.
4. Fix properly: add the attack to the suite, retrain or adjust, and replace the emergency rule with a tested one.
5. Review whether detection, defence in depth or process failed, and strengthen the weakest layer, not only the one that was bypassed.

First vs later: first, verdict events, block-rate and fail-open dashboards, shadow mode and a per-product kill switch; without them you can neither see nor stop anything. Later, automated red-teaming, the full review program and appeal flows.

Conclusion: Bringing It All Together

From far away a guardrail layer looks like a content filter. Up close it is a security boundary, a data-loss prevention system and a latency-critical service at once. The choices that matter come from treating it as infrastructure: one service, one policy model, called by default from the gateway, fast enough that nobody has a reason to bypass it.

If I had to sum up the design in five decisions:

1. Label trust before detecting anything. Untrusted content is fenced and scanned hardest; trusted static content is skipped by hash.
2. Layer detectors and escalate only uncertainty. Rules, then small classifiers, then a judge, with calibrated thresholds so the expensive tier sees a small slice.
3. Check streams in windows with hold-back, and cancel generation on a block.
4. Make policy versioned configuration with a fail mode per check, compiled ahead of time, pushed as snapshots and rolled out through shadow mode.
5. Assume some attacks get through. Least-privilege tools, taint tracking, kill switches and a loop from verdicts to review to test suites limit the damage when detection misses.

Related reading: Designing an LLM Gateway, Designing an AI Agent Platform, Designing an Enterprise RAG Assistant, and Designing an LLM Inference Platform.