# Designing an MCP Tool Platform: Registry, Gateway, Sandboxes and Credentials at Scale ## Blog Details - **Author**: Naveen R. - **Date**: November 3, 2026 - **Tags**: MCP, AI Agents, System Design, OAuth, Platform Architecture - **Read Time**: 20 mins An agent is only as useful as the tools it can reach. The Model Context Protocol (MCP) gave the industry a common way to plug tools into models: a client inside the AI app talks JSON-RPC 2.0 to a server that exposes tools, resources and prompts. That solved the "every integration is custom" problem and created a new one: a company with fifty agents and a few hundred MCP servers now has a few hundred pieces of third-party code, a few hundred OAuth integrations, and thousands of tool descriptions competing for the model's context window. The platform I design here sits in the middle. Agents connect to one endpoint. Behind it, the platform hosts or brokers thousands of MCP servers, keeps a versioned registry of every tool, picks the few tools that matter for each request, holds each user's OAuth tokens so no agent sees them, runs untrusted server code in isolation, and meters every call. In this post I walk through how I would build that from scratch. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives: - Tool registry and discovery - Auth and credentials - Sandboxed execution and multi-tenancy for hosted servers - Observability, quotas and rate limits ## Scoping the Problem and Clarifying Assumptions I am building the **tool side** of the agent stack: the registry, the gateway in front of many MCP servers, credential brokering, and the runtime that hosts server code. The agent loop (planning, memory, durable runs, approvals) is a separate system that calls mine; I covered it in [Designing an AI Agent Platform](/blog/system-design-ai-agent-platform). **What I'm including:** - A **public catalog** of MCP servers plus **private servers** each tenant publishes for its internal APIs - **Hosted servers** (packaged code, often written for stdio) that I run, and **remote servers** run by third parties that I reach over Streamable HTTP - A **gateway** that looks like one MCP server to clients and fans out to many upstreams - **Per-user OAuth** with a token vault and explicit consent - **Quotas, metering, traces and audit** per tool, user and tenant **What I'm NOT building here:** - The agent runtime and its memory. Clients bring their own loop. - LLM serving. Tool selection uses a small embedding model and reranker; the main model lives elsewhere. - Prompt-injection detection in tool outputs, which is covered in [Designing LLM Guardrails](/blog/system-design-llm-guardrails). **Key assumptions:** - Clients speak MCP, almost always over Streamable HTTP (IDE tools can use a small local stdio-to-HTTP bridge) - About 20,000 tenants, around 2,000 catalog servers and 30,000 private servers - A tenant connects 20 to 60 servers with 5 to 40 tools each, so **a tenant reaches hundreds to low thousands of tools** - Most calls are I/O-bound: wrap a SaaS API, wait 100 ms to a few seconds, return a few kilobytes **How this differs from a classic API gateway:** a classic gateway routes to services the company wrote and trusts, for callers who know which endpoint they want. Here the callers are models choosing tools from natural-language descriptions, so **the description is both routing logic and attack surface**. The upstream code is often written by strangers, and the credential on each call belongs to an end user, not the calling service. ## Functional and Non-Functional Requirements ### Functional Requirements 1. **Publish** an MCP server version (a package to host, or a remote URL); the platform introspects and records its tools, resources and prompts. 2. **Compose** tools from many servers into a **toolset** served at one gateway endpoint. 3. **Discover** the tools relevant to each request, filtered to what this user may use. 4. **Connect** a user's third-party account once via OAuth, without the client ever holding the token. 5. **Call** the right upstream with the right credential, with timeouts, size limits and cancellation. 6. **Govern** with quotas, audit logs, metering and admin policy per tenant. ### Non-Functional Requirements | Requirement | Target | Why | |-------------|--------|-----| | Gateway overhead | p50 under 15 ms, p99 under 50 ms added to a call | Agents chain many calls per task | | Discovery latency | p99 under 100 ms for top-k selection | It sits before the model's first token | | Availability | 99.95% for the gateway and registry reads | One outage breaks every agent at once | | Isolation | no cross-tenant access to code, memory, files or tokens | Hosted code is untrusted | | Credential safety | tokens never reach the client, the model, logs or server sandboxes at rest | Tokens are the crown jewels | | Fairness | one tenant or one runaway agent cannot exhaust a shared upstream | Shared OAuth apps share provider limits | I do not promise an SLO for upstream servers. A slow third-party server is not my outage, but **detecting it, containing it and telling the caller clearly is**. ### Back-of-the-Envelope Capacity Estimation These are planning assumptions, not measurements. The method: estimate call volume, then in-flight concurrency, then memory for hosted instances. **Tool calls:** - 500 million tool calls per day across all tenants - 500M / 86,400 s ≈ **5,800 calls per second on average**, about **17,000 at peak** (3x) **Gateway concurrency:** - With a mean call duration of 400 ms, in-flight calls at peak ≈ 17,000 × 0.4 ≈ **7,000 concurrent upstream calls** - If 1.5 million agent sessions are active at peak and a third keep a notification stream open, that is **500,000 mostly idle connections** on the edge tier **Discovery:** - One discovery per model turn with tools, about 150 million per day, so about **5,000 per second at peak** - If a tool definition (name, description, JSON Schema) averages 300 tokens, a tenant with 1,000 reachable tools would spend **300,000 tokens** just listing them. The top 20 cost about 6,000. **Hosted instances:** - I assume 40% of calls go to platform-hosted servers: about **6,800 calls per second at peak** - Instances are isolated per (tenant, server version) at minimum. If 60,000 such pairs are active at peak and each warm microVM uses 256 MB, that is about **15 TB of memory**, roughly 50 hosts with 384 GB each. Idle pairs are snapshotted and stopped, which keeps this bounded by active pairs, not by every tenant times every catalog server. **Token vault:** - 5 million connected accounts at about 2 KB each (encrypted refresh token plus metadata) is **10 GB**. The load is reads: the broker caches access tokens and refreshes about hourly per active grant. With 1 million grants active in a peak hour, that is about **280 refreshes per second**. **Audit and metering:** - 500M calls × about 1 KB of metadata ≈ **500 GB per day** of call events. Full payloads are stored only for tenants who opt in. ## High-Level Architecture and Core Components ![The tool platform request path: MCP clients send JSON-RPC over HTTP to an MCP gateway, which verifies the client token with the platform OAuth server, checks quotas, asks a tool selector backed by the tool registry which tools to expose, fetches an upstream credential from a credential broker backed by a token vault, and calls either sandboxed hosted MCP servers or remote MCP servers; every call emits an event to a stream feeding metrics, traces and audit.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-mcp-tool-platform/01-high-level-architecture.png) ### Component Walkthrough **MCP Gateway.** To the client it is **one MCP server**: it answers `initialize`, `tools/list`, `tools/call`, `resources/read` and `prompts/get`, and pushes notifications. Toward each upstream it is **an MCP client** with its own session, protocol version and capabilities. Tools are namespaced by server (`github.create_issue`, `jira.create_issue`) so names never collide. Session routing state lives in a shared cache, so pods stay stateless. **Platform OAuth server.** Issues the tokens clients use to call the gateway. This is a different leg from the user's third-party credentials, and I keep the two strictly apart. **Quota service.** Buckets and concurrency limits per tenant, user, tool and upstream. **Tool registry and selector.** The registry stores every server version and the exact, hashed definition of each tool. The selector picks the tools to list per request. Covered in the first deep-dive. **Credential broker and token vault.** The vault stores encrypted refresh tokens per (tenant, user, connection); the broker turns a call's identity into a short-lived upstream access token. Covered in the second deep-dive. **Hosted MCP servers.** Catalog or tenant packages, built into images and run in microVM sandboxes per tenant and server version. Covered in the third deep-dive. **Remote MCP servers.** Third-party servers I do not run but still treat as untrusted peers: strict timeouts, response size limits and a circuit breaker per server. **Call event stream.** One event per call, feeding metrics, traces, audit and billing. Covered in the fourth deep-dive. ### Request Flow 1. An agent opens a session: `initialize` to `https://mcp.example.com/t/acme/toolsets/support-agent` with a bearer token from the platform OAuth server. 2. The gateway verifies the token (signature, expiry, audience), resolves tenant, user and toolset, and returns capabilities including `tools` with list-change notifications. 3. Before a model turn, the client calls `tools/list`, optionally attaching the task text in request metadata. The selector returns 20 tools this user may call. 4. The model picks `github.create_issue`. On `tools/call`, the gateway validates arguments against the pinned schema, reserves quota, and gets this user's GitHub token from the broker. 5. The gateway forwards the call to this tenant's GitHub server sandbox; the token rides on the outgoing request and is never stored there. 6. The result returns to the client, and the gateway emits a call event. ### Scaling It Out ![The scaled-out platform: a global load balancer spreads MCP clients across two regions; each region runs stateless gateway pods that read tool definitions from a local registry replica, route stateful sessions to their own sandbox fleet with session affinity, and fetch credentials from a regional token vault.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-mcp-tool-platform/02-scalable-architecture.png) - **Stateless gateways, sticky sessions.** A Streamable HTTP server may assign an `Mcp-Session-Id` at initialisation and expect it on later requests. I keep a session record (client session id, upstream session ids, sandbox placement) in a regional cache, so any pod can serve any request while the hosted leg sticks to the same sandbox. - **Registry reads are local.** One writer region; read replicas per region plus an in-memory snapshot in every pod, refreshed on change events. A registry outage means "no new versions publish", not "no calls work". - **Data residency.** Token vaults are regional, so a European tenant's refresh tokens never leave the EU. - **Stable egress IPs** per region, for remote server operators who allowlist callers. ## Deep-Dive: Tool Registry and Discovery: Versioned Schemas and Picking the Right Tools per Request ![Registry and discovery: a server publisher pushes a version, and remote servers signal list changes, into an ingest worker that introspects the server and passes its tools through a schema and description scanner; approved versions land in a versioned registry, which feeds an indexing job and a hybrid tool index; at request time the tool selector filters by the toolset and the user's grants, searches the index with BM25 and vectors, and returns a top-k tool list.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-mcp-tool-platform/03-registry-and-discovery.png) ### What a Registry Entry Is The agent platform post already describes the per-tool record a runtime needs (schema plus hidden metadata like side effects and approval rules). At catalog scale the hard part is **change over time**: servers ship new versions, remote servers change their tool list silently, and the same tool name can mean different things a month apart. So every tool definition is immutable and addressed by a hash of its exact content: ```sql CREATE TABLE server_version ( server_id UUID REFERENCES mcp_server, version TEXT, -- '3.2.0' or an observed snapshot id for remote artifact TEXT, -- image digest (hosted) or URL (remote) protocol_ver TEXT, -- negotiated during introspection status TEXT, -- 'pending_review' | 'active' | 'deprecated' | 'blocked' PRIMARY KEY (server_id, version) ); CREATE TABLE tool_def ( def_hash BYTEA PRIMARY KEY, -- sha256 of canonical JSON (name, description, schemas, annotations) server_id UUID, version TEXT, name TEXT, definition JSONB, risk TEXT -- derived: 'read' | 'write' | 'destructive' ); CREATE TABLE toolset_pin ( toolset_id UUID, server_id UUID, version TEXT, -- pinned; upgrades are explicit allowed_tools TEXT[], -- NULL means all tools in that version PRIMARY KEY (toolset_id, server_id) ); ``` A toolset pins a **server version**, not a server. Upgrading is an explicit action with a diff, like bumping a library dependency. ### Ingestion: Introspect, Normalise, Scan When a version is published, an ingest worker starts it (or connects, for remote servers), runs the `initialize` handshake, and pages through `tools/list`, `resources/list` and `prompts/list` with the server's cursor. Then it: 1. **Canonicalises** each definition (sorted keys, normalised whitespace) and hashes it. 2. **Validates the input schema** and rejects shapes I cannot enforce, such as no `type` at the root. 3. **Diffs against the previous version,** labelling breaking changes like a removed required property. 4. **Scans descriptions.** A description goes straight into the model's context, so it is an injection channel. A poisoned one can tell the model to read `~/.ssh` and pass it as a parameter. This **tool poisoning** is a cousin of the indirect injection covered in [Designing LLM Guardrails](/blog/system-design-llm-guardrails); I run the same classifiers over descriptions, flag hidden characters, and send flagged versions to human review. Tool **annotations** such as `readOnlyHint` and `destructiveHint` are, per the MCP spec, hints that clients should not trust from untrusted servers. I verify them for catalog servers; for everything else I derive the risk level myself and default to "write". ### The Rug Pull Problem A remote server can change a tool's description after a tenant approved it. The protocol lets a server announce this with `notifications/tools/list_changed`; a malicious one would not bother. So on every upstream session, the gateway hashes the live `tools/list` and compares it with the pins. On a match it serves the pinned definition. On a mismatch it **still serves the pinned definition**, records a candidate version, scans it and notifies the tenant admin. If the upstream rejects the old call shape, that tool fails clearly until an admin approves. Clients only ever see descriptions the platform scanned and the tenant pinned. That one rule removes most of the "approved yesterday, malicious today" risk. ### Picking the Right Tools per Request Listing every tool burns context (the 300,000-token example), and tool choice gets worse as similar tools crowd each other. So my `tools/list` is **request-aware**: 1. **Filter by hard rules first:** tools in this toolset, from servers the user has connected, that tenant policy allows. Security is never left to ranking. 2. **Always include core tools** the toolset owner marks (for a support agent: ticket lookup and reply). 3. **Retrieve** over name, description and a few synthetic example queries: BM25 for exact terms ("Jira", "invoice"), embeddings for intent ("tell the team the deploy is done"), fused with reciprocal rank fusion. 4. **Rerank and diversify** with a small cross-encoder, capping tools per server. 5. **Return the top k** (I start with 20) **sorted by name**, not score, so the tool block stays identical across turns when the selection does not change, which keeps provider prompt caches warm. ```python def select_tools(ctx, k=20, per_server_cap=6): allowed = registry.allowed_tools(ctx.toolset, ctx.user, ctx.tenant_policy) # hard filter core = [t for t in allowed if t.core] query = ctx.task_text or ctx.last_user_message lexical = bm25.search(query, within=allowed, n=50) semantic = vectors.search(embed(query), within=allowed, n=50) fused = rrf(lexical, semantic) ranked = reranker.rerank(query, fused[:50]) picked, per_server = list(core), Counter(t.server for t in core) for t in ranked: if len(picked) >= k: break if t not in picked and per_server[t.server] < per_server_cap: picked.append(t) per_server[t.server] += 1 return sorted(picked, key=lambda t: t.qualified_name) # stable for prompt caching ``` When the right tool is missing, the model needs a way out: one meta-tool, `platform.search_tools(query)`. If the model calls it, the gateway adds the found tools to the session and sends a list-changed notification so the client refreshes. It is the idea from [Progressive Disclosure: Designing Context for LLM Agents](/blog/progressive-disclosure-designing-context-for-llm-agents): a small surface plus a way to ask for more. ### Failure Modes and What I Would Do First - **Selection misses** show up as `search_tools` calls. I log whether each called tool was in the initial list; that "selection recall" is the selector's main quality metric. - **Stale index:** the filter step reads the registry snapshot, not the index, so a revoked tool is never served even before reindexing. - **Clients that cache `tools/list` forever:** every `tools/call` is validated against the session's allowed set anyway. First: immutable hashed definitions, version pinning, description scanning and a static filtered list. Later, once toolsets pass roughly 50 tools: retrieval and the `search_tools` meta-tool. ## Deep-Dive: Auth and Credentials: Per-User OAuth, a Token Vault, and Scoped Consent ![Auth and credentials: the MCP client gets an audience-bound token from the platform OAuth server using PKCE and calls the gateway with it; the gateway checks consent records for granted scopes and asks a credential broker for the user's upstream token; the broker reads an encrypted refresh token from the token vault, unwraps it with a per-tenant key, refreshes against the provider's OAuth server when needed, and the gateway sends the call through a credential-injecting proxy to the third-party API.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-mcp-tool-platform/04-auth-token-vault.png) ### Two Legs, Two Different Tokens There are two separate trust relationships, and mixing them up is the classic mistake: - **Client to gateway.** The agent proves it may use this toolset on behalf of this user. The gateway is the resource server. - **Gateway to upstream.** The platform proves to GitHub, Google or a remote MCP server that this user granted access. The gateway (through the broker) is the OAuth client. The MCP authorization spec forbids passing a token received from the client through to an upstream API. A token minted for my gateway is accepted only by my gateway; forwarding it would let any upstream replay it elsewhere. ### Leg One: Clients to the Gateway For HTTP transports, MCP authorization builds on OAuth 2.1: 1. An unauthenticated request gets `401` with a `WWW-Authenticate` header pointing to the gateway's **protected resource metadata**, which names my authorization server. 2. The client runs the authorization code flow with **PKCE**. How clients register has been evolving across spec revisions, so I support whichever mechanisms the current revision describes. 3. The client sends a **resource indicator** for the gateway endpoint, so the token's audience is that endpoint only. 4. On every request the gateway checks signature, expiry, audience and toolset scope. Headless agents (a nightly triage bot) use client credentials with a service identity, and can run only tools configured with tenant-level credentials. ### Leg Two: Connecting a User's Account When a selected tool needs the user's GitHub account and none is connected, the call returns a **tool result with `isError: true`**, a short explanation the model can relay, and a connect link in the result's metadata. The user opens the link, the platform runs the provider's authorization code flow with PKCE, and the refresh token goes to the vault. I use a tool result rather than a JSON-RPC protocol error because the model should see it and explain it; protocol errors are for malformed requests. ### Scoped Consent A connection is not a blank cheque. A **consent record** stores the provider, the **scopes granted** (narrow ones where the provider offers them), **which toolsets** may use the connection (connecting GitHub for the coding agent does not authorise the marketing agent), expiry, and who granted it. Effective scopes are the **intersection** of what the tool needs, what consent allows for this toolset, and tenant policy. When a tool needs a scope not yet granted, the platform asks for **incremental consent** for just that scope. Users approve small, specific requests more readily, and a stolen token's blast radius stays small. ### The Token Vault Refresh tokens are long-lived and powerful, so the vault is a small secrets manager: - **Envelope encryption:** a data key per record, wrapped by a **per-tenant key** in a KMS. Plaintext exists only in broker memory. - **Crypto-shredding:** offboarding a tenant destroys its key, making every record unreadable, backups included. - **Narrow access:** only the broker can unwrap. Gateway pods ask the broker for tokens; they cannot read the vault. ### Refresh Without Races Many providers rotate refresh tokens on each refresh, and some treat reuse of an old one as theft and revoke the grant. With 7,000 concurrent calls, two pods refreshing one grant at once is not rare, so refresh is single-flight per grant: ```python def access_token(grant_id, scopes): tok = cache.get((grant_id, scopes)) if tok and tok.expires_at - now() > 60: return tok with lock(f"refresh:{grant_id}", ttl=10): # one refresher per grant, cluster-wide tok = cache.get((grant_id, scopes)) # another pod may have just refreshed if tok and tok.expires_at - now() > 60: return tok rec = vault.read(grant_id) # includes record version resp = provider(rec.provider).refresh(decrypt(rec), scopes) if resp.error == "invalid_grant": vault.mark_revoked(grant_id) # user must reconnect raise NeedsReconnect(grant_id) vault.write(grant_id, encrypt(resp.refresh_token or decrypt(rec)), expected_version=rec.version) # compare-and-set cache.put((grant_id, scopes), resp.access_token) return resp.access_token ``` The lock prevents duplicate refreshes, and the compare-and-set write protects against a lock that expired mid-refresh. ### Keeping Tokens Out of Hosted Code A hosted server wrapping an HTTP API needs the user's token, and untrusted code holding a token can log it or send it elsewhere. My preferred pattern is **credential injection at egress**: the server calls `api.github.com` with no credentials, and the egress proxy attaches the bearer token for the call's (tenant, user, connection), only for hosts that connection is bound to. The code never sees the secret. Some APIs need request signing and some SDKs refuse to send without a token. For those, the gateway passes a short-lived, narrowly scoped token on the call, and instances are recycled on a schedule. For remote MCP servers that run their own OAuth, the broker holds the token that server issued, separate per upstream and never the client's. ### Failure Modes and What I Would Do First - **Provider outage on refresh:** serve cached access tokens until expiry, then fail clearly. Never fall back to a broader shared credential. - **Revocation:** on disconnect, delete the record, broadcast a purge of cached access tokens, and call the provider's revocation endpoint where one exists. - **Admin-installed shared connections** are allowed only for tools marked tenant-level, and the audit log records the human behind each call. First: two-leg separation, audience-bound tokens, a vault with per-tenant keys, single-flight refresh and per-toolset consent. Later: incremental consent, egress injection for the top catalog servers, and anomaly detection on token use. ## Deep-Dive: Sandboxed Execution and Multi-Tenancy for Hosted MCP Servers ![Hosted execution: a build pipeline builds, scans and signs server images into a store of images and VM snapshots; the gateway sends each call with its tenant, server and session to a placement and session router, which restores instances from snapshots on cold start and routes to separate microVMs per tenant and server version; all sandbox traffic leaves through an allowlist egress proxy to upstream SaaS APIs.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-mcp-tool-platform/05-sandboxed-execution.png) ### Why Hosting Is Different from Running Agent Code The agent platform post covers sandboxes for **model-written code**: one per run, then thrown away. Hosted MCP servers are **long-running third-party programs** that serve many calls, keep sessions and caches, and were often written for one developer's laptop over stdio. My job is to turn "a package someone published" into "a multi-tenant network service" without letting that code see anything it should not. ### The Isolation Unit I choose the isolation boundary by asking what a buggy or malicious server could leak: - **Never across tenants.** A server caching an API response in a global variable would serve tenant B's data to tenant A, so the minimum unit is **(tenant, server version)**. - **Per user, for stateful servers** that keep a working directory, browser profile or connection per caller. The manifest declares it, and I default to per user when in doubt. - **Serialised, for stdio servers that are not concurrency-safe.** Inside the sandbox a small bridge exposes the stdio stream over Streamable HTTP; unless the manifest declares concurrency safety, it serialises calls and placement scales out instances instead. The boundary is a **Firecracker microVM**, since the code is untrusted and tenants are mutually hostile, with **gVisor** as a lighter option for first-party servers I build and review. The VMM, jailer and snapshot mechanics are in [Designing a Serverless Platform on MicroVMs](/blog/system-design-serverless-microvm). ### The Server Manifest Every hosted version ships a manifest, the contract the platform enforces: ```yaml server: github version: 3.2.0 image: registry.example.com/mcp/github@sha256:4b1e... # pinned digest, never a tag transport: stdio # bridge wraps it as Streamable HTTP inside the VM isolation: tenant # tenant | user | session concurrency_safe: true resources: vcpu: 1 memory_mb: 256 ephemeral_disk_mb: 512 limits: call_timeout_s: 30 max_response_bytes: 1048576 egress: - host: api.github.com credential: github_oauth # injected by the egress proxy - host: uploads.github.com credential: github_oauth recycle: max_calls: 5000 max_age_minutes: 120 ``` Egress is **deny by default**: the manifest lists the hosts the server needs and which credential the proxy may inject for each. Anything else is refused and shows up on the tenant's security dashboard. ### Build and Supply Chain Most hosted-catalog risk is in what goes into the image. The pipeline builds from a pinned commit or package version with network access only to the package mirror, locks and scans every dependency, records a software bill of materials, and signs the image; hosts refuse unsigned images or digest mismatches. It then boots the server once, runs introspection, and takes a **VM snapshot** after `initialize`, so cold starts restore a ready server. Tenant private servers go through the same pipeline, minus my team's manual review. ### Lifecycle: Warm, Idle, Snapshot, Gone Most (tenant, server) pairs are dormant at any moment, so instances move through four states: - **Warm** while it has calls or a live session, pinned to its host. - **Idle** after a few minutes without calls: snapshot, then stop. - **Recycled** after `max_calls` or `max_age`, which bounds memory leaks and how long any secret passed on a call could linger. - **Gone** when the version is deprecated and unused. Since the model waits on cold starts, I pre-warm a tenant's core tools when an agent session initialises. ### Sessions, Cancellation and Limits - **Session affinity.** A stateful upstream's session id lives in the gateway's session record with the instance address. If the instance dies, the client is told the session ended, rather than being silently routed to a fresh instance without its state. - **Cancellation.** The gateway forwards `notifications/cancelled` upstream and stops waiting; past the timeout the bridge kills the request, and repeated hangs mark the version unhealthy. - **Response limits.** Size caps and content-type checks apply before anything reaches the client, and output is tagged untrusted for the client's guardrail layer. - **Noisy neighbours.** The host enforces per-VM CPU and memory, and per-tenant caps on concurrent instances protect the shared pool. ### Failure Modes and What I Would Do First - **Crash-looping version:** after N crashes in a window the breaker opens and calls fail fast; the tenant can roll the pin back. - **Host failure:** placement restores elsewhere from the snapshot; stateful sessions end and clients reinitialise. - **A compromised server** can reach only allowlisted hosts, with credentials it never sees, inside a VM holding one tenant's data. That containment is the point of the design. First: microVM isolation per (tenant, server version), signed pinned images, deny-by-default egress, per-call timeouts and size caps. Later: snapshot restore, per-user isolation for stateful servers, credential injection for more catalog servers, and predictive pre-warming. ## Deep-Dive: Observability, Quotas, and Rate Limits per Tool and Tenant ![Quotas and observability: the gateway reserves capacity from a hierarchical quota service backed by buckets per tenant, user and tool, then sends admitted calls through an adaptive upstream limiter that caps concurrency to each upstream MCP server; every call emits a span and usage event to a stream that feeds metrics and traces, an append-only audit log, and a usage and billing warehouse.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-mcp-tool-platform/06-observability-quotas.png) ### What Needs Protecting Each limit protects something different: | Limit | Scope | Protects | |-------|-------|----------| | Calls per minute and concurrency | Tenant, and user within tenant | My gateway and sandbox fleet, and fairness between tenants | | Calls per minute per tool | Toolset and tool | Against a looping agent hammering one tool | | Upstream budget | Upstream server or provider, per OAuth app | The third party's limits, which tenants may share | | Destructive calls per hour | User and tool risk level | Blast radius of a confused or injected agent | The upstream row is specific to this kind of platform. When many tenants use the catalog GitHub server through **one shared OAuth app**, providers may limit per app as well as per user, so one tenant's bulk import could get the app throttled for everyone. Each tenant gets a slice of the shared budget, and can bring its own OAuth app for more. ### Hierarchical Buckets A call must pass every level: user, tenant, tool and upstream. I check them in one atomic script against Redis token buckets (the mechanics are in [Designing a Distributed Rate Limiter](/blog/system-design-distributed-rate-limiter)), with limits as per-tenant configuration. Calls are not all equal, so a tool can carry **cost units**: a bulk export might count as 10, a lookup as 1. Concurrency slots are reserved at admission and released on completion or timeout, under a lease so a crashed gateway pod cannot leak them. ### Adaptive Limits Toward Upstreams Static budgets are a guess; upstreams reveal the truth through `429` with `Retry-After`, or rising latency. Per upstream, the gateway runs an **adaptive concurrency limiter** (additive increase, multiplicative decrease): it raises concurrency slowly while latency is stable and cuts it sharply on 429s, timeouts or latency spikes, honouring `Retry-After` for the whole upstream. ### Telling the Agent, Not Just the Client A limit hit is information the model can use: it should switch strategy or tell the user, not retry in a tight loop. So limits come back as a **tool result with `isError: true`**, a short explanation and a retry-after hint. Rejected retries still count against the caller, which makes loops self-limiting. ### What I Log for Every Call One event per call, with a stable shape, feeds metrics, traces, audit and billing: ```json { "ts": "2026-11-03T09:14:22.481Z", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "tenant": "acme", "user": "u_7f3a", "toolset": "support-agent", "tool": "github.create_issue", "def_hash": "sha256:9c1d...", "server_version": "github@3.2.0", "placement": "sandbox:euw1-h17-vm0412", "in_initial_list": true, "status": "ok", "upstream_status": 201, "latency_ms": {"gateway": 6, "credential": 2, "upstream": 388}, "bytes": {"request": 612, "response": 2240}, "cost_units": 1 } ``` **`def_hash`** ties each call to the exact tool text the model saw. **`in_initial_list`** measures selector recall. **The latency split** answers the first incident question: me, the credential path, or the upstream? Trace context from the client, carried in request metadata, links the call into the agent's own trace. Payloads stay out of this event; they go to a separate redacted store only when the tenant opts in. ### Dashboards and Alerts Per tool version I track success rate, p50 and p99 latency, timeouts and **argument validation failure rate**; a high one usually means the description or schema confuses the model, which is useful feedback for the publisher. I alert on error rate or p99 jumping after an upgrade (with a one-click pin rollback), an upstream's adaptive limit collapsing, destructive calls far above a user's baseline, and sandbox egress denials. The audit log is **append-only** in object storage with write-once retention, answering "who called what, as whom, with which credential" months later, independent of the metrics pipeline. ### Failure Modes and What I Would Do First - **Quota store unavailable:** fail open with a local per-pod limit for read-only tools, fail closed for destructive ones. Briefly exceeding quota on a search is fine; an unbounded loop of deletions is not. - **Event stream lagging:** buffer locally, and shed metrics before audit events. First: tenant and user limits, per-call events with the latency split, the audit log and validation-failure dashboards. Later: shared-app upstream slicing, adaptive concurrency per upstream, cost units, and risk-based limits on destructive tools. ## Conclusion: Bringing It All Together A hosted MCP tool platform looks like an API gateway from outside, but three things make it different. Callers are models that choose tools by reading text, so **descriptions are routing data and attack surface**. The code is often written by strangers, so **hosting means isolation first**. And every call acts as a specific person, so **credentials belong to users, not services**. If I had to summarise the design in four decisions: 1. **Make tool definitions immutable, hashed and pinned, and select per request,** with a meta-tool to ask for more. 2. **Keep the two OAuth legs separate and tokens in a vault:** audience-bound client tokens, per-toolset consent, single-flight refresh, no passthrough. 3. **Isolate hosted servers per tenant and version in microVMs with deny-by-default egress,** injecting credentials outside the sandbox where possible. 4. **Limit at every level and log one rich event per call,** with errors the model can understand and an audit trail tied to the exact tool text. Related reading: [Designing an AI Agent Platform](/blog/system-design-ai-agent-platform), [Designing an LLM Gateway](/blog/system-design-llm-gateway), [Designing a Serverless Platform on MicroVMs](/blog/system-design-serverless-microvm), and [Designing LLM Guardrails](/blog/system-design-llm-guardrails).