Designing an MCP Tool Platform: Registry, Gateway, Sandboxes and Credentials at Scale

An agent is only as useful as the tools it can reach. The Model Context Protocol (MCP) gave the industry a common way to plug tools into models: a client inside the AI app talks JSON-RPC 2.0 to a server that exposes tools, resources and prompts. That solved the "every integration is custom" problem and created a new one: a company with fifty agents and a few hundred MCP servers now has a few hundred pieces of third-party code, a few hundred OAuth integrations, and thousands of tool descriptions competing for the model's context window.

The platform I design here sits in the middle. Agents connect to one endpoint. Behind it, the platform hosts or brokers thousands of MCP servers, keeps a versioned registry of every tool, picks the few tools that matter for each request, holds each user's OAuth tokens so no agent sees them, runs untrusted server code in isolation, and meters every call.

In this post I walk through how I would build that from scratch. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives:

• Tool registry and discovery
• Auth and credentials
• Sandboxed execution and multi-tenancy for hosted servers
• Observability, quotas and rate limits

Scoping the Problem and Clarifying Assumptions

I am building the tool side of the agent stack: the registry, the gateway in front of many MCP servers, credential brokering, and the runtime that hosts server code. The agent loop (planning, memory, durable runs, approvals) is a separate system that calls mine; I covered it in Designing an AI Agent Platform.

What I'm including:

• A public catalog of MCP servers plus private servers each tenant publishes for its internal APIs
• Hosted servers (packaged code, often written for stdio) that I run, and remote servers run by third parties that I reach over Streamable HTTP
• A gateway that looks like one MCP server to clients and fans out to many upstreams
• Per-user OAuth with a token vault and explicit consent
• Quotas, metering, traces and audit per tool, user and tenant

What I'm NOT building here:

• The agent runtime and its memory. Clients bring their own loop.
• LLM serving. Tool selection uses a small embedding model and reranker; the main model lives elsewhere.
• Prompt-injection detection in tool outputs, which is covered in Designing LLM Guardrails.

Key assumptions:

• Clients speak MCP, almost always over Streamable HTTP (IDE tools can use a small local stdio-to-HTTP bridge)
• About 20,000 tenants, around 2,000 catalog servers and 30,000 private servers
• A tenant connects 20 to 60 servers with 5 to 40 tools each, so a tenant reaches hundreds to low thousands of tools
• Most calls are I/O-bound: wrap a SaaS API, wait 100 ms to a few seconds, return a few kilobytes

How this differs from a classic API gateway: a classic gateway routes to services the company wrote and trusts, for callers who know which endpoint they want. Here the callers are models choosing tools from natural-language descriptions, so the description is both routing logic and attack surface. The upstream code is often written by strangers, and the credential on each call belongs to an end user, not the calling service.

Functional and Non-Functional Requirements

Functional Requirements

1. Publish an MCP server version (a package to host, or a remote URL); the platform introspects and records its tools, resources and prompts.
2. Compose tools from many servers into a toolset served at one gateway endpoint.
3. Discover the tools relevant to each request, filtered to what this user may use.
4. Connect a user's third-party account once via OAuth, without the client ever holding the token.
5. Call the right upstream with the right credential, with timeouts, size limits and cancellation.
6. Govern with quotas, audit logs, metering and admin policy per tenant.

Non-Functional Requirements

I do not promise an SLO for upstream servers. A slow third-party server is not my outage, but detecting it, containing it and telling the caller clearly is.

Back-of-the-Envelope Capacity Estimation

These are planning assumptions, not measurements. The method: estimate call volume, then in-flight concurrency, then memory for hosted instances.

Tool calls:

• 500 million tool calls per day across all tenants
• 500M / 86,400 s ≈ 5,800 calls per second on average, about 17,000 at peak (3x)

Gateway concurrency:

• With a mean call duration of 400 ms, in-flight calls at peak ≈ 17,000 × 0.4 ≈ 7,000 concurrent upstream calls
• If 1.5 million agent sessions are active at peak and a third keep a notification stream open, that is 500,000 mostly idle connections on the edge tier

Discovery:

• One discovery per model turn with tools, about 150 million per day, so about 5,000 per second at peak
• If a tool definition (name, description, JSON Schema) averages 300 tokens, a tenant with 1,000 reachable tools would spend 300,000 tokens just listing them. The top 20 cost about 6,000.

Hosted instances:

• I assume 40% of calls go to platform-hosted servers: about 6,800 calls per second at peak
• Instances are isolated per (tenant, server version) at minimum. If 60,000 such pairs are active at peak and each warm microVM uses 256 MB, that is about 15 TB of memory, roughly 50 hosts with 384 GB each. Idle pairs are snapshotted and stopped, which keeps this bounded by active pairs, not by every tenant times every catalog server.

Token vault:

• 5 million connected accounts at about 2 KB each (encrypted refresh token plus metadata) is 10 GB. The load is reads: the broker caches access tokens and refreshes about hourly per active grant. With 1 million grants active in a peak hour, that is about 280 refreshes per second.

Audit and metering:

• 500M calls × about 1 KB of metadata ≈ 500 GB per day of call events. Full payloads are stored only for tenants who opt in.

High-Level Architecture and Core Components

Component Walkthrough

MCP Gateway. To the client it is one MCP server: it answers initialize, tools/list, tools/call, resources/read and prompts/get, and pushes notifications. Toward each upstream it is an MCP client with its own session, protocol version and capabilities. Tools are namespaced by server (github.create_issue, jira.create_issue) so names never collide. Session routing state lives in a shared cache, so pods stay stateless.

Platform OAuth server. Issues the tokens clients use to call the gateway. This is a different leg from the user's third-party credentials, and I keep the two strictly apart.

Quota service. Buckets and concurrency limits per tenant, user, tool and upstream.

Tool registry and selector. The registry stores every server version and the exact, hashed definition of each tool. The selector picks the tools to list per request. Covered in the first deep-dive.

Credential broker and token vault. The vault stores encrypted refresh tokens per (tenant, user, connection); the broker turns a call's identity into a short-lived upstream access token. Covered in the second deep-dive.

Hosted MCP servers. Catalog or tenant packages, built into images and run in microVM sandboxes per tenant and server version. Covered in the third deep-dive.

Remote MCP servers. Third-party servers I do not run but still treat as untrusted peers: strict timeouts, response size limits and a circuit breaker per server.

Call event stream. One event per call, feeding metrics, traces, audit and billing. Covered in the fourth deep-dive.

Request Flow

1. An agent opens a session: initialize to https://mcp.example.com/t/acme/toolsets/support-agent with a bearer token from the platform OAuth server.
2. The gateway verifies the token (signature, expiry, audience), resolves tenant, user and toolset, and returns capabilities including tools with list-change notifications.
3. Before a model turn, the client calls tools/list, optionally attaching the task text in request metadata. The selector returns 20 tools this user may call.
4. The model picks github.create_issue. On tools/call, the gateway validates arguments against the pinned schema, reserves quota, and gets this user's GitHub token from the broker.
5. The gateway forwards the call to this tenant's GitHub server sandbox; the token rides on the outgoing request and is never stored there.
6. The result returns to the client, and the gateway emits a call event.

Scaling It Out

• Stateless gateways, sticky sessions. A Streamable HTTP server may assign an Mcp-Session-Id at initialisation and expect it on later requests. I keep a session record (client session id, upstream session ids, sandbox placement) in a regional cache, so any pod can serve any request while the hosted leg sticks to the same sandbox.
• Registry reads are local. One writer region; read replicas per region plus an in-memory snapshot in every pod, refreshed on change events. A registry outage means "no new versions publish", not "no calls work".
• Data residency. Token vaults are regional, so a European tenant's refresh tokens never leave the EU.
• Stable egress IPs per region, for remote server operators who allowlist callers.

Deep-Dive: Tool Registry and Discovery: Versioned Schemas and Picking the Right Tools per Request

What a Registry Entry Is

The agent platform post already describes the per-tool record a runtime needs (schema plus hidden metadata like side effects and approval rules). At catalog scale the hard part is change over time: servers ship new versions, remote servers change their tool list silently, and the same tool name can mean different things a month apart. So every tool definition is immutable and addressed by a hash of its exact content:

CREATE TABLE server_version (
  server_id     UUID REFERENCES mcp_server,
  version       TEXT,                 -- '3.2.0' or an observed snapshot id for remote
  artifact      TEXT,                 -- image digest (hosted) or URL (remote)
  protocol_ver  TEXT,                 -- negotiated during introspection
  status        TEXT,                 -- 'pending_review' | 'active' | 'deprecated' | 'blocked'
  PRIMARY KEY (server_id, version)
);

CREATE TABLE tool_def (
  def_hash      BYTEA PRIMARY KEY,    -- sha256 of canonical JSON (name, description, schemas, annotations)
  server_id     UUID,
  version       TEXT,
  name          TEXT,
  definition    JSONB,
  risk          TEXT                  -- derived: 'read' | 'write' | 'destructive'
);

CREATE TABLE toolset_pin (
  toolset_id    UUID,
  server_id     UUID,
  version       TEXT,                 -- pinned; upgrades are explicit
  allowed_tools TEXT[],               -- NULL means all tools in that version
  PRIMARY KEY (toolset_id, server_id)
);

A toolset pins a server version, not a server. Upgrading is an explicit action with a diff, like bumping a library dependency.

Ingestion: Introspect, Normalise, Scan

When a version is published, an ingest worker starts it (or connects, for remote servers), runs the initialize handshake, and pages through tools/list, resources/list and prompts/list with the server's cursor. Then it:

1. Canonicalises each definition (sorted keys, normalised whitespace) and hashes it.
2. Validates the input schema and rejects shapes I cannot enforce, such as no type at the root.
3. Diffs against the previous version, labelling breaking changes like a removed required property.
4. Scans descriptions. A description goes straight into the model's context, so it is an injection channel. A poisoned one can tell the model to read ~/.ssh and pass it as a parameter. This tool poisoning is a cousin of the indirect injection covered in Designing LLM Guardrails; I run the same classifiers over descriptions, flag hidden characters, and send flagged versions to human review.

Tool annotations such as readOnlyHint and destructiveHint are, per the MCP spec, hints that clients should not trust from untrusted servers. I verify them for catalog servers; for everything else I derive the risk level myself and default to "write".

The Rug Pull Problem

A remote server can change a tool's description after a tenant approved it. The protocol lets a server announce this with notifications/tools/list_changed; a malicious one would not bother. So on every upstream session, the gateway hashes the live tools/list and compares it with the pins. On a match it serves the pinned definition. On a mismatch it still serves the pinned definition, records a candidate version, scans it and notifies the tenant admin. If the upstream rejects the old call shape, that tool fails clearly until an admin approves.

Clients only ever see descriptions the platform scanned and the tenant pinned. That one rule removes most of the "approved yesterday, malicious today" risk.

Picking the Right Tools per Request

Listing every tool burns context (the 300,000-token example), and tool choice gets worse as similar tools crowd each other. So my tools/list is request-aware:

1. Filter by hard rules first: tools in this toolset, from servers the user has connected, that tenant policy allows. Security is never left to ranking.
2. Always include core tools the toolset owner marks (for a support agent: ticket lookup and reply).
3. Retrieve over name, description and a few synthetic example queries: BM25 for exact terms ("Jira", "invoice"), embeddings for intent ("tell the team the deploy is done"), fused with reciprocal rank fusion.
4. Rerank and diversify with a small cross-encoder, capping tools per server.
5. Return the top k (I start with 20) sorted by name, not score, so the tool block stays identical across turns when the selection does not change, which keeps provider prompt caches warm.

def select_tools(ctx, k=20, per_server_cap=6):
    allowed = registry.allowed_tools(ctx.toolset, ctx.user, ctx.tenant_policy)  # hard filter
    core = [t for t in allowed if t.core]
    query = ctx.task_text or ctx.last_user_message
    lexical = bm25.search(query, within=allowed, n=50)
    semantic = vectors.search(embed(query), within=allowed, n=50)
    fused = rrf(lexical, semantic)
    ranked = reranker.rerank(query, fused[:50])
    picked, per_server = list(core), Counter(t.server for t in core)
    for t in ranked:
        if len(picked) >= k:
            break
        if t not in picked and per_server[t.server] < per_server_cap:
            picked.append(t)
            per_server[t.server] += 1
    return sorted(picked, key=lambda t: t.qualified_name)          # stable for prompt caching

When the right tool is missing, the model needs a way out: one meta-tool, platform.search_tools(query). If the model calls it, the gateway adds the found tools to the session and sends a list-changed notification so the client refreshes. It is the idea from Progressive Disclosure: Designing Context for LLM Agents: a small surface plus a way to ask for more.

Failure Modes and What I Would Do First

• Selection misses show up as search_tools calls. I log whether each called tool was in the initial list; that "selection recall" is the selector's main quality metric.
• Stale index: the filter step reads the registry snapshot, not the index, so a revoked tool is never served even before reindexing.
• Clients that cache tools/list forever: every tools/call is validated against the session's allowed set anyway.

First: immutable hashed definitions, version pinning, description scanning and a static filtered list. Later, once toolsets pass roughly 50 tools: retrieval and the search_tools meta-tool.

Deep-Dive: Auth and Credentials: Per-User OAuth, a Token Vault, and Scoped Consent

Two Legs, Two Different Tokens

There are two separate trust relationships, and mixing them up is the classic mistake:

• Client to gateway. The agent proves it may use this toolset on behalf of this user. The gateway is the resource server.
• Gateway to upstream. The platform proves to GitHub, Google or a remote MCP server that this user granted access. The gateway (through the broker) is the OAuth client.

The MCP authorization spec forbids passing a token received from the client through to an upstream API. A token minted for my gateway is accepted only by my gateway; forwarding it would let any upstream replay it elsewhere.

Leg One: Clients to the Gateway

For HTTP transports, MCP authorization builds on OAuth 2.1:

1. An unauthenticated request gets 401 with a WWW-Authenticate header pointing to the gateway's protected resource metadata, which names my authorization server.
2. The client runs the authorization code flow with PKCE. How clients register has been evolving across spec revisions, so I support whichever mechanisms the current revision describes.
3. The client sends a resource indicator for the gateway endpoint, so the token's audience is that endpoint only.
4. On every request the gateway checks signature, expiry, audience and toolset scope.

Headless agents (a nightly triage bot) use client credentials with a service identity, and can run only tools configured with tenant-level credentials.

Leg Two: Connecting a User's Account

When a selected tool needs the user's GitHub account and none is connected, the call returns a tool result with isError: true, a short explanation the model can relay, and a connect link in the result's metadata. The user opens the link, the platform runs the provider's authorization code flow with PKCE, and the refresh token goes to the vault. I use a tool result rather than a JSON-RPC protocol error because the model should see it and explain it; protocol errors are for malformed requests.

Scoped Consent

A connection is not a blank cheque. A consent record stores the provider, the scopes granted (narrow ones where the provider offers them), which toolsets may use the connection (connecting GitHub for the coding agent does not authorise the marketing agent), expiry, and who granted it.

Effective scopes are the intersection of what the tool needs, what consent allows for this toolset, and tenant policy. When a tool needs a scope not yet granted, the platform asks for incremental consent for just that scope. Users approve small, specific requests more readily, and a stolen token's blast radius stays small.

The Token Vault

Refresh tokens are long-lived and powerful, so the vault is a small secrets manager:

• Envelope encryption: a data key per record, wrapped by a per-tenant key in a KMS. Plaintext exists only in broker memory.
• Crypto-shredding: offboarding a tenant destroys its key, making every record unreadable, backups included.
• Narrow access: only the broker can unwrap. Gateway pods ask the broker for tokens; they cannot read the vault.

Refresh Without Races

Many providers rotate refresh tokens on each refresh, and some treat reuse of an old one as theft and revoke the grant. With 7,000 concurrent calls, two pods refreshing one grant at once is not rare, so refresh is single-flight per grant:

def access_token(grant_id, scopes):
    tok = cache.get((grant_id, scopes))
    if tok and tok.expires_at - now() > 60:
        return tok
    with lock(f"refresh:{grant_id}", ttl=10):          # one refresher per grant, cluster-wide
        tok = cache.get((grant_id, scopes))             # another pod may have just refreshed
        if tok and tok.expires_at - now() > 60:
            return tok
        rec = vault.read(grant_id)                      # includes record version
        resp = provider(rec.provider).refresh(decrypt(rec), scopes)
        if resp.error == "invalid_grant":
            vault.mark_revoked(grant_id)                # user must reconnect
            raise NeedsReconnect(grant_id)
        vault.write(grant_id, encrypt(resp.refresh_token or decrypt(rec)),
                    expected_version=rec.version)       # compare-and-set
        cache.put((grant_id, scopes), resp.access_token)
        return resp.access_token

The lock prevents duplicate refreshes, and the compare-and-set write protects against a lock that expired mid-refresh.

Keeping Tokens Out of Hosted Code

A hosted server wrapping an HTTP API needs the user's token, and untrusted code holding a token can log it or send it elsewhere. My preferred pattern is credential injection at egress: the server calls api.github.com with no credentials, and the egress proxy attaches the bearer token for the call's (tenant, user, connection), only for hosts that connection is bound to. The code never sees the secret.

Some APIs need request signing and some SDKs refuse to send without a token. For those, the gateway passes a short-lived, narrowly scoped token on the call, and instances are recycled on a schedule. For remote MCP servers that run their own OAuth, the broker holds the token that server issued, separate per upstream and never the client's.

Failure Modes and What I Would Do First

• Provider outage on refresh: serve cached access tokens until expiry, then fail clearly. Never fall back to a broader shared credential.
• Revocation: on disconnect, delete the record, broadcast a purge of cached access tokens, and call the provider's revocation endpoint where one exists.
• Admin-installed shared connections are allowed only for tools marked tenant-level, and the audit log records the human behind each call.

First: two-leg separation, audience-bound tokens, a vault with per-tenant keys, single-flight refresh and per-toolset consent. Later: incremental consent, egress injection for the top catalog servers, and anomaly detection on token use.

Deep-Dive: Sandboxed Execution and Multi-Tenancy for Hosted MCP Servers

Why Hosting Is Different from Running Agent Code

The agent platform post covers sandboxes for model-written code: one per run, then thrown away. Hosted MCP servers are long-running third-party programs that serve many calls, keep sessions and caches, and were often written for one developer's laptop over stdio. My job is to turn "a package someone published" into "a multi-tenant network service" without letting that code see anything it should not.

The Isolation Unit

I choose the isolation boundary by asking what a buggy or malicious server could leak:

• Never across tenants. A server caching an API response in a global variable would serve tenant B's data to tenant A, so the minimum unit is (tenant, server version).
• Per user, for stateful servers that keep a working directory, browser profile or connection per caller. The manifest declares it, and I default to per user when in doubt.
• Serialised, for stdio servers that are not concurrency-safe. Inside the sandbox a small bridge exposes the stdio stream over Streamable HTTP; unless the manifest declares concurrency safety, it serialises calls and placement scales out instances instead.

The boundary is a Firecracker microVM, since the code is untrusted and tenants are mutually hostile, with gVisor as a lighter option for first-party servers I build and review. The VMM, jailer and snapshot mechanics are in Designing a Serverless Platform on MicroVMs.

The Server Manifest

Every hosted version ships a manifest, the contract the platform enforces:

server: github
version: 3.2.0
image: registry.example.com/mcp/github@sha256:4b1e...   # pinned digest, never a tag
transport: stdio            # bridge wraps it as Streamable HTTP inside the VM
isolation: tenant           # tenant | user | session
concurrency_safe: true
resources:
  vcpu: 1
  memory_mb: 256
  ephemeral_disk_mb: 512
limits:
  call_timeout_s: 30
  max_response_bytes: 1048576
egress:
  - host: api.github.com
    credential: github_oauth    # injected by the egress proxy
  - host: uploads.github.com
    credential: github_oauth
recycle:
  max_calls: 5000
  max_age_minutes: 120

Egress is deny by default: the manifest lists the hosts the server needs and which credential the proxy may inject for each. Anything else is refused and shows up on the tenant's security dashboard.

Build and Supply Chain

Most hosted-catalog risk is in what goes into the image. The pipeline builds from a pinned commit or package version with network access only to the package mirror, locks and scans every dependency, records a software bill of materials, and signs the image; hosts refuse unsigned images or digest mismatches. It then boots the server once, runs introspection, and takes a VM snapshot after initialize, so cold starts restore a ready server. Tenant private servers go through the same pipeline, minus my team's manual review.

Lifecycle: Warm, Idle, Snapshot, Gone

Most (tenant, server) pairs are dormant at any moment, so instances move through four states:

• Warm while it has calls or a live session, pinned to its host.
• Idle after a few minutes without calls: snapshot, then stop.
• Recycled after max_calls or max_age, which bounds memory leaks and how long any secret passed on a call could linger.
• Gone when the version is deprecated and unused.

Since the model waits on cold starts, I pre-warm a tenant's core tools when an agent session initialises.

Sessions, Cancellation and Limits

• Session affinity. A stateful upstream's session id lives in the gateway's session record with the instance address. If the instance dies, the client is told the session ended, rather than being silently routed to a fresh instance without its state.
• Cancellation. The gateway forwards notifications/cancelled upstream and stops waiting; past the timeout the bridge kills the request, and repeated hangs mark the version unhealthy.
• Response limits. Size caps and content-type checks apply before anything reaches the client, and output is tagged untrusted for the client's guardrail layer.
• Noisy neighbours. The host enforces per-VM CPU and memory, and per-tenant caps on concurrent instances protect the shared pool.

Failure Modes and What I Would Do First

• Crash-looping version: after N crashes in a window the breaker opens and calls fail fast; the tenant can roll the pin back.
• Host failure: placement restores elsewhere from the snapshot; stateful sessions end and clients reinitialise.
• A compromised server can reach only allowlisted hosts, with credentials it never sees, inside a VM holding one tenant's data. That containment is the point of the design.

First: microVM isolation per (tenant, server version), signed pinned images, deny-by-default egress, per-call timeouts and size caps. Later: snapshot restore, per-user isolation for stateful servers, credential injection for more catalog servers, and predictive pre-warming.

Deep-Dive: Observability, Quotas, and Rate Limits per Tool and Tenant

What Needs Protecting

Each limit protects something different:

The upstream row is specific to this kind of platform. When many tenants use the catalog GitHub server through one shared OAuth app, providers may limit per app as well as per user, so one tenant's bulk import could get the app throttled for everyone. Each tenant gets a slice of the shared budget, and can bring its own OAuth app for more.

Hierarchical Buckets

A call must pass every level: user, tenant, tool and upstream. I check them in one atomic script against Redis token buckets (the mechanics are in Designing a Distributed Rate Limiter), with limits as per-tenant configuration. Calls are not all equal, so a tool can carry cost units: a bulk export might count as 10, a lookup as 1. Concurrency slots are reserved at admission and released on completion or timeout, under a lease so a crashed gateway pod cannot leak them.

Adaptive Limits Toward Upstreams

Static budgets are a guess; upstreams reveal the truth through 429 with Retry-After, or rising latency. Per upstream, the gateway runs an adaptive concurrency limiter (additive increase, multiplicative decrease): it raises concurrency slowly while latency is stable and cuts it sharply on 429s, timeouts or latency spikes, honouring Retry-After for the whole upstream.

Telling the Agent, Not Just the Client

A limit hit is information the model can use: it should switch strategy or tell the user, not retry in a tight loop. So limits come back as a tool result with isError: true, a short explanation and a retry-after hint. Rejected retries still count against the caller, which makes loops self-limiting.

What I Log for Every Call

One event per call, with a stable shape, feeds metrics, traces, audit and billing:

{
  "ts": "2026-11-03T09:14:22.481Z",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "tenant": "acme",
  "user": "u_7f3a",
  "toolset": "support-agent",
  "tool": "github.create_issue",
  "def_hash": "sha256:9c1d...",
  "server_version": "github@3.2.0",
  "placement": "sandbox:euw1-h17-vm0412",
  "in_initial_list": true,
  "status": "ok",
  "upstream_status": 201,
  "latency_ms": {"gateway": 6, "credential": 2, "upstream": 388},
  "bytes": {"request": 612, "response": 2240},
  "cost_units": 1
}

def_hash ties each call to the exact tool text the model saw. in_initial_list measures selector recall. The latency split answers the first incident question: me, the credential path, or the upstream? Trace context from the client, carried in request metadata, links the call into the agent's own trace. Payloads stay out of this event; they go to a separate redacted store only when the tenant opts in.

Dashboards and Alerts

Per tool version I track success rate, p50 and p99 latency, timeouts and argument validation failure rate; a high one usually means the description or schema confuses the model, which is useful feedback for the publisher. I alert on error rate or p99 jumping after an upgrade (with a one-click pin rollback), an upstream's adaptive limit collapsing, destructive calls far above a user's baseline, and sandbox egress denials.

The audit log is append-only in object storage with write-once retention, answering "who called what, as whom, with which credential" months later, independent of the metrics pipeline.

Failure Modes and What I Would Do First

• Quota store unavailable: fail open with a local per-pod limit for read-only tools, fail closed for destructive ones. Briefly exceeding quota on a search is fine; an unbounded loop of deletions is not.
• Event stream lagging: buffer locally, and shed metrics before audit events.

First: tenant and user limits, per-call events with the latency split, the audit log and validation-failure dashboards. Later: shared-app upstream slicing, adaptive concurrency per upstream, cost units, and risk-based limits on destructive tools.

Conclusion: Bringing It All Together

A hosted MCP tool platform looks like an API gateway from outside, but three things make it different. Callers are models that choose tools by reading text, so descriptions are routing data and attack surface. The code is often written by strangers, so hosting means isolation first. And every call acts as a specific person, so credentials belong to users, not services.

If I had to summarise the design in four decisions:

1. Make tool definitions immutable, hashed and pinned, and select per request, with a meta-tool to ask for more.
2. Keep the two OAuth legs separate and tokens in a vault: audience-bound client tokens, per-toolset consent, single-flight refresh, no passthrough.
3. Isolate hosted servers per tenant and version in microVMs with deny-by-default egress, injecting credentials outside the sandbox where possible.
4. Limit at every level and log one rich event per call, with errors the model can understand and an audit trail tied to the exact tool text.

Related reading: Designing an AI Agent Platform, Designing an LLM Gateway, Designing a Serverless Platform on MicroVMs, and Designing LLM Guardrails.