# Designing an AI Coding Assistant like Cursor and GitHub Copilot ## Blog Details - **Author**: Naveen R. - **Date**: October 18, 2026 - **Tags**: system design, AI coding assistant, LLM, code completion, agent mode - **Read Time**: 20 mins When I design an AI coding assistant like Cursor or GitHub Copilot, I am really designing three products that happen to share a sidebar. The first is inline completion: grey "ghost text" that appears while the developer types, and that is useless if it shows up half a second after they have already moved on. The second is chat about the codebase, where the quality of the answer depends almost entirely on whether I found the right ten files out of ten thousand. The third is agent mode, where the model plans a change, edits several files, runs the test suite, reads the failures and tries again, which means I am now running untrusted, model-chosen commands on someone's machine. Completion is a latency problem with enormous request volume and tiny outputs. Chat is a retrieval problem. Agent mode is a safety and cost problem, because one task can make dozens of frontier model calls with a context that grows on every step. The design work is giving each mode what it needs while sharing the codebase index, the context assembler and the model router. In this post I walk through how I would build this system from scratch. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives: - Codebase indexing with AST-aware chunks, embeddings and a symbol graph - The inline completion hot path - Agent mode with tool use and sandboxed execution - Context assembly, prompt caching and model routing ## Scoping the Problem and Clarifying Assumptions I am building the **assistant platform**: the IDE extension, the backend services behind it, and the serving and routing layer for the models. I treat the large frontier models as something I call, either from external providers or from a separate inference platform, rather than something this system trains or hosts. **What I'm including:** - An IDE extension (VS Code-style editors and JetBrains-style IDEs) with three modes: inline completion, codebase chat, and agent mode - A codebase index per repository, kept fresh as files change - A small, fast completion model that I host myself, plus routing to frontier models for chat and agents - Tool execution for agents: reading and searching files, editing, and running terminal commands and tests - Telemetry for acceptance rates, latency and cost **What I'm NOT building here:** - Model training and fine-tuning pipelines. I note where telemetry would feed them, but the training system is separate. - The GPU inference engine internals (batching, KV cache paging). I rely on a standard serving engine for those. - Code review bots and CI integrations. They can reuse the same index and agent runtime, but they are separate products. **Key assumptions:** - About **5 million daily active developers**, concentrated in working hours across a handful of time zones - Completion requests fire on most keystroke pauses, after a short debounce - Repositories range from small side projects to monorepos with hundreds of thousands of files - Many customers are companies with strict rules about where their source code may go, so privacy controls are a requirement, not a feature **How this differs from classic code search or autocomplete:** classic autocomplete is local and driven by a type checker, and classic code search is queried by a human. Here the index is queried by a model many times per minute, and the consumer is a context window with a hard token budget, not a results page. I would rather return three precisely right snippets than thirty plausible ones. ## Functional and Non-Functional Requirements ### Functional Requirements 1. **Inline completion**: as the developer types, suggest the next few tokens or lines at the cursor, using code both before and after the cursor. 2. **Codebase chat**: answer questions about the repository, citing files and symbols, and propose edits the user can apply. 3. **Agent mode**: given a task, plan it, read and search the code, edit multiple files, run commands and tests, and iterate until done or out of budget. 4. **Indexing**: build and incrementally update a semantic and structural index of each repository. 5. **Safety controls**: permission prompts or sandboxing for commands, checkpoints and one-click undo for agent edits, and privacy modes that control what code leaves the machine. 6. **Telemetry**: record shown, accepted, rejected and edited suggestions, plus latency and token cost per request. ### Non-Functional Requirements | Requirement | Target | Why | |-------------|--------|-----| | Inline completion latency | p50 under 300 ms, p95 under 600 ms from keystroke pause to ghost text | Slower suggestions arrive after the developer has moved on | | Chat time to first token | under 1.5 s | Feels responsive in a sidebar | | Index freshness | changed files searchable within about 1 minute of save | Agents and chat must see the code as it is now | | Availability | 99.9% for completion and chat, with graceful fallback | Developers notice immediately when it breaks | | Privacy | no plaintext code retained server-side in privacy mode | Enterprise requirement | | Cost | bounded cost per active developer per month | Agent mode can otherwise cost more than the subscription | I measure completion latency from the moment the developer pauses, because debounce, client work and the network are all part of what the user feels. ### Back-of-the-Envelope Capacity Estimation These are planning assumptions, not measurements. The method matters more than the exact values. **Inline completion traffic:** - I assume a developer spends about 2 hours per day actively editing and that, after debounce and client-side caching, one completion request goes out about every 4 seconds of active editing: 7,200 / 4 = **1,800 requests per developer per day** - 5M × 1,800 = **9 billion requests per day** - 9B / 86,400 s ≈ **104,000 requests per second on average**, and with working-hours concentration I plan for about **300,000 per second at peak** - I assume roughly 40% of these are cancelled before they finish, because the developer kept typing **Completion tokens:** - Each request carries about 1,500 input tokens (prefix, suffix and a few related snippets) and produces about 25 output tokens - Input: 104K × 1,500 ≈ **156 million prompt tokens per second** on average - Output: 104K × 25 ≈ **2.6 million output tokens per second** - Consecutive requests from one developer share almost all of their prefix. If I route a developer to the same replica and assume about 80% of each prompt is reused from its cache, fresh prefill drops to around **31 million tokens per second**. This is why session affinity shows up in the architecture. **Chat and agent traffic:** - Chat: 15 messages per developer per day, so 75M per day, about **870 per second on average**, each with around 20,000 input tokens of retrieved context and history - Agent: I assume 20% of developers (1M) run 5 agent tasks per day, each making about 25 model calls: 125M calls per day, about **1,450 per second on average** - Agent calls carry a growing transcript. At an average of 50,000 input tokens per call, that is about **72 million input tokens per second** to frontier models. Without prompt caching, agent mode would dominate the entire cost structure. **Index storage:** - I assume about 3 million distinct workspaces indexed (teammates on the same repo share content-addressed chunks), averaging 5,000 files with 4 chunks per file: **20,000 chunks per workspace** - A 768-dimension embedding in FP16 is 1.5 KB, about 2 KB with metadata, so 20,000 × 2 KB ≈ **40 MB per workspace** - 3M × 40 MB ≈ **120 TB of vectors**, before deduplication across branches and forks - Re-index churn: if each developer's saves change about 400 chunks per day, that is 5M × 400 / 86,400 ≈ **23,000 chunk embeddings per second**, before content hashing removes duplicates. ## High-Level Architecture and Core Components ![The request path for the coding assistant: the IDE extension talks to an API gateway, which sends inline requests to a completion service backed by a small fill-in-the-middle model pool and sends chat and agent requests to an orchestrator; the orchestrator builds prompts through a context assembler that retrieves from a per-repo vector and symbol index, calls frontier model providers, runs commands in a sandbox runner, and the gateway emits acceptance telemetry to an event stream.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-ai-coding-assistant/01-high-level-architecture.png) ### Component Walkthrough **IDE extension.** Not a thin client. It watches keystrokes and saves, keeps a local Merkle tree for change detection, parses files into syntax trees, caches completions, and builds inline completion context locally because that is where open tabs and recent edits live. It also hosts the local half of agent tools: reading files, applying edits and running commands. **API Gateway.** Authenticates, applies per-organisation policy (privacy mode, allowed models, retention), and keeps long-lived HTTP/2 connections so completions never pay a fresh TLS handshake. **Completion service.** A latency-obsessed layer in front of the self-hosted fill-in-the-middle (FIM) model: it formats the prompt, picks a replica with session affinity, enforces a deadline and cancels work when the client resets the stream (second deep-dive). **Chat and agent orchestrator.** Runs the conversation and agent loop, executes tool calls, enforces budgets and records checkpoints (third deep-dive). **Context assembler.** Gathers candidate context, ranks it and packs it under a token budget in a cache-friendly order (fourth deep-dive). **Per-repo index.** Vector and lexical indexes of code chunks plus a symbol graph, partitioned by workspace (first deep-dive). **Model router, sandbox runner and telemetry.** The router picks a model per request with provider failover. The sandbox runs commands for cloud agents. Telemetry records shown, accepted and rejected suggestions with latency and tokens, feeding dashboards, cost accounting and model evaluation. ### Request Flow For an inline completion: 1. The developer pauses typing. The extension waits out the debounce, checks its local cache and, on a miss, builds a prompt from the prefix, the suffix and a few related snippets. 2. The request goes over an existing HTTP/2 connection to the gateway and on to the completion service. 3. The completion service routes to the replica this session used last, which likely still holds the prefix in its KV cache, and streams tokens back. 4. The extension renders ghost text as soon as it has a useful first line. If the developer types again, it resets the stream and the server stops generating. For a chat or agent turn, the extension sends the message with local signals (open files, selection, recent edits), the orchestrator gets a packed prompt from the context assembler, the router picks a model, and tool calls run in a loop. Edits come back as diffs the extension applies locally after taking a checkpoint. ### Scaling It Out ![The scaled-out system: a latency-based global load balancer sends developers to regional edges, each with its own fill-in-the-middle model pool using session affinity; agent orchestrators call a model router that fails over across frontier providers and retrieve from index shards partitioned by repository, while a re-index queue feeds embedding workers that upsert vectors into those shards.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-ai-coding-assistant/02-scalable-architecture.png) - **Completion pools in every region.** Inline completion cannot absorb a cross-ocean round trip, so the FIM model runs in each region close to developers. This is the only part of the system where I would pay for GPUs in many regions. - **Chat and agents can be centralised.** A 100 ms round trip is invisible next to a multi-second frontier model call, so orchestrators and frontier capacity can live in fewer regions, subject to data-residency rules. - **Index shards by workspace.** Every query is scoped to one workspace, so partitioning by workspace id gives perfect query locality. Huge monorepos get their own shards. - **Re-indexing is asynchronous.** Saves enqueue changed chunks for embedding workers, so a branch switch delays freshness slightly but never touches completion latency. ## Deep-Dive: Codebase Indexing: Chunks, Embeddings, and the Symbol Graph ![The indexing pipeline: a file save watcher marks dirty files in a local Merkle tree, changed subtrees go to an AST parser and chunker, symbols go to a symbol and reference graph, chunks go to an index sync API that enqueues only content hashes the server has not seen, and an embedding pipeline caches embeddings by content hash and upserts them into a per-repo vector index.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-ai-coding-assistant/03-codebase-indexing.png) ### Why AST-Aware Chunking Fixed 50-line windows cut functions in half and glue unrelated code together, so embeddings are muddy and retrieved chunks are fragments. I would parse each file with an incremental parser such as Tree-sitter and chunk along syntax boundaries: - A function or method is the natural unit, if it fits the chunk budget (around 300 to 500 tokens). - Large classes are split by method, and each chunk carries a header with the file path, class name and enclosing signature, so it makes sense on its own. - Very long functions split at block boundaries; small top-level items merge with neighbours. Unsupported languages fall back to line windows split at blank lines. ### Embeddings and Hybrid Retrieval Each chunk gets an embedding from a code-tuned model, but I would not rely on vectors alone. Code is full of exact identifiers (`parseInvoiceLineItems`, `ERR_QUOTA_EXCEEDED`) that a lexical index matches perfectly and an embedding may blur. So every chunk also goes into a BM25 index, and results are merged with reciprocal rank fusion (RRF). Most workspaces are small enough for brute-force vector search; HNSW pays off on large monorepos. ### The Symbol and Reference Graph Embeddings answer "what code is about this topic", not "who calls this function". So the parser also extracts a symbol graph: definitions, references, imports and inheritance, keyed by qualified name. The context assembler follows its edges to the types used near the cursor or the callers of the function being edited. For agents, "find all callers before changing a signature" is a graph query, not a similarity query. Where the IDE has a language server, I prefer its precise answers and use the graph as fallback. ### Incremental Re-Index with a Merkle Tree Re-embedding a whole repository on every save is out of the question. The extension keeps a Merkle tree of the workspace: each file's hash is its content hash, and each directory's hash is the hash of its children's names and hashes. The server stores the same tree for its copy of the index. Sync becomes a tree comparison: ```python def sync(local, remote_hashes, path=""): """Walk the local Merkle tree, descending only into subtrees whose hash differs.""" if remote_hashes.get(path) == local.hash: return [] # whole subtree unchanged, skip it if local.is_file: return [path] # changed or new file: re-chunk it changed = [] for name, child in local.children.items(): changed += sync(child, remote_hashes, f"{path}/{name}") return changed ``` A branch switch touching 200 of 50,000 files costs a walk down 200 paths. Deletions fall out of the same comparison. Embeddings are also **content-addressed**: a chunk's key is the hash of its normalised text, and the sync API embeds only hashes it has not seen. Teammates on one repo, or one developer on two branches, share almost all embeddings. I also debounce re-indexing so repeated saves of one file collapse into one job. ### Privacy: What Leaves the Machine Enterprise customers ask about this first, so I make the boundary explicit: - **Local only:** the Merkle tree, syntax trees and full file contents. - **Sent for embedding:** chunk text, in transit. In privacy mode the server embeds it and discards the plaintext immediately. - **Stored server-side:** embeddings, content hashes, and line ranges with paths encrypted under a client-held key. Queries return chunk ids, and the **client reads the text from disk** before it enters a prompt. I do not oversell this. Research has shown text can sometimes be partially reconstructed from embeddings, so they are not a cryptographic guarantee. The honest promise is "no plaintext at rest", and the strict option is a fully local index with a small local embedding model, at some cost in quality. Indexes are strictly **per workspace and per organisation**, and the authorisation check runs inside the index service, not only at the gateway, because a cross-tenant code leak is the worst failure this system can have. **First vs later:** first, AST chunking, hybrid retrieval and Merkle sync, because they drive quality and cost. Later, a server-side symbol graph for languages without a good language server, and cross-repository indexing. ## Deep-Dive: The Inline Completion Hot Path ![The inline completion path: keystrokes go through a debounce and cancel stage that first checks a client completion cache, on a miss a local context builder sends prefix and suffix over an HTTP/2 edge to the completion service, which calls a fill-in-the-middle model on the same-session replica that holds the cached prefix, tokens or cache hits render as ghost text, and accept or reject outcomes flow to telemetry.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-ai-coding-assistant/04-inline-completion.png) ### The Latency Budget Ghost text is only useful if it appears before the developer's next keystroke. I set a budget for each stage and treat any stage that exceeds it as a bug: | Stage | Budget (p50) | |-------|--------------| | Debounce after last keystroke | 50 to 75 ms | | Client context build (prefix, suffix, snippets) | under 15 ms | | Network round trip to the regional edge | 20 to 50 ms | | Queue plus prefill on the model | 30 to 60 ms | | Decode first useful line | 60 to 100 ms | | **Total** | **about 200 to 300 ms** | These are targets, not measurements. The levers are clear: a small model, a mostly cached prompt, and a nearby network. ### Debounce and Cancel Firing on every keystroke would multiply traffic and mostly answer prompts the developer has moved past, so a short debounce comes first. The second half is cancellation: when the developer types again, the extension resets the HTTP/2 stream, and the completion service tells the model server to drop the sequence at its next step. Otherwise my 40% cancelled requests would still burn full decode time. ```python class CompletionTrigger: def __init__(self, debounce_ms=60): self.debounce_ms = debounce_ms self.inflight = None async def on_keystroke(self, doc, cursor): if self.inflight: self.inflight.cancel() # resets the HTTP/2 stream await asyncio.sleep(self.debounce_ms / 1000) if doc.version_changed_since(cursor): return # user kept typing if hit := local_cache.lookup(doc, cursor): # typed-through or seen before return render_ghost(hit) prompt = build_fim_prompt(doc, cursor) # local, under 15 ms self.inflight = asyncio.create_task(stream_completion(prompt)) ``` I also skip requests where suggestions are unwanted, for example mid-word when the language server has a precise completion, or after several rejections in a row. ### A Small, Fast Fill-in-the-Middle Model The completion model is small (a few billion parameters or less) and trained for fill-in-the-middle: it sees code before and after the cursor and generates what goes between. The suffix matters: seeing `return total` below tells the model to compute `total` and stops it duplicating a closing brace. A StarCoder-style FIM prompt looks like this: ```text # path: billing/invoice.py # related: billing/models.py class LineItem: amount: Decimal tax_rate: Decimal def invoice_total(items: list[LineItem]) -> Decimal: return total ``` I host it in every region. A frontier model would miss the latency budget and multiply cost per request, which dominates at 300,000 requests per second. ### Context Assembly for Completions The completion prompt is built on the client from the prefix and suffix (trimmed to syntax boundaries), recently edited regions in other files, open tabs ranked by a cheap token-set similarity to the cursor region, and definitions of nearby symbols from the language server. It stays around 1,500 tokens. Remote retrieval is too slow for this path. ### Prefix Caching and Session Affinity Consecutive requests from one developer differ by a few characters. On the same replica, the serving engine reuses the KV cache for the shared prefix and prefills only the new tokens. So I route by session key with bounded-load consistent hashing, and keep stable parts (path header, snippets) at the top of the prompt and the cursor region at the bottom. ### Client-Side Caching and Typing Through Two client tricks remove requests entirely. **Typing through:** if the ghost text is `total = sum(i.amount for i in items)` and the developer types `tot`, the extension trims those characters and keeps showing the rest. And a **small LRU cache** keyed by prompt hash catches backspace-and-retype and undo. ### Acceptance Metrics - **Acceptance rate**: accepted over shown, where shown means displayed long enough to read. - **Retention**: the fraction of accepted characters still in the file 30 seconds and a few minutes later. - **Latency split by outcome**: tells me how much a millisecond is worth. I segment by language and model version, and a new model ships only if it wins on acceptance and retention in an A/B test. **Failure modes:** under overload I shed completions (raise the debounce, skip low-confidence contexts) rather than let latency rise, because a late completion is worse than none. If a region is down, completions simply stop while chat fails over. ## Deep-Dive: Agent Mode: Multi-File Edits, Tool Use, and Sandboxed Execution ![The agent loop: a developer task enters the agent loop, which asks a frontier model for a plan and the next action, checks a step and cost budget, and sends tool calls to a tool router; read, search and edit tools snapshot files to checkpoints before editing, and terminal commands pass through a permission gate before running in a sandbox that executes commands and tests.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-ai-coding-assistant/05-agent-mode.png) ### The Loop An agent is a loop around a model with tools. The model sees the task, the context and the transcript so far, and returns either a final answer or one or more tool calls. The orchestrator executes the tools, appends the results to the transcript and calls the model again. ```python def run_agent(task, ctx, budget): transcript = [system_prompt(), rules_files(ctx), user_turn(task)] for step in range(budget.max_steps): reply = model.call(transcript, tools=TOOLS) budget.charge(reply.usage) if budget.exhausted(): return pause_for_user("Budget reached", transcript) if not reply.tool_calls: return reply.final_answer for call in reply.tool_calls: if call.name == "edit_file": checkpoints.snapshot(call.args["path"]) result = tools.execute(call, permissions=ctx.permissions) transcript.append(tool_result(call, truncate(result, 4000))) if repeating_same_failure(transcript, n=3): return pause_for_user("Stuck on the same error", transcript) return pause_for_user("Step limit reached", transcript) ``` The first tools are small and composable: `read_file`, `search` (hybrid index plus grep), `find_references` (symbol graph), `edit_file`, `run_command` and `list_dir`. Output is compact: a 20,000-line test log is truncated to head, tail and error lines. I would expose tools through MCP so organisations can plug in their own. For multi-file tasks the model first writes a short plan: files to touch and how to verify. The developer can stop a bad plan before any edit, and keeping it near the top of the transcript reduces drift twenty steps later. ### Applying Diffs Safely Rewriting whole files is slow, expensive and risky: the model may silently drop a function. I would have it emit **search-and-replace blocks**, applied deterministically: 1. The search text must match exactly once, otherwise the edit is rejected with an error the model can fix next step. 2. The edit applies to the version the model read. If the developer changed the file since, the model re-reads and redoes it. 3. Afterwards, the parser and language server diagnostics run, and new errors go back in the tool result. An alternative is a small "apply" model that merges a sketched change into the file, at the cost of an extra call. I would start with search-and-replace and add it where edits fail most. ### Commands: Permission Prompts or a Sandbox Model-chosen commands are the most dangerous part of the product, and prompt injection (instructions hidden in a README or dependency) makes it worse. I treat every command as untrusted and offer two modes. **Local mode** runs commands in the developer's terminal behind a permission gate, configurable per workspace: ```yaml # .assistant/permissions.yaml (workspace policy, checked into the repo) commands: allow: # run without asking - "npm test*" - "pytest*" - "git status" - "git diff*" ask: # always prompt the developer - "npm install*" - "git push*" deny: # never run, even if approved in chat - "rm -rf*" - "curl * | sh" network: ask write_outside_workspace: deny ``` String matching alone is weak (a chain can hide a dangerous command), so I parse commands, match each part of a pipeline, and default to "ask" for anything unparseable. **Sandbox mode** runs commands in a gVisor-hardened container or a Firecracker microVM with a repo copy, no developer credentials, and egress limited to package registries. It is the only acceptable mode for background agents. To avoid slow setup, I keep pre-built environment images per repository, rebuilt when lockfiles change. ### Running Tests and Iterating The feedback loop (edit, test, read the failure, fix) is what makes agents better than chat. I keep it cheap: run **targeted** tests first (found through the symbol graph) and the full suite at the end, parse output into structured failures, and stop to ask the developer if the same error repeats three times. ### Checkpoints, Undo and Budgets Before every edit, the orchestrator snapshots files into a **shadow git repository** outside the user's history, so each step is a one-click restore point that never touches their branch or stash. Side effects outside the workspace (migrations, installs) cannot be undone this way, which is why they sit in the "ask" list. Every task also gets a **step limit** (say 25 to 50 tool calls), a **token and cost budget** tied to the user's plan, a **wall-clock limit** for sandboxes, and repetition detection. When a limit trips, the agent pauses with a summary and a "continue" button, so no work is lost. **First vs later:** first, local mode with permission prompts, search-and-replace edits, checkpoints and limits, because it is useful with a small blast radius. Later, cloud sandboxes for background agents, parallel agents on separate branches, and an apply model. ## Deep-Dive: Context Assembly and Prompt Caching ![Context assembly: a chat or agent turn goes to a candidate gatherer that pulls vector and BM25 hits and symbol graph hits, a ranker selects candidates under a token budget, a prompt packer combines them with compacted conversation memory with the stable prefix first, and a model router sends easy requests to a small model pool and hard ones to a frontier model with prompt caching.](https://d5osvdbc8um23.cloudfront.net/static-asset/blog_images/system-design-ai-coding-assistant/06-context-assembly.png) ### Ranking Under a Token Budget I can always gather more candidate context than fits: retrieval hits, symbol definitions, open files, selection, recent edits, terminal output, rules and history. More context costs money, slows the first token, and can bury the relevant snippet. So assembly is ranking under a budget: gather from all sources in parallel, rerank retrieval hits with a cross-encoder, deduplicate overlapping ranges, then pack greedily by score per token, with the selection, the edited file and rules files always included. ```python def pack(candidates, budget_tokens, required): chosen, used = list(required), sum(c.tokens for c in required) pool = sorted((c for c in candidates if c not in required), key=lambda c: c.score / c.tokens, reverse=True) for c in pool: if used + c.tokens <= budget_tokens and not overlaps(c, chosen): chosen.append(c) used += c.tokens return order_for_cache(chosen) # stable items first, volatile last ``` In agent mode the model can search and read files itself, so the initial context can be smaller and it pulls what it needs. ### Stable Prefixes for Prompt Caching Frontier providers offer prompt caching: a request that starts with the same tokens as a recent one is processed faster and billed lower for the cached part. It only works on an **exact prefix**, so I order every prompt from most to least stable: 1. System prompt and tool definitions (identical for every user of this product version) 2. Organisation and repository rules files (stable for weeks) 3. A repository summary or file map (stable for hours) 4. Conversation history and agent transcript (append-only within a task) 5. Freshly retrieved context and the latest user message (changes every turn) In agent mode each step appends to the transcript, so step 20 shares nearly all its tokens with step 19. At my assumed 72 million agent input tokens per second, the cache hit rate decides whether the product is viable. The rules are easy to break: no timestamps or random ids near the top, never reorder tools, never re-render history, and append rather than edit. ### Conversation Memory and Compaction Truncating the oldest turns loses the original task. Instead, at a threshold (say 70% of the window) I **compact**: a model call summarises older turns into a note covering goal, decisions, files changed and open problems. Stale tool outputs go first, since they are most of the tokens. Compaction breaks the cache, so I do it rarely and in large steps. ### Rules Files Teams want to say "use our logger" or "run `make test-fast` to test". A rules file in the repository, with organisation and user rules layered on top, goes in the stable part of the prompt. Rules scoped to a directory are included only when its files are in context. ### Model Routing Not every request needs the most capable model. I route by task: | Task | Model | |------|-------| | Inline completion | Small self-hosted FIM model | | Commit messages, short explanations, rename suggestions | Small or mid model | | Applying a sketched edit to a file | Small apply model | | Codebase chat | Frontier model, or mid model for simple lookups | | Agent planning and multi-file reasoning | Frontier model | | Compaction summaries | Mid model | The router also fails over between providers. I log every decision and evaluate it offline, because a router that cuts cost by quietly lowering quality is a regression. ### Cost per Active Developer The number I would put on the dashboard is **cost per active developer per month**, broken down by mode: ```text cost per dev-month = completion: requests x (fresh prefill tokens x p_in + output tokens x p_out) + chat: turns x (uncached in x p_in + cached in x p_cached + out x p_out) + agent: steps x (uncached in x p_in + cached in x p_cached + out x p_out) + indexing: embedded chunks x p_embed + storage ``` What moves it, in rough order of impact: 1. **Agent prompt cache hit rate.** Agent steps carry the largest prompts, and almost all of each prompt is a repeat. 2. **Agent step count.** Good tools, compact outputs and repetition detection cut wasted steps. 3. **Model mix.** How much work the router can send to cheaper models without hurting quality. 4. **Completion cancellations and skips.** Requests never sent cost nothing. 5. **Context size.** Every unnecessary retrieved chunk is paid for on every turn it stays in the transcript. Heavy agent users cost many times more than light ones, so plans need usage limits, and the agent loop's budget enforcement makes them real. **First vs later:** first, the stable-prefix layout, greedy packing and a rule-based router, because they are cheap with large effects. Later, learned rerankers and routing, and per-repository context budgets tuned on task success. ## Conclusion: Bringing It All Together An AI coding assistant looks like a chat box and a grey suggestion, but it is three systems sharing a core. Inline completion is a latency system: a small model in every region, debounce and cancellation, and session affinity for prefix reuse. Chat is a retrieval system: AST-aware chunks, hybrid search and a symbol graph kept fresh by Merkle sync. Agent mode is a safety and cost system: a tool loop with permission gates or sandboxes, checkpoints, and hard budgets. If I had to summarise the design in four decisions: 1. **Index the code the way a compiler sees it.** Chunk by syntax, keep a symbol graph next to the vectors, and sync only what changed. 2. **Treat completion latency as a budget across the whole path,** from keystroke pause to ghost text, and cancel aggressively. 3. **Never let a model-chosen command or edit run without a gate and an undo.** Permissions, sandboxes and checkpoints are what make agent mode safe to use. 4. **Order every prompt from stable to volatile and route by task.** Prompt caching and model choice decide whether the product makes money. Related reading: [Designing an LLM Inference Platform](/blog/system-design-llm-inference-platform), [AI Agents and Agentic Workflows](/blog/ai-agents-agentic-workflows), [Progressive Disclosure: Designing Context for LLM Agents](/blog/progressive-disclosure-designing-context-for-llm-agents), and [LLM Memory Compaction](/blog/llm-memory-compaction).