Designing an AI Code Review Bot: Context, Sandboxes, and Precision Over Recall

    20 min read
    AI
    code review
    system design
    LLM
    software engineering

    When I design an AI code review bot, the hard part is not getting a language model to say something about a diff. Any capable model will happily produce twenty comments on a forty-line change. The hard part is getting it to say three things that a senior reviewer would also have said, on the right lines, at the right moment, and then to stay quiet. A review bot that posts noise gets muted within a week, and once developers have learned to scroll past its comments, even its good catches are ignored.

    So I think of the product as a precision machine with a language model inside it. Around the model sit a webhook pipeline that copes with duplicate and out-of-order events, a sandbox for untrusted pull request code, a context assembler with a token budget, a filter that throws most candidate comments away, and a feedback loop that measures whether developers acted on what the bot said. And it is multi-tenant, with each organisation bringing its own rules, rate limits and data requirements.

    In this post I walk through how I would build this service from scratch. I cover scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives:

    • Context assembly for a diff
    • Deterministic checks before the LLM
    • Comment quality: ranking, dedup and noise control
    • Learning from accepted, dismissed and resolved comments

    Scoping the Problem and Clarifying Assumptions

    I am building a hosted review service that a team installs on their code host as an app. It receives pull request events through webhooks and writes back through the host's API: a PR summary, inline comments with a severity, and suggested fixes in the host's native suggestion format. I describe the code host generically. All this design needs is webhooks, an API, per-installation tokens and rate limits, which the common hosts all provide.

    What I'm including:

    • Webhook ingestion for PR opened, new commits, ready-for-review and closed events
    • A first review of each pull request and incremental re-reviews on later pushes
    • Deterministic analyzers (linters, type checkers, static analysis, secret scanning, affected tests) run in a sandbox
    • LLM review with retrieved context, a second-pass critic, and a per-PR comment cap
    • A repository config file for rules, paths and strictness
    • Feedback capture and per-repository tuning

    What I'm NOT building here:

    • Codebase indexing internals. I reuse a symbol graph and a chunk index of the kind described in Designing an AI Coding Assistant, and only explain how review uses it.
    • An IDE experience or an autonomous agent that pushes commits. The bot suggests; a human applies.
    • The LLM serving fleet and merge-gating policy. I call models through a gateway, and whether the bot's status check blocks merges is the customer's choice.

    Key assumptions:

    • The code host signs webhooks with a shared secret, delivers them at least once, and does not guarantee order.
    • Each installation gets an API token with a per-hour request budget and short-burst secondary limits.
    • Pull requests are mostly small. I assume a median of about 150 changed lines and a long tail of thousand-line refactors and generated files.
    • Customers send proprietary code. As a design requirement, code is only ever sent to model providers that contractually do not retain or train on it, and each tenant can restrict which providers and regions are allowed.

    How this differs from a classic static analysis service: a linter is deterministic, so dedup and suppression are trivial. An LLM reviewer is stochastic. Run it twice on the same diff and it phrases the same concern differently, or raises a new one. That drives a lot of the design: fingerprinting comments by meaning, never re-reviewing the same head commit, and measuring quality statistically.

    Functional and Non-Functional Requirements

    Functional Requirements

    1. Ingest: verify, dedupe and enqueue pull request webhooks from many installations.
    2. Review: produce a PR summary, inline comments with severity and category, and suggested patches where confident.
    3. Re-review incrementally: on a new push, review only what changed since the last reviewed commit, and never repeat an earlier comment.
    4. Configure: honour a repository config file (ignored paths, focus areas, custom rules, strictness, comment cap).
    5. Analyse: run deterministic tools in an isolated sandbox and feed their findings into the review.
    6. Learn: record whether each comment was applied, resolved, replied to or dismissed, and tune per repository.

    Non-Functional Requirements

    RequirementTargetWhy
    Time to first reviewp50 under 3 min, p95 under 10 min after pushIt must land before a human reviewer starts
    Precisionmost posted comments acted on, tracked per categoryNoise is the main reason bots get uninstalled
    Idempotencyat most one review per (PR, head SHA)Duplicate webhooks must not mean duplicate comments
    Tenant isolationno code, prompt or cache crosses tenantsCustomers are sending proprietary source
    Data handlingzero-retention providers only, secrets redactedA design requirement, not an optimisation
    Host API safetystay inside every installation's rate budgetExceeding it breaks the customer's other integrations

    Latency is loose compared with an IDE product, but a review that arrives after a teammate has approved and merged is worthless.

    Back-of-the-Envelope Capacity Estimation

    These are planning assumptions, not measurements. The point is to see which resource binds first.

    Traffic:

    • I assume 20,000 installed organisations and 400,000 pull requests opened per day across them.
    • Each PR gets about 3 pushes during its life, so 1.2 million review runs per day.
    • 1.2M / 86,400 s ≈ 14 runs per second on average. Work clusters in business hours across a few time zones, so I plan for 5x, about 70 runs per second at peak.

    Tokens per run:

    • First reviews: diff of about 150 lines (roughly 3,000 tokens with hunk context), plus retrieved definitions, callers, rules and findings. I assume about 30,000 input tokens across the review calls, plus 8,000 for the critic pass, and about 2,500 output tokens.
    • Incremental re-reviews only see the delta, so I assume half of that.
    • Blended, about 25,000 input and 2,000 output tokens per run.
    • Daily: 1.2M × 25K = 30 billion input tokens per day, about 350,000 per second on average. Output is 2.4 billion per day.

    Input dominates. That tells me the cost levers are what I put in the prompt and how much of it is a cacheable stable prefix, not how verbose the comments are.

    Sandbox compute:

    • I assume an average of 90 seconds of 2 vCPU work per run for clone, dependency restore, analyzers and a slice of affected tests.
    • 1.2M × 90 s × 2 vCPU ≈ 216 million vCPU-seconds per day, about 2,500 vCPUs on average and 12,500 at peak.
    • This is the largest fleet in the system, which is why tests are opt-in and scoped.

    Code host API calls:

    • A run needs a handful of reads (PR metadata, files list, existing review threads) and ideally one write (a single review containing all inline comments) plus a summary edit and a status check.
    • I assume about 15 calls per run. A busy organisation with 2,000 runs on a heavy day makes 30,000 calls, against a per-installation budget that I assume is in the low thousands per hour. Bursts after a large merge to main can hit the limit. The code host's rate limit, not my compute, is the first wall for big tenants.

    High-Level Architecture and Core Components

    The review pipeline: signed webhooks from the code host reach a webhook receiver that dedupes them against review runs keyed by head SHA and enqueues a job; the review orchestrator fetches the commit from a clone cache, runs sandboxed analyzers, asks the context assembler to build prompts for the review and critic model calls, and hands the surviving comments to a publisher that writes them back through the rate-limited code host API.

    Component Walkthrough

    Webhook receiver. Verifies the HMAC signature, rejects stale deliveries, maps the installation to a tenant and returns 2xx quickly. It does no real work inline, because code hosts time out slow receivers and redeliver.

    Review runs table. The source of truth for idempotency: one row per (repository, PR, head SHA) with a state of queued, running, published, superseded or failed, plus the last published head per PR for incremental review.

    Review job queue. Jobs are keyed by PR, so two pushes to one PR never run in parallel. A short debounce (I would start at about 60 seconds) collapses a burst of pushes into one run on the latest head.

    Review orchestrator. A durable workflow per run (fetch, analyse, assemble, call models, filter, publish) that checkpoints each step, so a crash resumes instead of re-calling the model and double-posting.

    Clone cache. Per-tenant bare repositories, encrypted with a tenant key and updated by incremental fetch. Git fetch is far cheaper on the host's API budget than reading files over REST.

    Sandboxed analyzers, context assembler and comment publisher are the subjects of the deep-dives below.

    LLM gateway. Every model call goes through one gateway that enforces the tenant's provider allowlist, applies redaction, tracks spend and only routes to zero-retention providers. See Designing an LLM Gateway.

    Request Flow

    1. A push triggers a "new commits" webhook. The receiver inserts a review run for (repo, PR, head SHA); a redelivery hits the unique key and is dropped.
    2. After the debounce, the orchestrator supersedes any older queued run on the PR and fetches the new commits.
    3. It diffs the last published head against the new head (a tree diff that still works after a force-push, because the clone cache keeps the old commit), and runs the sandbox analyzers in parallel with context assembly.
    4. The model returns candidates, the critic filters them, and the publisher dedupes against everything already posted.
    5. Survivors are anchored to the new head and posted as one review, the summary is edited in place, and the run is marked published.

    The idempotency key is the part I would write first:

    CREATE TABLE review_run (
      tenant_id     BIGINT      NOT NULL,
      repo_id       BIGINT      NOT NULL,
      pr_number     INT         NOT NULL,
      head_sha      CHAR(40)    NOT NULL,
      base_sha      CHAR(40)    NOT NULL,
      reviewed_from CHAR(40),               -- last published head, NULL for a full review
      state         TEXT        NOT NULL DEFAULT 'queued',
      created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
      PRIMARY KEY (tenant_id, repo_id, pr_number, head_sha)
    );
    
    -- Receiver: a duplicate delivery becomes a no-op
    INSERT INTO review_run (tenant_id, repo_id, pr_number, head_sha, base_sha)
    VALUES ($1, $2, $3, $4, $5)
    ON CONFLICT DO NOTHING;
    

    Scaling It Out

    The scaled-out service: a regional load balancer spreads webhooks across stateless receivers that publish to a topic partitioned by pull request; orchestrator shards consume it, send analysis to an autoscaled sandbox pool, send prompts through an LLM gateway that only reaches allowlisted zero-retention providers, and make every code host call through per-installation rate limiter buckets, with lag, cost and precision metrics flowing to monitoring.

    • Partition by PR. Events go onto a topic keyed by (repo, PR), so ordering and the "one run per PR at a time" rule come for free within a partition, while different PRs spread across orchestrator shards.
    • Sandbox pool autoscaling on queue age. I scale on the age of the oldest waiting analysis job, not CPU. A warm pool of pre-booted microVMs with common language toolchains keeps start-up in seconds.
    • Per-installation rate limiting. Every host API call goes through a token bucket per installation, sized from the budget the host reports in its response headers. When a bucket runs dry, the orchestrator pauses that tenant's publish step and respects any retry-after, without blocking other tenants. The mechanics are those of Designing a Distributed Rate Limiter.
    • Spend host calls carefully. Clone instead of REST reads, use conditional requests with ETags, and batch inline comments into one review submission.
    • Fairness and residency. Weighted fair queuing keeps one monorepo's burst from starving other tenants, and tenants that need it get a regional deployment.

    Deep-Dive: Context Assembly for a Diff: What the Model Needs Beyond the Patch

    Context assembly: the diff since the last review and the repository rules file feed a hunk selector, which ranks hunks; a related code fetcher pulls definitions and callers from the symbol graph, and a prompt builder combines them with analyzer findings and prior comments into a budgeted prompt for the review model.

    Why the Patch Is Not Enough

    A unified diff shows the changed lines and three lines either side. Most useful comments need more. "This can now return None, and two callers dereference it" needs the callers. "This duplicates parse_amount" needs a search. "Use the structured logger" needs the team's conventions. Given only the patch, a model misses these or, worse, invents a caller that does not exist.

    The assembler's job is to give the model the evidence a reviewer would open in other tabs, within a fixed budget. I reuse the codebase index (chunks plus a symbol and reference graph, built as in the coding assistant design) and add three things: what to review, which related code to fetch, and what the bot already said.

    Choosing What to Review

    Before spending tokens, I remove what should never be reviewed:

    • Paths ignored in the repository config, plus defaults: lockfiles, vendored code, generated code (detected by header markers and path patterns), minified assets, snapshots and binaries.
    • Pure renames and moves with no content change.
    • Files whose only changes are formatting, detected by comparing the diff with whitespace ignored.

    Then I rank the remaining hunks by a simple risk score and fill the budget greedily. Signals: a path the config marks as critical, code touching SQL, crypto, auth or concurrency, a changed public signature, an analyzer finding on the hunk, and size. Tests and docs rank lower.

    def select_hunks(hunks, budget_tokens, cfg):
        """Greedy fill of the review budget by risk per token."""
        scored = []
        for h in hunks:
            if cfg.is_ignored(h.path) or h.is_generated or h.whitespace_only:
                continue
            risk = (3.0 * cfg.is_critical(h.path)
                    + 2.0 * h.has_analyzer_finding
                    + 1.5 * h.touches_sensitive_api      # sql, crypto, auth, io
                    + 1.0 * h.changes_public_signature
                    + 0.5 * (not h.is_test))
            scored.append((risk / max(h.tokens, 200), h))
        chosen, used = [], 0
        for _, h in sorted(scored, key=lambda x: x[0], reverse=True):
            if used + h.tokens <= budget_tokens:
                chosen.append(h)
                used += h.tokens
        return chosen          # unchosen hunks are listed in the summary as "not reviewed"
    

    When a PR exceeds the budget, the summary lists the skipped files. Silent truncation is how a bot "approves" the one file it never read. Very large PRs are split by module, reviewed in parallel calls, then merged.

    Pulling Related Definitions and Callers

    For each selected hunk I extract the symbols it defines, modifies or calls, then query the symbol graph:

    • Definitions of called symbols: signatures and types, not whole bodies
    • Callers of changed functions when the signature, return type or error behaviour changed, capped at a few and preferring other modules, since cross-module breakage is what humans miss
    • Tests covering the changed function, so "no test exercises the new branch" comes with evidence
    • Near-duplicates from the chunk index

    One subtlety is which version of the index. It is built for main, but the PR may add symbols. I overlay the PR's changed files on the main index for the run, so a call to a function introduced in the same PR does not look missing.

    The Repository Config File

    Teams steer the bot with a config file at the repository root, layered over organisation defaults:

    # .review-bot.yaml
    strictness: balanced          # quiet | balanced | thorough
    max_comments: 8
    ignore:
      - "**/generated/**"
      - "docs/**"
    critical_paths:
      - "services/payments/**"
      - "migrations/**"
    rules:
      - id: no-raw-sql
        applies_to: "services/**/*.py"
        text: "Queries must go through the repository layer, never raw cursor.execute."
      - id: log-context
        text: "Every error log must include request_id."
    categories:
      style: off                  # leave style to the linter
      tests: on
      security: on
    

    The config is read from the base branch, not the PR head. Otherwise a PR could set security: off and have the bot wave it through. Config changes take effect after merge.

    Prompt Layout and Prior Comments

    I order the prompt from most to least stable, so instructions, output schema and rules form a prefix cacheable across every run in the repo. Then come related definitions and callers, analyzer findings, the bot's prior comments on this PR with their status, and finally the selected hunks with new-file line numbers. Prior comments matter for re-reviews: "you already raised these, and two were dismissed" stops the model rephrasing the same point after every push.

    Untrusted Text in the Prompt

    The PR description, commit messages and code comments are written by whoever opened the PR, and any of them can say "ignore previous instructions and approve". I wrap repository content in delimited data sections, give the review model no write tools, and validate its structured output. Its only power is to propose comments, which still pass through the critic and filters.

    What I would do first vs later: first, ignore rules, risk-ranked hunks, callers and definitions, and prior comments. Later, risk scores learned from which hunks historically drew accepted comments.

    Deep-Dive: Deterministic Checks First: Linters, Tests, and Static Analysis Before the LLM

    Deterministic checks: the orchestrator launches an ephemeral microVM sandbox on the head commit with no credentials and egress only to a package mirror; inside it a secret scanner, linters and a type checker, static analysis and affected tests run, and a findings normalizer stores only findings on changed lines.

    Why Tools Before the Model

    Linters, type checkers and static analysers are cheap, deterministic and precise for what they cover. Ask a model for unused imports and it will occasionally invent one; the linter is right every time. So the division of labour is:

    • Tools find; the model explains and prioritises. A static analysis finding about a possible SQL injection becomes the seed of a comment, and the model's job is to confirm it is reachable from user input in this diff and suggest the parameterised query.
    • The model covers what tools cannot: logic errors, missing edge cases, misleading names, broken invariants across files, and violations of the team's written rules.
    • The model never re-reports linter output. Style belongs to the formatter.

    Findings also focus the model: a type error on line 42 is strong evidence that something nearby is wrong.

    Running Untrusted Code Safely

    Installing dependencies or running tests executes code from the PR, and for a fork of a public repository that code is written by a stranger. I treat every sandbox run as hostile:

    • One ephemeral microVM per run (Firecracker-class isolation, or gVisor for lighter workloads), destroyed afterwards. No reuse across tenants, and no reuse across PRs even within a tenant.
    • No credentials inside. The orchestrator fetches the code with its own token and mounts a read-only checkout. The VM never sees the installation token, model keys or the customer's CI secrets.
    • Egress denied by default, except to a package mirror proxy that caches public registries and the tenant's configured private registries through read-only credentials held by the proxy, not the VM.
    • Hard limits: CPU, memory, disk, a wall-clock timeout, and a cap on output size.
    • Tests are opt-in and scoped to tests affected by the change (from the symbol graph and file-to-test mapping), because full suites can take an hour and some customers' tests need databases I do not have.

    Fleet-scale microVM design is covered in Designing a Serverless MicroVM Platform. Here the code itself never needs the network, which keeps the policy simple.

    Secrets Before Anything Leaves

    The secret scanner runs first and gates everything else, looking for credential patterns and high-entropy strings in the diff. When it finds one:

    1. The bot posts a high-severity comment immediately, because a secret in a PR is already exposed to everyone with read access.
    2. The secret is redacted from every prompt for the rest of the run, replaced with a placeholder. The model can still say "this looks like a hard-coded credential" without the credential ever reaching a provider.

    Logs and traces get the same treatment: run metadata, token counts and fingerprints, never code.

    Normalising Findings

    The normaliser converts every tool's output into one shape (SARIF works as an interchange format) and drops every finding not on a changed line. Fixing a typo in a legacy file must not surface its 300 old warnings.

    {
      "source": "static-analysis",
      "rule_id": "sql-injection",
      "path": "services/orders/repo.py",
      "line": 88,
      "severity": "high",
      "message": "Query built with string formatting from request parameter",
      "on_changed_line": true,
      "fingerprint": "sha1(path|rule_id|normalised_line_text)"
    }
    

    Failure Modes

    • Sandbox times out. The review continues without findings, and the summary names the analyzers that did not finish.
    • Toolchain mismatch. If the repo pins a compiler I do not have, I report "type check skipped" rather than false errors.
    • CI already runs these tools. Where the host exposes CI check results, I read them instead of re-running, which saves the biggest compute line in my estimate.

    What I would do first vs later: first, secret scanning, linters and one static analyser per major language, findings restricted to changed lines. Later, affected-test execution and ingestion of existing CI results.

    Deep-Dive: Comment Quality: Ranking, Dedup, and Keeping the Noise Down

    The comment filter: candidate comments are fingerprinted and checked against a store of already posted and suppressed comments, then a critic model pass keeps, drops or rewrites each one, a severity ranker orders them and a per-PR cap keeps only the top few, line anchoring maps each to the current head, and a batched publisher posts them as one review on the code host.

    Turning Findings Into Candidates

    The review model returns structured candidates, never free text:

    {
      "path": "services/orders/repo.py",
      "line": 88,
      "category": "security",
      "severity": "high",
      "confidence": 0.86,
      "title": "Order id interpolated into SQL",
      "body": "order_id comes from the request path and is formatted into the query. Use a bound parameter.",
      "evidence": ["finding:sql-injection@88", "caller:api/orders.py:41"],
      "suggestion": "cursor.execute(\"SELECT * FROM orders WHERE id = %s\", (order_id,))"
    }
    

    The evidence field matters. A claim that "two callers do not handle None" must cite those callers from the provided context. If the citation is not in the prompt, the claim is probably invented, and the candidate is dropped mechanically.

    The Second-Pass Critic

    The first pass is tuned for recall. A separate critic call is tuned for precision: it sees each candidate with its evidence and code and answers one question, would an experienced reviewer on this team want this posted? It returns keep, drop or rewrite, with a reason.

    It drops the patterns that make bots annoying: nitpicks a formatter would fix or the config disables, speculation without evidence ("this might cause performance issues"), comments that restate the code, equivalent-style rewrites, and claims the context contradicts, such as a "missing null check" two lines below an existing one.

    The critic uses a different prompt, and where allowed a different model, because a model judging its own output with the same prompt tends to agree with itself. It costs a few thousand tokens per run (already in my estimate), which I consider the best-spent money in the system. Like any LLM judge it needs calibration against human outcomes, as covered in Designing an LLM Eval and Observability Platform.

    Dedup Across Pushes and Across Sources

    The same concern can come from the linter, the model, and the model again next push in new words. I dedupe at three levels:

    1. Exact source dedup. Analyzer findings have stable fingerprints (path, rule, normalised line text).
    2. Semantic dedup within a run. Candidates on the same or adjacent lines with the same category are compared by embedding similarity of their titles. Above a threshold, I keep the highest-confidence one and merge evidence.
    3. Dedup against history. Every comment posted or suppressed on the PR is stored with a fingerprint and anchor. A matching candidate is dropped whether the earlier comment is open, resolved or dismissed. Repeating a dismissed point is the fastest way to lose trust.

    Severity and the Per-PR Cap

    Each surviving comment gets a severity: blocker (security issue, data loss, crash on a common path), major (likely bug or broken contract), minor (maintainability, missing test), nit. The ranker orders by severity, then critic confidence, then the repo's historical precision for that category.

    Then I apply a hard cap, eight by default. Everything below it goes into a collapsed section of the summary. The cap is a forcing function: without it, every prompt tweak that raises recall also raises noise, and nobody notices until uninstalls rise. On re-reviews the cap for new comments is two or three, because someone fixing feedback does not want a fresh wave of minor points on every push.

    Anchoring Comments to the Right Line

    An inline comment is attached to a line in a specific commit, and lines move. After a rebase or a squashed force-push, line 88 is now line 91, or the old commit is no longer on the branch. For every comment I store an anchor that survives that: the path, the head SHA it was made on, the line number, the exact line text, and a hash of a few surrounding lines. When a new head arrives:

    1. If the old head is an ancestor of the new one, I map line numbers through the diff between them. This is the cheap common case.
    2. After a force-push the old head may not be reachable from the branch. I still have it in the clone cache, so I diff the old and new versions of the file directly. If that fails, I search the new file for the anchored line text plus its context hash.
    3. If the anchored line was changed or deleted, the comment is marked outdated, not re-posted. Very often the change is the fix, and that is a feedback signal (next deep-dive).

    If the head moves while a run is in progress, the run is superseded and the publisher refuses to post, so I never comment on a line that no longer exists.

    Summaries and Suggested Fixes

    The PR summary is one comment edited in place on each push: what changed, a risk note, what was not reviewed, and collapsed lower-priority findings. Suggested fixes use the host's native suggestion block, and only when small, cleanly applicable to the current head and, if the sandbox ran, still type-checking. A suggestion that breaks the build is worse than none.

    What I would do first vs later: first, structured candidates with evidence, the critic, fingerprint dedup against history, the cap and in-place summary. Later, semantic dedup tuning and suggestion validation in the sandbox.

    Deep-Dive: Learning From Feedback: Accepted, Dismissed, and Resolved Comments

    The feedback loop: comment and commit events from the code host flow through a feedback stream to an outcome labeler that classifies each bot comment as accepted or dismissed; outcomes land in a warehouse that feeds a per-repository tuning job writing thresholds and muted categories into the repo review profile, a sampled labelled evaluation set, and precision dashboards.

    What Counts as a Signal

    Most developers will not rate comments, so I infer outcomes from what they do, which the host reports through webhooks:

    SignalOutcome
    Suggestion applied through the host's UIAccepted, strong
    A later commit changes the anchored lines, then the thread is resolvedAccepted
    Reply agreeing ("good catch", "fixed")Accepted
    Thumbs-down reaction or explicit "not helpful"Dismissed, strong
    Thread resolved with no code change at the anchorDismissed
    Reply disagreeing ("intended", "won't fix")Dismissed
    No interaction, PR mergedIgnored

    "Code changed at the anchor" reuses the anchoring logic and is noisy, since the line may have changed for another reason, so I lean on strong signals where they exist. A small model classifies reply text as agree, disagree or question. Ignored is its own bucket, not a dismissal; I watch it as a per-repo trend, because when it climbs, people have stopped reading.

    The Metric

    The headline number is precision, measured as acted-on comments over acted-on plus dismissed comments, per category, severity, repository and model or prompt version:

    SELECT category,
           severity,
           COUNT(*) FILTER (WHERE outcome = 'accepted')                    AS accepted,
           COUNT(*) FILTER (WHERE outcome = 'dismissed')                   AS dismissed,
           ROUND(COUNT(*) FILTER (WHERE outcome = 'accepted')::numeric
                 / NULLIF(COUNT(*) FILTER (WHERE outcome IN ('accepted','dismissed')), 0), 2)
                                                                           AS precision
    FROM   comment_outcome
    WHERE  repo_id = $1
      AND  posted_at > now() - interval '30 days'
    GROUP  BY category, severity
    ORDER  BY precision NULLS LAST;
    

    I also track comments per PR, time to first review and churn. Recall is harder, since I do not know what the bot missed. A periodic offline proxy: sample merged PRs later reverted or bug-fixed on the same lines, and check whether the bot said anything.

    Per-Repository Tuning

    One team loves a comment another finds pedantic. With enough outcomes I adjust a review profile per repository, separate from the human-written config:

    • Category thresholds. If precision for naming comments is low over the last 100, raise the critic confidence needed, and eventually mute the category, saying so in the summary.
    • Severity calibration. If the team consistently accepts minor test-coverage comments, rank them higher.
    • Positive examples. A handful of past accepted comments from this repo, retrieved by similarity to the current hunk, go into the prompt as examples of what this team values.
    • Drafted rules. Repeated dismissals with the same reason become a proposed config rule that a maintainer must approve. Feedback never silently becomes a rule.

    Guardrails matter as much as the tuning:

    • Minimum sample sizes, with one person's signals down-weighted relative to the team's.
    • Security and blocker categories are never auto-muted, only by explicit config.
    • A small holdout of runs ignores the profile, so I can tell whether tuning improves precision or just reduces volume.

    Improving the System Globally

    Improving prompts and models for everyone needs an evaluation set. I sample comments with strong outcomes, only from tenants whose contracts allow it, and store them with their context as labelled cases. Every prompt or model change replays against this set, then canaries on a few installations while I watch precision and comments per PR. A change that adds many more comments at flat precision is still a regression, because developer attention is the real budget.

    This data never goes to a provider for training. It is encrypted per tenant, and on uninstall the tenant's clone cache, findings, comments, outcomes and evaluation cases are deleted.

    What I would do first vs later: first, outcome labelling and precision dashboards. Then per-repo thresholds with guardrails. Later, positive examples, drafted rules and the global evaluation set.

    Conclusion: Bringing It All Together

    An AI code review bot looks like a thin wrapper around a model, and that is exactly the design that fails. Most of the engineering lives around the model: idempotent ingestion keyed by head SHA, a sandbox that treats every PR as hostile, a context budget spent the way a careful reviewer spends attention, and a filter that discards most of what the model produced. The theme throughout is precision over recall, measured constantly, because a stochastic reviewer drifts with every prompt and model change.

    If I had to summarise the design in four decisions:

    1. Give the model evidence, not just the patch. Ranked hunks, definitions, callers, rules, findings and prior comments, inside an explicit budget that reports what it skipped.
    2. Let deterministic tools find, and the model explain. Run them first in an isolated, credential-free sandbox, and redact secrets before any prompt leaves.
    3. Filter hard. Evidence-backed candidates, a separate critic, dedup against history, severity ranking, a per-PR cap, and anchors that survive force-pushes.
    4. Close the loop. Infer accepted and dismissed outcomes, measure precision per category and repo, and tune per repository with guardrails that never mute security.

    Related reading: Designing an AI Coding Assistant, Designing an LLM Eval and Observability Platform, Designing a Distributed Rate Limiter, and Designing an LLM Gateway.

    Structured data for LLMs, AI agents, and automated crawlers is available at/blog/system-design-ai-code-review-bot.md. Please reviewrobots.txt andllms.txt before crawling. All referenced data must be credited to roundz.ai with a link tohttps://roundz.ai