Designing an AI Recruiting Agent: From Job Description to Shortlist with Humans in the Loop

When I design an AI recruiting agent, I start from an uncomfortable fact: almost every useful action it takes touches a real person. A search over candidate profiles is harmless. An email to 300 strangers is not, and neither is a rejection or a shortlist that quietly filters out everyone over 45. A coding agent that makes a mistake produces a bad diff that someone reviews. A recruiting agent that makes a mistake sends something that cannot be unsent, to someone who never asked to hear from you.

So the design problem is not "how do I get an LLM to read job descriptions". Models already do that well. The problem is building a long-running, event-driven hiring pipeline where the agent does the tedious work (reading hundreds of profiles, drafting outreach, chasing replies, booking screens) while a recruiter keeps control of the risky steps, and where every decision can be explained afterwards to the recruiter, the candidate and an auditor.

In this post I walk through how I would build it: scoping, requirements with capacity estimates, a high-level architecture, and four deep-dives:

• Sourcing and matching: building the candidate pool and ranking it
• A durable hiring workflow with an event inbox and approval gates
• Outreach at scale: consent, rate limits, deliverability and replies
• Audit trail, fairness and explainable decisions

Scoping the Problem and Clarifying Assumptions

I am building the employer-facing agent: a recruiter opens a job, and the agent takes it from job description to a proposed shortlist. The agent proposes and executes; the recruiter approves at defined gates.

What I'm including:

• Turning a job description into a structured requirement profile and a hiring plan
• Sourcing from inbound applicants, the employer's talent pool, referrals and consented external sources, then ranking
• Personalised email outreach, follow-ups and reply handling
• Scheduling a screening step (for example an AI interview or a recruiter call) and reacting when it completes
• A shortlist with evidence, approval gates for plan, outreach, shortlist and closing the job, and an audit trail with bias monitoring

What I'm NOT building here:

• Resume parsing and matching internals. I treat them as a service that returns structured profiles and per-requirement evidence.
• The screening system itself. It receives a booking and later emits a "screening completed" event.
• A full applicant tracking system (ATS). The agent integrates with the employer's ATS rather than replacing it.
• Offers, background checks and onboarding. The agent stops at a human-approved shortlist.
• Generic agent infrastructure (model gateway, tool runtime), as described in Designing a Hosted AI Agent Platform.

Key assumptions:

• Multi-tenant SaaS: each employer is a tenant, and one tenant's private pool is never visible to another.
• Recruiters work business hours, so approvals take hours or days. The pipeline must be comfortable waiting.
• A job stays open about a month, and the agent works on it in bursts: a sourcing pass, an outreach wave, replies, a screening week.

How this differs from a classic ATS: an ATS is a system of record. Humans move candidates between stages and the software stores the result. Here the software acts first: it decides who to contact, writes the words, and moves candidates unless someone stops it. That shifts the main risk from "the data is wrong" to "an action was wrong", which is why approval gates, idempotent side effects and an audit log sit at the centre of the design.

Functional and Non-Functional Requirements

Functional Requirements

1. Plan: produce a requirement profile and hiring plan from a job description, and wait for approval or edits.
2. Source and rank: build a deduplicated, eligible candidate pool and rank it with evidence per requirement.
3. Outreach: send personalised messages for an approved batch within consent and rate rules, follow up, stop on reply.
4. React to events: classify replies, and handle new applications and completed screenings from other systems.
5. Shortlist and close: propose a shortlist with reasons; on close, stop outreach and update remaining candidates, both after approval.
6. Explain and protect data: show why any candidate was contacted, ranked or not progressed, and honour opt-outs and erasure.

Non-Functional Requirements

What is missing is telling: there is no tight latency target on the agent. A ranking pass that takes four minutes is fine. Sending the same email twice is not. I trade speed for correctness almost everywhere.

Back-of-the-Envelope Capacity Estimation

These are planning assumptions, not measurements. The goal is to find which resource matters.

Jobs: 3,000 tenants with 10,000 jobs open in agent mode. Jobs stay open about 30 days (about 330 new per day), and each open job gets a weekly re-sourcing pass (10,000 / 7 ≈ 1,430 per day): about 1,800 ranking passes per day.

LLM ranking: each pass retrieves about 2,000 candidates cheaply and sends the top 300 to an LLM reranker: 1,800 × 300 ≈ 540,000 rerank calls per day, about 6 per second. At about 3,000 input tokens each, that is roughly 1.6 billion input tokens per day, the largest variable cost in the system. So I rerank only after cheap filtering, use a smaller model for the first pass, and cache grades per (candidate profile version, requirement profile version) so weekly passes only rerank what changed.

Outreach: about 100 first-touch messages per job over two weeks: 10,000 × 100 / 14 ≈ 71,000 per day, up to about 200,000 with follow-ups. Spread over 8 business hours, about 7 sends per second. Throughput is trivial; the real limits are per-mailbox and per-domain sending rates.

Events and workflow jobs: about 135,000 inbound events per day (applications, replies, completed screenings), under 2 per second. Budgeting generously, each event and agent step becomes workflow jobs totalling about 2 million rows per day: about 23 claims per second on average, perhaps 100 at peak. That is well within a single Postgres primary's range for short transactions, though I would load-test the claim query before relying on it.

State: a candidate index of 20 million profiles at about 7 KB each (text plus a 1,024-dimension float32 embedding) is about 140 GB. Active candidate-job rows: 10,000 jobs × up to 2,000 ≈ 20 million. Audit events at about 1 KB each come to roughly 20 GB per day, so monthly partitions and archival to object storage.

High-Level Architecture and Core Components

Component Walkthrough

Recruiter console. Creates jobs, shows approvals and explanations. Each approval states exactly what will happen ("send these 120 messages from this mailbox over 3 days, with this template"), never a vague "continue?".

API Gateway and hiring agent API. The gateway authenticates, resolves the tenant and enforces roles (who may approve outreach or close a job). The API records jobs and approvals and writes state plus the next workflow jobs in one transaction. It runs no agent logic itself.

Postgres. One database holds per-job and per-candidate pipeline state, the workflow job table with leases, approval gates and the event inbox. Keeping them together is deliberate, as the second deep-dive explains.

Agent workers. Stateless processes that claim a job, run one bounded step (plan, rank a batch, draft a batch, classify a reply, build a shortlist), write the result and next jobs, and release the lease. An orchestrator picks the next step; specialist agents for sourcing, outreach, scheduling and shortlisting each get a narrow tool set.

LLM via a model gateway. Per-tenant budgets, prompt and version logging, and model versions pinned per step type.

Candidate index. Hybrid keyword and vector search over the profiles a tenant may see.

Outreach sender. The only component allowed to send email. It checks suppression and consent, applies rate limits and sends with an idempotency key.

Event inbox. Where everything external lands first: ATS applications, completed screenings, email replies, and recruiter approvals.

Audit log. An append-only record of every action, model version and human decision, used for explanations and fairness monitoring.

Request Flow

1. A recruiter creates a job. The API stores it in DRAFT and enqueues a plan job.
2. A worker extracts a requirement profile and drafts a plan. The job moves to PLAN_PENDING and a gate opens.
3. The recruiter turns one must-have into a nice-to-have and approves. The approval lands in the inbox, and the dispatcher enqueues sourcing.
4. Workers retrieve, filter and rank candidates in batches, then propose an outreach batch at the outreach gate.
5. After approval, the sender delivers messages over a few days within rate limits.
6. Replies land in the inbox and are classified. Interested candidates get a screening invite; an "unsubscribe" suppresses the address at once.
7. As screening results arrive, the orchestrator proposes a shortlist at the shortlist gate. The recruiter approves five candidates for the normal interview loop.
8. Later the recruiter asks to close the job. After the close gate, outreach stops and remaining candidates get a courteous update.

Scaling It Out

• Separate worker pools by work type. Ranking and drafting are slow and LLM-bound; sending and event dispatch are fast and IO-bound. Shared, one big ranking pass would starve reply handling. Each job row has a queue column and each pool claims only its own.
• Console reads from a replica. Dashboards tolerate a second of lag. Approvals and transitions read from the primary.
• Partition by tenant. Candidate-job state and audit events are partitioned so one tenant's export or erasure touches a bounded set of partitions.
• Fairness between tenants. A tenant that opens 500 jobs at once must not delay everyone's replies, so claims rotate across tenants and LLM budgets are enforced per tenant.
• When Postgres is no longer enough, I would first split tenants into cells with their own databases, and only then consider a dedicated workflow engine.

Deep-Dive: Sourcing and Matching: Building the Candidate Pool and Ranking It

Start with a Requirement Profile, Not the Job Description

Job descriptions mix real requirements ("production Kubernetes experience") with boilerplate ("passionate team player"). Matching raw text rewards candidates who echo the wording. So the first step extracts a structured requirement profile, which the recruiter approves as part of the plan:

{
  "job_id": "job_8f2c",
  "profile_version": 3,
  "level": "senior",
  "must_have": [
    {"id": "R1", "skill": "Backend services in Go or Java", "min_years": 4},
    {"id": "R2", "skill": "Operating Postgres in production"}
  ],
  "nice_to_have": [{"id": "N1", "skill": "Event-driven systems (Kafka or similar)"}],
  "constraints": {"location": "Bengaluru or remote within India", "work_authorization": "required"},
  "excluded_signals": ["name", "photo", "age", "gender", "graduation_year", "address"],
  "approved_by": "recruiter_42"
}

The profile is versioned: every ranking records which version it used, so when a must-have changes I know exactly which rankings are stale. It also lists signals the ranker must never see, enforced by the pipeline rather than requested in a prompt (more in the fourth deep-dive).

Where Candidates Come From

Each source carries its own basis for contact, stored on the candidate's source record:

• Inbound applicants applied to this job, so contact about it is expected.
• The talent pool (past applicants, silver medalists, talent-community members): contact depends on what they agreed to when their data was collected.
• Referrals, with the referrer's note as extra evidence.
• External sources only where the candidate opted in to be discovered or a partner API's terms allow it. I do not scrape sites whose terms forbid it.

Before indexing, identity resolution (normalised email and phone hashes, plus fuzzy name and employer matching) merges the same person from three sources into one candidate. Without it, the agent emails one person three times, which is the fastest way to look like spam.

Two-Stage Retrieval and Ranking

1. Retrieve about 2,000 candidates from the hybrid index. BM25 catches exact skills and tools; vector search catches paraphrases ("built payment ledgers" for "financial systems"). Reciprocal rank fusion merges the lists without calibrating scores between them.
2. Filter deterministically, in code: hard constraints, consent, suppression, "already in this pipeline", and "contacted by this tenant recently".
3. Rerank the top 300 with an LLM that grades each candidate against each requirement (strong, partial, none, unclear) and must quote the profile text behind each grade.

The reranker never outputs one opaque score. Plain code computes the final score from the grades and recruiter-set weights. So every rank is explainable, weights can change without rerunning the model, and a model upgrade shows up as a diff per requirement rather than a mysterious reshuffle. The console shows tiers (meets all must-haves, partial, stretch) instead of raw scores, because candidate 14 and candidate 15 are not meaningfully different, and outreach plans target tiers rather than a top-K cut-off.

Failure Modes

• Keyword echo. Evidence that matches the job description word for word gets flagged for review.
• Stale or thin profiles. I show profile age and down-weight old evidence. unclear is a valid grade, and the plan can turn an unclear must-have into a screening question instead of a rejection.
• Feedback loops. Training a ranker on which candidates recruiters advanced learns past preferences, including biased ones. I would hold off on learned ranking until fairness monitoring is in place, and then train on screening outcomes rather than clicks.

What I Would Build First vs Later

First: hybrid retrieval, deterministic filters, rubric reranking with evidence, and caching by profile versions. Later: a cheaper first-pass reranker distilled from the LLM's grades, new consented sources, and "more like this" queries, carefully constrained so they do not become "more people who look like this person".

Deep-Dive: A Durable Hiring Workflow: Long-Running Jobs, Event Inbox, and Approval Gates

Generic durable agent runs (checkpoints, resuming on another worker, durable waits for humans) are covered in Designing a Hosted AI Agent Platform, and workflow engines in Designing a Workflow Orchestration Engine. Here I focus on what recruiting adds: explicit pipeline state, external events, and gates a recruiter can trust.

Pipeline State: Two State Machines

The core of the system is not the agent's conversation history. It is two explicit state machines in the database.

Per job: DRAFT → PLAN_PENDING → SOURCING → OUTREACH → SCREENING → SHORTLIST_PENDING → SHORTLISTED → CLOSE_PENDING → CLOSED, with PAUSED reachable from any active state.

Per candidate in a job: SOURCED → RANKED → OUTREACH_QUEUED → CONTACTED → REPLIED → SCREEN_INVITED → SCREENED → SHORTLISTED, with terminal states NOT_PROGRESSED, WITHDREW, OPTED_OUT and NO_RESPONSE.

The agent never writes a status directly. It calls a transition function that checks an allowed-transitions table inside one transaction. "Contact a candidate who opted out" is not a prompt-engineering problem; it is a constraint violation that fails the transaction. The LLM decides which transition to request; the database decides whether it is legal. As a bonus, the pipeline view is a GROUP BY status, and "why is this job stuck?" is a query rather than a transcript to read.

Why a Postgres Job Table Is Enough Here

Each unit of agent work is a row, claimed with a lease:

CREATE TABLE workflow_jobs (
  id            BIGSERIAL PRIMARY KEY,
  tenant_id     UUID        NOT NULL,
  hiring_job_id UUID        NOT NULL,
  queue         TEXT        NOT NULL,           -- 'llm' or 'io'
  kind          TEXT        NOT NULL,           -- 'rank_batch', 'send_batch', ...
  dedupe_key    TEXT        NOT NULL UNIQUE,    -- e.g. 'rank:job_8f2c:profile_v3:batch_7'
  payload       JSONB       NOT NULL,
  run_after     TIMESTAMPTZ NOT NULL DEFAULT now(),
  lease_owner   TEXT,
  lease_until   TIMESTAMPTZ,
  attempts      INT         NOT NULL DEFAULT 0,
  status        TEXT        NOT NULL DEFAULT 'ready'  -- ready, leased, done, dead
);
CREATE INDEX ON workflow_jobs (queue, run_after) WHERE status IN ('ready', 'leased');
-- Hard guarantee: at most one leased job per hiring pipeline
CREATE UNIQUE INDEX one_lease_per_pipeline ON workflow_jobs (hiring_job_id) WHERE status = 'leased';

-- Claim the oldest runnable job; expired leases are reclaimable
UPDATE workflow_jobs j
SET    status = 'leased', lease_owner = $1,
       lease_until = now() + interval '5 minutes', attempts = attempts + 1
WHERE  j.id = (
  SELECT id FROM workflow_jobs c
  WHERE  c.queue = $2 AND c.run_after <= now()
    AND  (c.status = 'ready' OR (c.status = 'leased' AND c.lease_until < now()))
    AND  NOT EXISTS (SELECT 1 FROM workflow_jobs o
                     WHERE o.hiring_job_id = c.hiring_job_id AND o.id <> c.id
                       AND o.status = 'leased' AND o.lease_until >= now())
  ORDER BY c.run_after
  FOR UPDATE SKIP LOCKED
  LIMIT 1)
RETURNING j.*;

FOR UPDATE SKIP LOCKED lets many workers claim concurrently without blocking. The NOT EXISTS clause skips pipelines that already have work in flight; because that check can race, the partial unique index is the real guarantee, and a worker whose claim hits it simply claims something else. So two workers never reason about the same pipeline at once. The unique dedupe_key makes enqueueing idempotent. A dead worker's lease expires and the job is reclaimed; on completion a worker commits only if it still owns the lease, so a slow worker cannot overwrite a newer result.

I would choose this over Temporal or a similar engine here, and it is a real trade-off.

What I gain: a state transition (candidate moves to CONTACTED) and the jobs it causes (a follow-up in 4 days) commit in one transaction, with no dual write between an engine and my database. At about 100 claims per second, Postgres is not the bottleneck, and it is one fewer system to run. And "all jobs waiting on a recruiter for over two days" is plain SQL.

What I give up: no workflow-as-code with deterministic replay. I own timers (run_after), retries (attempts with backoff) and versioning (migrating state machines carefully), and workers poll the table, which is cheap at this scale with back-off and LISTEN/NOTIFY as a wake-up hint, but not forever. I would revisit the choice if volume grew by one or two orders of magnitude, workflows needed complex fan-out across services, or many teams needed to author their own workflows.

The Event Inbox

The pipeline is driven as much by the outside world as by the agent, and all of it goes through one event inbox:

• Unique key per event, (source, external_id), for example ('screening', 'session_9931'). Senders retry webhooks; the constraint turns duplicates into no-ops.
• Write fast, process later. The webhook handler validates the signature, inserts and returns. A slow model call never makes a sender time out and retry.
• Same-database sources (such as an ATS sync table) feed the inbox through a trigger or transactional outbox, so the event commits with the change that caused it.
• The dispatcher maps each event to a hiring job and enqueues work with a dedupe_key derived from the event key. Since work is serialised per pipeline, a reply and a screening result for the same job are applied one at a time.
• Unroutable events (a reply to an unknown thread, an event for a closed job) go to a review queue, not the floor.

Treating recruiter approvals as inbox events too is a small decision with a large payoff: one path wakes the pipeline, with the same deduplication, ordering and audit for everything.

Approval Gates

A gate is a row: what is being approved, a snapshot of it, who may approve, and when it expires.

Three rules make gates trustworthy:

1. Approve a snapshot, not a pointer. The gate stores a hash of the batch (candidate ids, template version, sender). The send step refuses to run unless its batch hash matches an approved gate. Adding ten candidates later means a new gate.
2. Enforce in the tool layer. The sender requires an approved gate id per batch. The agent cannot talk its way past it, and neither can a prompt injection hidden in a candidate's profile.
3. Edits are first-class. The recruiter can remove candidates or rewrite the template; the edited version is what gets approved, and the diff is logged.

Gates also expire: approval from three weeks ago is not consent to send today. A tenant may later relax a low-risk gate (say, auto-approving follow-ups to people already contacted), but only as an explicit, logged policy, and never for first contact or the shortlist.

First vs later: first the two state machines, the leased job table, the inbox with unique keys, and four gates enforced in the tool layer. Later, tenant policies for low-risk gates, LISTEN/NOTIFY wake-ups, and cells when claim volume demands it.

Deep-Dive: Outreach at Scale: Consent, Rate Limits, Deliverability, and Replies

General notification delivery is covered in Designing a Notification System. Recruiting outreach adds three twists: the recipient may never have heard of you, consent varies per candidate and source, and every send is a step in a conversation someone may answer.

Consent, Suppression and Erasure

Before every send, at send time and not only at planning time, the worker checks:

• Basis for contact on the source record. None recorded means no send.
• Suppression: unsubscribes, complaints, hard bounces and erasure requests, at two levels: per tenant ("not about this employer's roles") and platform-wide ("not at all").
• Candidate status: someone who withdrew or was shortlisted an hour ago gets no follow-up.

Outreach spreads over days, so a candidate who unsubscribes on Monday must not get the Wednesday follow-up approved last Friday. Every message has a one-click unsubscribe, and it updates suppression synchronously, before the confirmation page renders.

For erasure, I delete or irreversibly anonymise the candidate's profile and pipeline records, and keep a salted hash of the email in suppression so a later sourcing pass does not re-import and re-contact them. Whether that hash may be kept, and for how long, is for the employer's privacy counsel to decide per jurisdiction; the system supports both choices.

Rate Limits at Four Levels

The per-candidate cap is the one people forget. If a tenant has ten roles that fit the same engineer, the agent should pick the best one or combine them, not send ten emails. Each limit is a token bucket (as in Designing a Distributed Rate Limiter); a send without a token is rescheduled, not dropped. Sends also go out in the candidate's local business hours when I know their time zone.

Deliverability

A well-targeted message in the spam folder does nothing. So: SPF, DKIM and DMARC aligned for whichever domain sends (the employer's own or a dedicated recruiting subdomain); warm-up with low caps that rise over weeks; hard bounces suppressed immediately; and a rising complaint rate pausing that sender and alerting the recruiter. LLM drafting helps by citing something real from the candidate's profile instead of generic praise, but every draft is checked against the approved template's facts (title, location, any stated pay range) so the model cannot invent perks.

Idempotent Sending

Each planned message has a deterministic key, (hiring_job_id, candidate_id, sequence_step):

def send_step(msg_key: str):
    msg = db.lock_message(msg_key)                 # SELECT ... FOR UPDATE
    if msg.status in ("SENT", "SKIPPED"):
        return                                     # retry is a no-op
    if msg.status == "SENDING":                    # crashed mid-send last time
        if provider.lookup(idempotency_key=msg_key):
            db.mark(msg_key, "SENT"); return
    reason = policy.blocked(msg.candidate_id, msg.tenant_id)   # suppression, consent, status
    if reason:
        db.mark(msg_key, "SKIPPED", reason=reason); return
    if not limiter.try_take(msg.sender, msg.domain, msg.candidate_id):
        db.reschedule(msg_key, run_after=limiter.next_slot()); return
    db.mark(msg_key, "SENDING"); db.commit()
    provider.send(msg.render(), idempotency_key=msg_key)
    db.mark(msg_key, "SENT")

The SENDING state plus the provider lookup turns at-least-once retries into at-most-once delivery in effect. If a provider offers neither idempotency keys nor lookup, I mark the message for manual review instead of guessing: a missed follow-up is cheaper than a double send.

Replies

Replies arrive by webhook or a connected mailbox, are matched to the thread by headers, and a classifier labels the intent:

• Interested: stop the sequence, send the screening invite (an AI interview link or slot picker), set a reminder job.
• Not now / not interested: stop, record a recontact date if given, or mark NOT_PROGRESSED for this job only.
• Question: answer from the approved job FAQ if possible, otherwise route to the recruiter.
• Unsubscribe in words ("please remove me"): an opt-out, even without the link.
• Auto-reply: ignored for intent; the next follow-up moves back.

Any reply stops the automated sequence; the worst experience is answering a recruiter and then getting their pre-scheduled "just bumping this up". Low-confidence labels go to the recruiter, because a wrong "not interested" silently loses a good candidate.

First vs later: first send-time checks, the four limits, idempotent sends, bounce handling and reply classification with a human fallback. Later, send-time optimisation, other opted-in channels under the same suppression and caps, and more automatic FAQ answers once their accuracy is measured.

Deep-Dive: Audit Trail, Fairness, and Explainable Decisions

Hiring is high-stakes and regulated in many places, and an agent that ranks and contacts people will be asked to justify itself. I design for that from the start.

The Audit Log

Every step, tool call, model call, transition and human decision produces an event:

{
  "event_id": "evt_01JC9",
  "tenant_id": "t_77",
  "hiring_job_id": "job_8f2c",
  "candidate_ref": "cand_4410",
  "actor": {"type": "agent", "agent": "ranker", "model": "model-x-2026-09", "prompt_version": "rank-v12"},
  "action": "RANKED",
  "inputs": {"requirement_profile_version": 3, "candidate_profile_version": 8},
  "output": {"tier": "strong", "grades": {"R1": "strong", "R2": "partial", "N1": "none"}},
  "evidence_ref": "ev_99c1",
  "prev_hash": "a3f9...",
  "hash": "7b21..."
}

• Append-only and hash-chained per tenant, so tampering or deletion in the middle is detectable. Periodically anchoring the latest hash in a separate store strengthens this.
• Ids, not personal data. Names, emails and resume text live in a PII vault, encrypted with a per-candidate key.
• Erasure without breaking the chain. Deleting a candidate's key (crypto-shredding) makes their data unreadable while the chain still shows that a decision happened, by whom, under which model and profile version.
• Versions on everything, or "why did this candidate rank lower in October than in September?" has no answer.

Keeping Protected Attributes Out of the Features

The ranker and shortlist step never see protected attributes such as gender, ethnicity, religion, age or disability. The hard part is proxies: names and photos suggest gender and ethnicity, graduation years suggest age, addresses can correlate with ethnicity and income, and employment gaps with caregiving or disability.

So a redaction step in code builds the condensed profile the model sees: skills, roles, responsibilities and achievements, with names, photos, contact details, birth dates, graduation years and addresses removed, experience expressed as relevant years, and location reduced to "meets the location constraint: yes". The recruiter sees the full profile; the model's grades do not depend on those fields. Counterfactual tests in the evaluation suite check that the same profile with different names, pronouns or gap patterns gets the same grades, and a model or prompt change that fails them does not ship.

Adverse-Impact Monitoring

Removing attributes does not prove outcomes are fair, because proxies leak and sources may be skewed. So I measure outcomes. Candidates may voluntarily self-identify; that data lives in a separate, restricted store, never joins the features, and is readable only by the monitoring job and a small compliance role. A batch job computes selection rates per group at each stage (sourced to contacted, contacted to replied, screened to shortlisted). A commonly used heuristic is the four-fifths rule: a group whose selection rate is below 80% of the highest group's rate at a stage deserves a closer look.

def adverse_impact(stage_counts: dict[str, tuple[int, int]], min_n: int = 30):
    """stage_counts: group -> (selected, considered) for one stage of one job family."""
    rates = {g: s / n for g, (s, n) in stage_counts.items() if n >= min_n}
    if len(rates) < 2:
        return []                               # too little data to say anything
    best = max(rates.values())
    return [(g, round(r / best, 2)) for g, r in rates.items() if best > 0 and r / best < 0.8]

The output is an alert to a human, not an automatic correction. The ratio is noisy for small samples, so I aggregate by job family and time window and show confidence intervals next to it. Investigation uses the audit log: which stage, which requirement grades, which sources. Often the cause is upstream, such as a source that reaches a narrow population, and the fix is a sourcing change. Rules on automated hiring tools differ by jurisdiction, and some require bias audits, notices or human review, so I make all of this configurable per tenant and exportable, and leave the legal reading to the employer's counsel.

Explainable Decisions

Because grades carry quoted evidence and the score is computed in code, an explanation is a lookup, not a second model call that might rationalise after the fact:

• For the recruiter: "Strong on R1 ('led the Go migration of the order service, 6 years'), partial on R2 (MySQL, not Postgres), no evidence for N1", with profile version and date.
• For a candidate who asks: which stated requirements were judged met, and that a human made the final selection, without exposing other candidates or internal weights.
• For an auditor: the full chain: inputs by version, model and prompt version, grades, gate id and approver.

One structural rule holds it together: the agent proposes, a human decides on selection. The agent can recommend and deprioritise, but an applicant moves to NOT_PROGRESSED only through a human-approved shortlist or close decision, so a person is accountable for every selection.

First vs later: first the hash-chained, ids-only audit log, the PII vault, code-level redaction, evidence-based grades and the human-selection rule. Later, counterfactual suites per job family, the self-identification flow and adverse-impact dashboards (they need volume to mean anything), and exports for external audits.

Conclusion: Bringing It All Together

An AI recruiting agent looks like chat over a candidate database, but its hard parts are those of a payments system: side effects that cannot be undone, external events that arrive twice or out of order, and a duty to explain every decision. The LLM is the easy part. The design work is the structure around it, so that it can only do the right thing.

If I had to summarise the design in four decisions:

1. Rank with evidence, not scores. An approved, versioned requirement profile, cheap hybrid retrieval and deterministic filters, then an LLM that grades each requirement with quoted evidence, and a final score computed in code.
2. Keep pipeline state explicit and next to the job queue. Two state machines enforced by the database, a Postgres job table with leases and dedupe keys, one inbox for everything external including approvals, and gates that approve a snapshot and are enforced in the tool layer.
3. Treat outreach as the most dangerous tool. Consent and suppression checked at send time, limits per mailbox, domain and candidate, idempotent sends that prefer missing a message to doubling it, and every reply stopping the sequence.
4. Build accountability into the data model. A hash-chained audit log with personal data in a separate vault, no protected attributes or proxies in what the ranker sees, four-fifths monitoring as a trigger for human review, and a human approving every selection.

Related reading: Designing a Hosted AI Agent Platform, Designing a Workflow Orchestration Engine, Designing a Notification System, and Interview Rounds Explained.