Designing an AI Candidate Evaluation System: Rubrics, Calibrated Judges, and Fair Scores

When I design a system that evaluates job candidates with AI, I start from an uncomfortable fact: this is one of the few places where a model's output directly changes someone's life. A chatbot that gives a mediocre answer wastes a minute. An evaluator that scores a candidate unfairly can cost them a job, and if it does so systematically it can quietly exclude whole groups of people. So the bar is not "does the score look reasonable". The bar is: can I explain every score, reproduce it months later, show it agrees with careful human graders, and show it does not move when I change things that should not matter, like a candidate's name or accent.

The raw material is what an interview leaves behind: a transcript of the conversation, code submissions with their test results, and system design diagrams. The output is a set of per-criterion scores, an overall recommendation on a scale from strong hire to strong no hire, and a report where every claim points back to the exact words or lines of code that justify it. In between sits a pipeline that has to be cheap enough to run on every interview, strict enough to survive an audit, and humble enough to hand hard cases to a person.

In this post I walk through how I would build it: scoping, requirements with capacity estimates, a high-level architecture, and then four deep-dives:

• Rubric design: criteria, weights, levels and versioning
• Calibrating an LLM judge against human graders
• Bias testing and explainable, evidence-linked scores
• Appeals, re-grading and the audit trail

Scoping the Problem and Clarifying Assumptions

I am building the evaluation side: the system that runs after an interview ends and turns its artefacts into a scored, explained judgement. How the interview itself is conducted (the live voice agent, the code editor, the whiteboard) is a separate system. I treat it as an upstream producer that emits a "completed" event and stores its artefacts.

What I'm including: versioned rubrics per role and level, deterministic code signals, model-graded criteria, aggregation into a recommendation with a human-review band, judge calibration and bias probes, and an evidence-linked report with appeals and an immutable audit trail.

What I'm NOT building here:

• The live interview runtime (voice, video, editor). See Designing a Real-Time Voice AI Agent for that shape.
• General LLM evaluation of model outputs. Evaluating a model and evaluating a person share tools but have very different stakes. I covered the model side, including position and verbosity bias in judges and kappa-based CI gates, in Designing an LLM Evaluation and Observability Platform, and I will not repeat it here.
• The final hiring decision. The system produces a recommendation and evidence. A human makes the decision.

Key assumptions:

• Interviews are 45 to 60 minutes with several phases (introduction, coding, design, behavioural), and the transcript has speaker labels and timestamps.
• Code runs against hidden test suites in a sandbox, so pass/fail and runtime data already exist or can be regenerated.
• Design diagrams are available in a structured form (components, connections, labels) rather than only as pixels.
• Results are needed within tens of minutes, not seconds. Nobody is waiting on a spinner.

How this differs from classic grading systems: an automated code grader is deterministic: same submission, same score. A human panel is the opposite: rich judgement, low throughput and known disagreement between graders. This system sits between them. It keeps determinism where it exists, uses models only where judgement is needed, and must prove the model is as consistent as a good panel. The hard part is reproducibility and evidence, not prompting.

Functional and Non-Functional Requirements

Functional Requirements

1. Evaluate: given an interview id, produce per-criterion scores against the rubric pinned for that job and level.
2. Ground: attach quoted evidence spans (transcript offsets, code line ranges, diagram elements) to every score. No score without evidence.
3. Recommend: aggregate criterion scores into an overall score and a recommendation band, with a confidence measure.
4. Route to humans: send borderline, low-confidence or flagged evaluations to a human review queue.
5. Report: render a report for recruiters and hiring managers, and a candidate-facing summary where the product offers one.
6. Appeal and re-grade: accept appeals, replay the original evaluation exactly, and record any human override.
7. Audit: answer "why did this candidate get this score" for any evaluation, at any later date.

Non-Functional Requirements

Back-of-the-Envelope Capacity Estimation

These are planning assumptions, not measurements. The method is what matters: count judge calls, then tokens, then storage.

Volume:

• I assume 20,000 completed interviews per day, with completions bunched into business hours, so a peak of about 4x the average rate.
• 20,000 / 86,400 s ≈ 0.23 interviews per second on average, about 1 per second at peak.

Judge calls:

• I assume a rubric of 8 criteria, of which 2 are fully deterministic and 6 need a model.
• I score each model-graded criterion in its own call, with only the relevant transcript phase in context. Long, all-in-one prompts are cheaper but make scores bleed into each other.
• I re-run borderline criteria (say 25% of them) twice more for a consistency check.
• 6 × (1 + 0.25 × 2) = 9 calls per interview, so 20,000 × 9 ≈ 180,000 judge calls per day.

Tokens:

• I assume each call carries about 4,000 tokens of transcript segment and artefacts, plus 1,000 tokens of rubric and instructions: 5,000 input tokens.
• Output is a structured score with quoted spans and a short rationale: about 600 output tokens.
• Input: 180,000 × 5,000 ≈ 900 million input tokens per day. Output: 180,000 × 600 ≈ 108 million.
• The rubric and instructions are the same for every interview of a given job, so they form a stable prompt prefix. Prefix caching makes that 1,000-token block cheap after the first call.

Human review: I assume 10% of evaluations land in the human band, plus appeals: about 2,000 reviews per day. At 20 minutes each that is roughly 670 reviewer-hours per day. That number, not GPU cost, is often the real constraint, so the width of the review band is a business dial.

Storage: transcript, code, test logs, diagrams and the evaluation record come to roughly 180 KB per interview under my assumptions, so 20,000 × 180 KB ≈ 3.6 GB per day, about 1.3 TB per year. Audio and video are referenced, not copied.

The takeaway is that this is not a throughput-heavy system. It is a correctness-heavy one. The money goes on judge tokens and human review time, and the risk goes on every score being defensible.

High-Level Architecture and Core Components

Component Walkthrough

Completion events. The interview runtime publishes interview.completed with the interview id once all artefacts are durably stored. The evaluator never reads from the live session, only from finished, immutable artefacts.

Artefact store. Transcripts, code submissions, test logs and structured diagrams in object storage, each written once and addressed by content hash. Content addressing is what later lets me prove that an appeal replay used exactly the same inputs.

Rubric registry. Versioned, immutable rubrics. A job posting pins a rubric version when it opens. The orchestrator resolves that pin at evaluation time, so an interview is always scored against the rubric the candidate was actually interviewed for. The first deep-dive covers it.

Evaluation orchestrator. A durable workflow per interview: fetch artefacts, segment the transcript by phase, fan out deterministic and model-graded tasks, collect results, aggregate, write, route. Each step is idempotent and keyed by (interview_id, evaluation_run_id, criterion_id), so a crash mid-run resumes rather than double-scoring.

Deterministic signal workers. Re-run hidden tests in a sandbox, record pass counts per test group, measure runtime on growing inputs to estimate complexity, and run static checks. Facts, not opinions.

Criterion judge pool. A pinned model version behind an internal gateway. Each call scores one criterion from the rubric anchors, the relevant transcript segment and the deterministic facts, and must return quoted evidence spans.

Aggregator. Verifies evidence, applies weights and thresholds, and computes confidence. Anything near a threshold, low in confidence or missing evidence goes to the human queue.

Evaluation store. A relational store for evaluations, scores, evidence spans, overrides and the audit log. The queries are relational ("all evaluations on rubric v7 with judge config v3 where criterion X was overridden"), and constraints like "every score has evidence" are easy to enforce there.

Human review queue and report UI. Reviewers see the AI's score and evidence next to the source and confirm or override with a reason. Recruiters see a report where every claim links to its source.

Request Flow

1. The interview runtime writes artefacts, then publishes interview.completed.
2. The orchestrator creates an evaluation run, resolves the job's pinned rubric version and the active judge config, and snapshots all input hashes into the run record.
3. Deterministic workers re-run tests and compute complexity and static signals.
4. The transcript is segmented into phases. For each model-graded criterion, the orchestrator builds a prompt from the rubric anchors, the relevant segments and the deterministic facts, and calls the judge.
5. Each judgement returns a level, quoted spans and a rationale. The span verifier checks that every quote exists verbatim at the cited offsets.
6. The aggregator computes the weighted score, the band and the confidence, and decides whether a human review is required.
7. The run is committed with its rubric version, model version and prompt hashes. The report becomes visible once the run is final or the human review is done.

Scaling It Out

• Priority queues. Fresh interviews with a recruiter waiting go ahead of bulk re-scores, which go ahead of calibration experiments. Re-scoring a whole golden set or an old cohort can use an off-peak batch pool.
• A gateway in front of the judge. Retries with backoff, per-workload quotas and, importantly, no silent fallback to a different model. If the pinned judge is unavailable, the job waits. A fallback model would produce scores on a different scale, which is a fairness bug wearing a reliability hat.
• Judgement cache. Keyed by a hash of (criterion prompt, rubric version, judge config, input segment), so a replay with identical inputs gets the stored judgement back. That helps cost and reproducibility together.
• Warehouse for analytics. Calibration, drift and fairness analysis read from a warehouse fed by change data capture, never from the live store.

Deep-Dive: Rubric Design: Criteria, Weights, Levels, and Versioning

Criteria Must Be Observable

The most common rubric failure is a criterion that cannot be observed in an interview. "Culture fit" or "intelligence" invite the judge (human or model) to fill gaps with impressions, and impressions are exactly where bias lives. I require every criterion to name what in the artefacts would count as evidence: "states the time complexity of the approach and justifies it", "identifies at least one failure mode of the proposed design and a mitigation", "asks a clarifying question before committing to an approach". If an author cannot write that sentence, the criterion is not ready.

Levels With Behavioural Anchors

I score each criterion on a small ordinal scale, typically 1 to 4, and every level has a behavioural anchor: a concrete description of what a response at that level looks like. Small anchored scales are far more stable than bare 1 to 10 scales, for humans and models alike. An even number of levels also removes the comfortable middle, which makes threshold decisions cleaner.

Anchors differ by seniority level. "Discusses trade-offs" at entry level might mean naming one alternative; at senior level it means comparing alternatives against stated requirements and choosing with reasons. I keep one criterion definition with level-specific anchors rather than three unrelated rubrics.

Deterministic First, Model Second

Each criterion declares its source:

• Deterministic: computed from facts. Correctness is the share of hidden test groups passed. Efficiency comes from measured runtime growth against input size, compared with the expected complexity class.
• Model-graded: judged from the transcript and artefacts, for things like communication, problem decomposition and design reasoning.
• Hybrid: a deterministic floor or cap plus a model judgement. For example, "code quality" is capped at level 2 if static checks find a correctness-critical issue, whatever the model thinks.

The rule is that the model is never asked to re-derive a fact. It receives "tests passed: 14 of 18; failing groups: empty input, duplicates" as context and judges how the candidate reasoned about those cases. Models are good at reading reasoning and bad at mentally executing code, so I put each to its strength.

rubric_id: backend-engineer
version: 7
levels_supported: [entry, mid, senior]
scale: { min: 1, max: 4 }
criteria:
  - id: correctness
    source: deterministic
    weight: 0.25
    signal: hidden_tests.pass_ratio_by_group
    mapping: { "1.0": 4, "0.8": 3, "0.5": 2, "0.0": 1 }
  - id: problem_decomposition
    source: model
    weight: 0.20
    phases: [coding]
    evidence_required: transcript
    anchors:
      senior:
        4: "Breaks the problem into parts unprompted, states invariants, and adapts the plan when a test fails."
        3: "Clear plan with minor gaps; needs one nudge to handle an edge case."
        2: "Starts coding before a plan; structure emerges only after hints."
        1: "No discernible plan; cannot explain the approach when asked."
  - id: design_tradeoffs
    source: model
    weight: 0.20
    phases: [system_design]
    evidence_required: [transcript, diagram]
recommendation:
  thresholds: { strong_hire: 3.5, hire: 3.0, lean_hire: 2.6, no_hire: 2.0 }
  human_review_band: 0.15
  hard_gates:
    - { criterion: correctness, min_level: 2, else_cap: no_hire }

Weights and Recommendation Thresholds

Weights must sum to 1 and are part of the rubric version. The overall score is a weighted mean of levels. Thresholds turn it into a band (strong hire, hire, lean hire, no hire, strong no hire). Two refinements matter in practice:

• Hard gates. Some criteria are non-compensatory: brilliant communication should not make up for code that does not work at all. A gate caps the band regardless of the weighted mean.
• A review band. Any overall score within a set distance of a threshold goes to a human. Thresholds are sharp; measurement is not. I would rather pay for a review than flip a recommendation on noise.

I set initial thresholds from the golden set: score the panel-graded interviews, then choose thresholds that best reproduce the panel's recommendations. I revisit them when the panel disagrees with the system often in one direction.

Versioning and Pinning

Rubrics are immutable once published. Fixing a typo creates version 8. The reasons are practical:

• Every evaluation stores the rubric_version it used, so scores are comparable only within a version, and analytics never mix them silently.
• A job pins a version when it opens. Candidates who interview for the same job are scored against the same rubric, even if a new version ships mid-process. Comparing two finalists scored on different rubrics is unfair to both.
• A new version goes through a dry run on the golden set before approval. The diff report shows which interviews would change band, by how much, and why. A rubric change that moves 20% of recommendations is a policy change and needs sign-off, not just a code review.

First vs later: first, a small number of carefully anchored criteria, deterministic signals for code, immutable versions and pinning. Later, an authoring UI with linting (unobservable criteria, missing anchors, weights not summing to 1), automatic dry runs and per-level threshold tuning.

Deep-Dive: LLM-as-Judge Calibration Against Human Graders

What "Calibrated" Means for People

For model evaluation, calibration usually means agreement with a set of reference labels. For people, there is no single ground truth: two experienced interviewers watching the same interview will sometimes disagree by a level. So I do not ask "does the AI match the human". I ask: is the AI at least as consistent with the panel as panel members are with each other? That framing sets a realistic target and gives an honest baseline.

The Golden Set and the Panel

The golden set is a pinned collection of consented real interviews (or careful synthetic ones), covering roles, levels, phases and the full score range, not just typical cases. Each is graded independently by at least two trained graders, blind to each other and to any AI score. Disagreements of more than one level go to an adjudicator, who records the final label and the reason.

I keep the individual panel scores, not just the adjudicated label, because they give the human-to-human agreement per criterion: the bar the AI must clear. Correctness usually has near-perfect human agreement; communication has much less. A single global target would be too strict for some criteria and too loose for others.

Measuring Agreement on an Ordinal Scale

Rubric levels are ordered, so a miss by one level is less bad than a miss by three. I use agreement measures that respect order, such as quadratically weighted kappa or Krippendorff's alpha for ordinal data, alongside two numbers anyone can read: exact agreement and "within one level". I also check direction: a judge that is systematically one level harsher than the panel on one criterion can have decent agreement and still shift recommendations.

def promotion_report(ai, panel_a, panel_b, adjudicated, criteria):
    """Per-criterion: is AI-vs-adjudicated agreement at least the human-vs-human baseline?"""
    report = {}
    for c in criteria:
        human_baseline = weighted_kappa(panel_a[c], panel_b[c])      # grader vs grader
        ai_agreement   = weighted_kappa(ai[c], adjudicated[c])       # AI vs final label
        bias = mean(a - h for a, h in zip(ai[c], adjudicated[c]))    # signed level offset
        within_one = mean(abs(a - h) <= 1 for a, h in zip(ai[c], adjudicated[c]))
        report[c] = {
            "human_baseline": human_baseline,
            "ai_agreement": ai_agreement,
            "signed_bias": bias,
            "within_one": within_one,
            "pass": ai_agreement >= human_baseline - MARGIN and abs(bias) <= MAX_BIAS,
        }
    return report

A criterion that fails is not dropped from the product. It is switched to human-graded only until it passes, and the report says so.

Consistency Across Re-Runs

Agreement with humans is one property; stability with itself is another. A judge that gives 3, then 2, then 4 on the same input averages fine and is useless as a measurement. I check two ways:

• Run-to-run: score each golden interview several times with the production config and measure the spread per criterion. Low temperature and a fixed output schema help, but I do not assume they make outputs identical; I measure.
• Perturbation: re-score with harmless changes such as reordered segments or reformatted whitespace. A score that moves on formatting is reacting to the wrong thing.

In production, criteria near a level boundary run three times and I take the median. If the runs disagree by more than one level, the criterion is low-confidence and goes to a human.

Drift When the Judge Changes

The judge config is everything that can change a score: model version, prompt template, rubric rendering, temperature, output schema and the evidence-segmentation logic. I hash it into a single judge_config_id. Any change makes a new id, and a new id must re-score the pinned golden set and pass the promotion gate before it touches a live candidate.

This matters most when a provider ships a new model version or deprecates the old one. A newer model can be better in general and still shift scores on my rubric, often in one direction. The re-score shows it before any candidate is affected: if the new judge is uniformly half a level stricter on one criterion, I decide whether to adjust the prompt or the thresholds, rather than discovering it in hiring outcomes. The report shows the config id, so two candidates being compared can be seen to share a judge.

Live drift. Reviewers confirm or override AI scores every day, which gives me a rolling agreement measure. A drop on one criterion, or overrides piling up in one direction, triggers an investigation even if nothing was deployed. Inputs drift too: a new question bank can push the judge into territory the golden set never covered.

First vs later: first, a golden set of a few hundred panel-graded interviews, per-criterion agreement against the human baseline, a hard promotion gate and no silent fallbacks. Later, rolling agreement from review overrides, automatic golden-set refresh with newly adjudicated cases, and separate golden sets per role family.

Deep-Dive: Bias Testing and Explainable, Evidence-Linked Scores

No Score Without Evidence

The single most useful rule in this system is that every model-graded score must cite evidence, as spans the system can check. The judge returns structured output, and the evidence is not free text but pointers into the artefacts:

{
  "criterion_id": "problem_decomposition",
  "level": 3,
  "evidence": [
    {
      "artefact": "transcript",
      "artefact_hash": "sha256:9f2c...",
      "start_char": 18234,
      "end_char": 18391,
      "quote": "Before I write anything, let me split this into parsing, the window logic, and the output step.",
      "supports": "plans before coding"
    },
    {
      "artefact": "code",
      "artefact_hash": "sha256:41ab...",
      "lines": [12, 30],
      "supports": "window logic isolated in its own function"
    }
  ],
  "against": [
    {
      "artefact": "transcript",
      "start_char": 24410,
      "end_char": 24502,
      "quote": "Oh, I didn't think about an empty list.",
      "supports": "edge case found only after a failing test"
    }
  ],
  "rationale": "Clear plan stated before coding; missed the empty-input edge case until a test failed, which matches level 3 for senior."
}

The span verifier checks each quote against the stored artefact at the cited offsets. If a quote does not exist verbatim, the judgement is rejected and retried once; a second failure sends the criterion to a human. This catches hallucinated evidence mechanically, without trusting the model's word for it. It also gives me a cheap quality signal: a rising rate of rejected quotes is an early sign that a model or prompt change has gone wrong.

I ask for evidence against the score as well as for it. A judge that only collects supporting quotes is building a case, not weighing one. Seeing both sides makes reviewers faster and makes the report fairer to read.

Explainable Reports

The report is built from stored evidence, not a fresh model summary. Each criterion shows its level, that level's anchor text, and quoted spans linked to the exact transcript moment or code lines; the band shows the weighted calculation and any hard gate. A hiring manager can disagree with a score, but can always see why it was given.

Removing Signals That Should Not Matter

Before the judge sees a transcript, an identity redaction step replaces names, self-identified demographic details, school and employer names where they are not part of a criterion, and other identity cues with neutral placeholders. The judge needs none of them to score problem decomposition. Redaction is not perfect, since accent and dialect show through in wording and speech-to-text errors, which is why I also test.

Speech-to-text deserves particular attention. Its accuracy varies with accent and audio quality, and a judge reading a garbled transcript can mark down "communication" for what is really an STT error. I pass per-segment STT confidence to the judge, flag low-confidence regions with an instruction not to penalise them, and track STT error rates across accent groups as a fairness metric of its own.

Counterfactual Testing as a Bias Probe

The core probe is simple: take a transcript, change only what should not matter, and check that the score does not move. On a regular sample of golden and production interviews (with consent), a counterfactual generator creates variants:

• Swap names and gendered pronouns across a balanced list
• Rewrite selected passages into different dialects or non-native phrasing while keeping technical content identical
• Change school or employer names that leak socio-economic signals
• Inject typical STT artefacts of different accents into otherwise identical text

The judge scores each variant with the production config. For every criterion I measure the score difference between the original and each variant. Under a fair judge the differences should be centred at zero and no larger than the judge's own run-to-run noise. A consistent offset for one class of variant is a bias finding, and it blocks promotion of a judge config just like a calibration failure.

def counterfactual_gap(judge, interviews, variant_fn, criteria, runs=3):
    gaps = {c: [] for c in criteria}
    for itv in interviews:
        base = median_scores(judge, itv, runs)              # median over repeated runs
        alt  = median_scores(judge, variant_fn(itv), runs)
        for c in criteria:
            gaps[c].append(alt[c] - base[c])
    # flag if the mean gap exceeds the judge's own noise band
    return {c: {"mean_gap": mean(g), "flag": abs(mean(g)) > noise_band[c]} for c, g in gaps.items()}

Counterfactuals test the judge in isolation. I also run outcome monitoring: recommendation rates by group, using voluntarily provided group data stored apart from the evaluation path so the judge never sees it. Selection-rate ratios (such as the four-fifths heuristic from US employment guidance) serve as alerts that call for investigation, not as proof either way. Gaps can come from the rubric, the questions, the sourcing funnel or the judge, and the evidence-linked records let me tell which.

The Legal Frame, Kept Generic

Several jurisdictions now regulate automated tools in hiring, with requirements such as bias audits, candidate notice, explanations and access to human review. I treat these as design inputs: evidence-linked scores, audits, human decisions and appeals are already in the architecture. The specifics vary by region and change over time, so they belong with counsel; the system's job is to make whatever they require cheap to produce.

First vs later: first, mandatory verified evidence spans, identity redaction, STT confidence flags and a counterfactual name-swap suite in the promotion gate. Later, dialect and STT-artefact counterfactuals, outcome monitoring with a separate demographic store, and per-region audit exports.

Deep-Dive: Appeals, Re-Grading, and the Audit Trail

Reproducibility Is the Foundation

An appeal is only meaningful if I can show exactly what the system did. So every evaluation run stores, at creation time:

• The content hash of every input: transcript, code, test logs, diagrams
• The rubric version and the job's pin
• The judge config id: model version, prompt template hash, parameters, output schema version
• The code version of the orchestrator, aggregator and deterministic workers
• Every raw judge response, the verifier's verdict on each quote, and the aggregation trace

With that record, a replay re-runs the deterministic steps with identical results and re-reads the stored judge responses instead of calling the model again. A fresh model score is always a separate run with its own record. Nothing overwrites the original.

Appeals and Re-Grading

An appeal can come from a candidate ("the transcript misrecognised my explanation") or from a recruiter ("this score does not match what I saw"). The case manager:

1. Opens a case tied to the original evaluation run, with the appellant's reason.
2. Runs a pinned replay to confirm the result reproduces and the inputs are intact.
3. Optionally runs a fresh re-score with the same pinned configs, to show whether the original was an unlucky sample.
4. Assigns an independent reviewer who was not part of the original review, with the original scores, evidence, replay, fresh run and source artefacts side by side.
5. Records a decision: upheld, adjusted per criterion, or escalated. Adjustments need a reason and their own evidence spans; the no-score-without-evidence rule applies to humans too.

If the appeal shows the inputs were wrong (a broken STT segment, a test harness bug), the fix is a corrected artefact with a new hash and a new evaluation run linked to the old one. If it shows the judge was wrong, the case is tagged and joins the next golden-set refresh. Appeals are the best source of hard cases I will ever get.

Systemic issues get bulk re-grading: when a harness bug affected one question for a week, I find every affected run by input hash and question id, re-run them as new runs, and notify downstream owners where a band changed. The query is easy precisely because runs record their inputs and versions.

The Audit Trail

The audit log is append-only. Nothing in the evaluation history is ever updated in place; a human override, a re-grade or a correction is a new row that points at what it supersedes. The "current" score is a view over the latest non-superseded record.

CREATE TABLE evaluation_event (
  event_id         BIGSERIAL PRIMARY KEY,
  evaluation_id    UUID        NOT NULL,
  run_id           UUID        NOT NULL,
  event_type       TEXT        NOT NULL,  -- run_created, criterion_scored, human_override, appeal_decided, regraded
  criterion_id     TEXT,
  level            SMALLINT,
  supersedes_event BIGINT      REFERENCES evaluation_event(event_id),
  actor            TEXT        NOT NULL,  -- 'system:judge_cfg_v3' or reviewer id
  rubric_version   INT         NOT NULL,
  judge_config_id  TEXT,
  inputs_hash      TEXT        NOT NULL,  -- hash over all artefact hashes
  reason           TEXT,
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
  prev_hash        TEXT        NOT NULL,  -- hash chain for tamper evidence
  row_hash         TEXT        NOT NULL
);
-- The application role has INSERT and SELECT only; UPDATE and DELETE are not granted.

Each row includes the hash of the previous row for the same evaluation, forming a chain. A periodic job verifies the chains and anchors daily checkpoints in write-once object storage. This does not make tampering impossible, but it makes it detectable, which is what an audit needs.

Retention and Deletion

Audit permanence collides with data-protection rights such as deletion requests. I keep personal content (transcripts, code, recordings) in the artefact store under the retention policy, and keep the audit log free of raw personal content: hashes, versions, levels and reasons only. On deletion, the artefact goes and the log gains a tombstone event. The chain stays intact, so the system can still prove what it did and when, even after it can no longer show the original words.

First vs later: first, input hashing and version pinning on every run, stored raw judge responses, append-only events and human overrides with reasons. Later, a self-service appeal flow, fresh re-score comparisons, hash-chain verification with external checkpoints and bulk re-grade tooling.

Conclusion: Bringing It All Together

An AI candidate evaluation system looks like a scoring pipeline from the outside. Inside, it is a measurement instrument, and it has to be engineered like one. The model is the least interesting part; what matters is everything around it: observable rubrics, facts the model never has to guess, evidence the system can check, and records that let anyone replay a score years later.

If I had to summarise the design in four decisions:

1. Make rubrics observable, anchored and immutable. Every criterion names its evidence, every level has a behavioural anchor, deterministic signals come first, and every evaluation pins the rubric version it used.
2. Hold the judge to the human baseline, per criterion. The AI must agree with adjudicated panel labels at least as well as panel members agree with each other, stay stable across re-runs, and re-score a pinned golden set before any model or prompt change touches a live candidate.
3. No score without verified evidence, and no score that moves when identity changes. Quoted spans are checked mechanically, identity cues are redacted, and counterfactual swaps of names, dialect and STT artefacts are part of the release gate.
4. Never overwrite; always replay. Inputs are content-addressed, versions are pinned, overrides and appeals are new events in an append-only log, and a human makes the final call.

Get those four right, and the scores become something an employer can act on and a candidate can challenge on equal terms.

Related reading: Designing an LLM Evaluation and Observability Platform, Designing LLM Guardrails, Designing an AI Agent Platform, and What Is an AI Interview?