Designing an AI Tutor Platform: Knowledge Tracing, Adaptive Selection, and Socratic Hinting

    20 min read
    AI tutor
    system design
    knowledge tracing
    adaptive learning
    LLM guardrails

    When I design an AI tutor platform, the first trap I try to avoid is building a chatbot with a question bank attached. It demos well: the learner gets stuck, asks the model, and the model explains the answer beautifully. The learner feels helped, and has learned very little, because the struggle that builds skill was done for them.

    A tutor that teaches does three things a chatbot does not. It knows what the learner knows, skill by skill, from noisy evidence. It chooses what to practise next, at the edge of the learner's ability and with old material revisited before it is forgotten. And it helps without giving the answer away, holding back from the very thing a helpful language model most wants to do. On top of that, the platform has to prove it produces learning, not just minutes in the app.

    In this post I walk through how I would build such a platform from scratch, generically: a multi-tenant system where learners practise coding, system design, behavioural interview answers or any other skill, with an AI tutor that hints and gives feedback. I cover scoping, requirements with capacity estimates, a high-level architecture, and four deep-dives:

    • The learner model: a skills graph and knowledge tracing
    • Choosing the next exercise: mastery, spacing and desirable difficulty
    • Socratic hinting that does not leak the answer
    • Measuring learning instead of engagement

    Scoping the Problem and Clarifying Assumptions

    I am building the practice loop: a learner gets an exercise, attempts it, asks the tutor for help, gets graded and feedback, and the system updates its picture of the learner and picks the next exercise.

    What I'm including:

    • Exercise types: coding problems with tests, short-answer and multiple-choice items, and open-ended answers (a design sketch, a behavioural story) graded against a rubric
    • An AI tutor that chats about the current exercise and gives graded hints and feedback
    • A per-learner mastery model over a skills graph, and adaptive selection with spaced review
    • Multi-tenancy: schools, bootcamps and companies (for interview coaching or upskilling) bring their own learners and optionally their own content and skills graph
    • Experimentation and learning-outcome measurement

    What I'm NOT building here:

    • Live, timed mock interviews with voice and a scored final report. That is a different shape of system, covered in Designing an AI Interviewer Platform.
    • Detailed rubric scoring and LLM-as-judge calibration for open-ended answers. I treat the grader as a component and point to Designing an AI Candidate Evaluation System for how it should work.
    • Content authoring, payments and classroom management, beyond the hooks I need.

    Key assumptions:

    • Each tenant has a library of exercises tagged with skills, organised in a prerequisite graph of a few hundred to a few thousand skills per domain
    • The tutor is a hosted or self-served LLM behind a gateway, provider-agnostic
    • Some tenants serve minors, so privacy defaults must be child-safe

    How this differs from a classic system: a classic e-learning platform is a content system with a fixed sequence: lesson 1, quiz 1, lesson 2. A classic recommender optimises for clicks or watch time. Here the sequence is decided per learner, per attempt, from a probabilistic estimate of what they know, and the objective is future performance on things they have not seen yet. Engagement is a constraint, not the goal, and that change in objective drives most of the design.

    Functional and Non-Functional Requirements

    Functional Requirements

    1. Serve the next exercise for a learner in a course or goal.
    2. Grade attempts: code against tests in a sandbox, short answers, or open-ended answers against a rubric.
    3. Tutor: chat about the current exercise, give hints, and give feedback after an attempt.
    4. Track mastery per skill after every graded attempt.
    5. Adapt: choose the next exercise from mastery, prerequisites, review schedules and tenant rules.
    6. Report progress by skill to learners, teachers and admins.
    7. Experiment: assign learners to policy variants and measure learning outcomes.

    Non-Functional Requirements

    RequirementTargetWhy
    Next-exercise latencyp99 under 300 msSelection sits between every pair of exercises
    Tutor time to first tokenp50 under 1 s, p99 under 3 sLearners give up on slow help and go search for the answer
    Grading latencyshort answers under 200 ms, code under 5 s typicalFeedback is most useful while the attempt is fresh
    Mastery updatevisible before the next selectionThe next pick must reflect the last attempt
    Answer leakagethe tutor never sends a full solution unless policy allowsLeaks destroy the learning value of the exercise
    Tenant isolationno cross-tenant reads of learner dataSchools and companies require it contractually
    Privacychild-safe defaults, data minimisation, deletion on requestSome tenants serve minors

    Back-of-the-Envelope Capacity Estimation

    These are planning assumptions, not measurements. The aim is to find which parts are expensive.

    Learners and attempts:

    • I assume 2 million monthly active learners and 300,000 daily active learners across all tenants.
    • Each active learner makes about 30 graded attempts per day: 300,000 × 30 = 9 million attempts per day.
    • 9M / 86,400 s ≈ 104 attempts per second on average. Usage clusters in school hours and evenings, so I plan for about 5x at peak: around 500 attempts per second.

    Tutor turns:

    • I assume 30% of attempts involve the tutor, with 3 turns on average: 9M × 0.3 × 3 ≈ 8 million tutor turns per day, about 94 per second on average and roughly 470 per second at peak.
    • Each turn sends about 3,000 input tokens (exercise, reference solution, learner's work, recent chat) and gets about 250 back. The answer-leak check adds a second call of roughly 1,000 input tokens.
    • Input: 8M × 4,000 ≈ 32 billion input tokens per day. Output: 8M × 250 ≈ 2 billion output tokens per day.

    Input dominates, and most of it is the same exercise and reference solution repeated across thousands of learners, so prompt caching (stable prefix first) is the biggest cost lever, as described in Designing an LLM Inference Platform.

    Learner state:

    • Each attempt touches about 3 skills: 9M × 3 = 27 million mastery updates per day, about 1,500 writes per second at peak.
    • A learner touches perhaps 300 skills over time, at about 64 bytes per skill state: about 20 KB per learner. For 2 million learners that is 40 GB, small enough for a sharded key-value store with room to spare.

    Event log:

    • About 1 KB per attempt event: 9M × 1 KB ≈ 9 GB per day, about 3.3 TB per year before compression. Every model is trained from this log and every learning claim is proven from it.

    The conclusion: the core loop is a modest transactional system. The expensive parts are LLM tokens and code sandboxes, and the hard parts are statistical, not infrastructural.

    High-Level Architecture and Core Components

    The practice loop: learners reach an API gateway that routes to the practice session service, which asks the exercise selector for the next item using the learner model and the content and skills graph, sends attempts to a grader with a code sandbox, sends help requests to a tutor orchestrator that drafts replies with an LLM and checks them with an answer-leak guard, and emits attempt events to a stream feeding a learning analytics warehouse.

    Component Walkthrough

    API Gateway. Authenticates the learner, resolves the tenant and applies per-tenant rate limits. Every data access downstream is scoped by tenant id.

    Practice Session Service. Owns one learner's session: the open exercise, attempts, hint usage and timers. It is the single writer for a session, which keeps attempt ordering simple.

    Content and Skills Graph. A relational store of exercises, their skill tags (the item-to-skill mapping, called a Q-matrix in the tutoring literature), reference solutions, rubrics and the prerequisite graph between skills.

    Grader. Deterministic checks for short answers, a sandboxed runner (containers or Firecracker microVMs) for code against hidden tests, and a calibrated LLM-as-judge rubric grader for open-ended answers.

    Learner Model Service. Holds each learner's per-skill mastery estimate and updates it from graded attempts. The first deep-dive.

    Exercise Selector. Picks the next exercise from mastery, prerequisites, review schedules and tenant rules. The second deep-dive.

    Tutor Orchestrator. Builds the tutor's context, calls the LLM, and passes every draft through an answer-leak guard before it reaches the learner. The third deep-dive.

    Event stream and warehouse. Every attempt, hint, tutor turn and selection decision becomes an event, feeding model fitting, teacher reports and experiment analysis. The fourth deep-dive.

    Request Flow

    1. The practice service asks the selector for the next exercise; the selector reads mastery and returns an item in under 300 ms.
    2. The learner asks for a hint. The orchestrator builds context, checks the hint ladder, drafts a reply, runs the leak guard and sends the hint.
    3. The learner submits, and the grader returns tests passed, rubric levels, or correct and incorrect.
    4. An attempt event (outcome, hints used, time taken) updates the mastery of the tagged skills.
    5. The tutor gives feedback grounded in the grader's result, and the selector picks the next exercise from the updated mastery.

    Scaling It Out

    The scaled-out platform: learners fetch static exercise assets from a CDN and reach stateless practice pods through a regional load balancer and a tenant-aware gateway; pods read tenant policy from a config registry, read and update learner state in a store sharded by learner id, read content from a cache, enqueue grading jobs for a sandbox runner pool, call an LLM gateway for tutor turns, and write events to a partitioned log.

    • Stateless pods, learner state sharded by learner id in a key-value store (DynamoDB, Cassandra or sharded Postgres all work). Each learner's updates hit one shard, so there are no cross-shard transactions.
    • Content is read-heavy and changes rarely, so it sits behind a cache and a CDN, invalidated by course version.
    • Grading is the bursty part. Code grading goes through a queue to sandbox workers that autoscale on queue depth, because a class of 40 submitting at once is the normal case.
    • LLM calls go through a gateway for provider failover, per-tenant token budgets and caching. When the LLM is down or a tenant is over budget, the tutor degrades to authored hints.
    • Data residency. I pin each tenant to a home region and keep its shards, events and logs there.

    Deep-Dive: The Learner Model: Skills Graph and Knowledge Tracing

    The learner model pipeline: graded attempt events carry skill tags into a knowledge tracer, which also reads prerequisites from the skills graph and fitted parameters from a feature store, and writes a posterior mastery update to per-learner state; offline, nightly fitting over the attempt history warehouse publishes those parameters, and a deep tracing model is compared in shadow.

    The Skills Graph

    Everything adaptive depends on a good answer to "what are the skills, and which come before which". I model it as a directed acyclic graph per domain:

    • Nodes are skills small enough to practise directly: "two-pointer technique on sorted arrays", "choosing a partition key", "structuring a STAR answer". "Algorithms" is a chapter, not a skill.
    • Edges are prerequisites: hash maps before "two-sum style lookups".
    • Exercises map to skills through the Q-matrix, usually one primary skill and up to two secondary ones.

    The graph is versioned, with a validation job that rejects cycles. Tenants can add skills and edges on top of a shared base graph and hide base nodes, but not delete them, so content written against the base keeps working. An LLM is useful for proposing skill tags and missing prerequisites, but its output is a draft for a subject expert. The data checks the graph later too: if learners who have not mastered A succeed at B as often as those who have, the A to B edge is probably wrong.

    Knowledge Tracing: Estimating What the Learner Knows

    Mastery is not observable. I see attempts: correct or not, with hints or not, fast or slow. Knowledge tracing is the family of models that turns that noisy sequence into an estimate of mastery per skill. There are three families worth knowing.

    Bayesian Knowledge Tracing (BKT). The classic model from the intelligent tutoring systems literature (Corbett and Anderson). Each skill is a hidden binary state, known or not known, and four parameters per skill describe it:

    • p_init: probability the learner already knows the skill
    • p_learn: probability of moving from unknown to known after a practice opportunity
    • p_guess: probability of answering correctly without knowing it
    • p_slip: probability of answering incorrectly despite knowing it

    After each attempt I apply Bayes' rule, then the learning transition:

    def bkt_update(p_known: float, correct: bool, p: dict) -> float:
        """One Bayesian Knowledge Tracing step for a single skill."""
        g, s, t = p["p_guess"], p["p_slip"], p["p_learn"]
        if correct:
            num = p_known * (1 - s)
            den = num + (1 - p_known) * g
        else:
            num = p_known * s
            den = num + (1 - p_known) * (1 - g)
        posterior = num / den                       # belief after seeing the evidence
        return posterior + (1 - posterior) * t      # chance they learned from this attempt
    
    # Mastery threshold is a tunable policy setting, for example 0.95
    

    BKT is cheap, interpretable and works from little data per learner. But it treats skills independently, ignores item difficulty, and assumes no forgetting.

    Logistic and IRT-style models. Item Response Theory models a correct answer as a function of learner ability and item parameters. In the two-parameter form, p(correct) = 1 / (1 + exp(-a * (theta - b))), where theta is ability, b is difficulty and a is discrimination. Tutoring extensions such as the Additive Factors Model and Performance Factors Analysis add per-skill terms and counts of prior successes and failures, which handles multi-skill items better than BKT. A lightweight online variant is an Elo-style update: after each attempt, move the learner's ability and the item's difficulty in opposite directions by K * (outcome - predicted), which calibrates item difficulty for free.

    Deep Knowledge Tracing (DKT). Piech and colleagues used a recurrent neural network over the attempt sequence to predict the next outcome, and later work uses attention-based models. These capture interactions between skills on large datasets, but are harder to explain to a teacher, need much more data, and can behave oddly for sparse learners.

    What I Would Actually Run

    I would start with BKT per skill plus an Elo-style item difficulty: interpretable, cheap per update and robust with sparse data. Then:

    • Fit parameters nightly per skill (expectation-maximisation or a grid search), with bounds. Unconstrained fitting can produce degenerate solutions, so I cap p_guess and p_slip at a sanity bound such as 0.3.
    • Run a logistic or deep model in shadow and compare next-attempt prediction on held-out learners (AUC, log loss, calibration). I switch only if the gain is clear and the outputs stay explainable enough for reports.
    • Keep the serving interface model-agnostic: mastery(learner, skill) and p_correct(learner, item). The selector does not care which family produced the number.

    Evidence Quality: Not Every Attempt Counts the Same

    The model is only as good as what I feed it:

    • Hints weaken evidence. A common convention treats any hinted attempt as incorrect for tracing; I prefer recording the hint level and discounting by it.
    • First attempts matter most. Retries after feedback are partly copying, so I trace the first attempt per item.
    • Open-ended grades are noisy. I convert rubric levels from the judge into a soft outcome rather than a hard pass or fail.
    • Prerequisites as priors. A new skill's p_init borrows from the mastery of its prerequisites. This is the main way the graph informs the tracer.

    Failure Modes

    • Wrong skill tags make mastery flow to the wrong skills. I monitor per-item fit: an item whose outcomes the model persistently mispredicts is flagged for review.
    • Gaming: clicking through hints to reach the answer. Hint-abuse detection feeds the evidence weighting.
    • Duplicate events from client retries would double-count evidence, so attempt events carry an idempotency key.

    Deep-Dive: Choosing the Next Exercise: Mastery, Spacing, and Desirable Difficulty

    Next-exercise selection: a request fans out to a frontier finder that reads unlocked skills from the graph and a spaced review scheduler that returns due reviews; a candidate pool builder merges them, a constraints filter applies tenant and course rules, a predictor scores each candidate's success probability and expected gain, and a bandit policy chooses the exercise while logging the decision and its propensity.

    Three Forces Pulling on the Choice

    The selector balances three goals that conflict:

    1. Progress: move to new skills once their prerequisites are mastered (the "frontier" of the graph).
    2. Retention: come back to mastered skills before they are forgotten.
    3. Difficulty: pick items the learner will probably, but not certainly, get right.

    The research on desirable difficulties (Bjork's term) is the guide: spacing, interleaving and effortful retrieval make practice feel harder yet improve long-term retention. A selector that maximises in-session correctness drills one easy skill, feels great, and teaches less.

    Candidate Generation

    As in a recommendation engine, I generate candidates cheaply and score a small set (the two-stage pattern from Designing a Recommendation Engine). The pool here is small, though: tens of items, not millions.

    • Frontier skills: skills not yet mastered whose prerequisites are all above the mastery threshold. If none are unlocked, the weakest prerequisite of the course goal becomes the target. This is remediation.
    • Due reviews: mastered skills whose predicted recall has dropped below a target. Spaced repetition schedulers do this for flashcards: SM-2 (from SuperMemo, long the default in Anki) grows the interval by an ease factor after each success, and FSRS, a newer open-source scheduler, models memory stability to predict recall directly. I apply the idea per skill: a stability estimate grows with successful spaced reviews, and a skill is due when predicted recall falls below, say, 0.9.
    • Interleaving: a few items from related skills, so the learner must work out which technique applies, not just apply the one they were just drilling.

    Then the constraints filter removes items seen recently, items outside the assigned unit or course settings, and items held out for assessment (see the last deep-dive).

    Scoring: Success Probability and Expected Gain

    For each candidate the learner model gives p_correct. I aim for a target success band, a planning setting I would start around 0.7 to 0.85 and tune by experiment, not a law. Too easy wastes time; too hard produces frustration and hint dependence.

    def score(candidate, learner, cfg):
        p = learner.p_correct(candidate.item)
        band_fit = -abs(p - cfg.target_success)          # prefer items near the target
        gain = candidate.expected_mastery_gain(learner)   # how much a success would move mastery
        review = cfg.review_weight if candidate.is_due_review else 0.0
        novelty = cfg.novelty_weight * candidate.unseen_bonus
        return band_fit + cfg.gain_weight * gain + review + novelty
    

    Expected gain uses the tracer itself: simulate a correct and an incorrect outcome, weight by p, and measure how much the skill's mastery would change. Items on skills where the learner is already at 0.99 score near zero.

    Exploration vs Exploitation: Bandits

    A pure argmax over that score never tries alternatives, so I never learn whether something else works better, and its deterministic choices make logged data useless for evaluating new policies. So the last step is a bandit policy: Thompson sampling or an upper-confidence-bound rule over the top few candidates. The details that matter:

    • Log the propensity: the probability with which the chosen item was selected, alongside every alternative considered. That enables off-policy evaluation of new policies before I run them on learners.
    • Bound the exploration: never explore outside the constraint set, and never pick an item with predicted success below a floor (say 0.4). Exploring into frustration is not acceptable in education.
    • Choose the reward carefully: next-attempt correctness is quick but encourages easy picks. I use mastery gain on later, unhinted attempts of the same skill as the reward, which arrives later but rewards what I care about.

    Tenant Rules and Teacher Overrides

    A teacher assigns "unit 4, due Friday"; a coaching program requires one behavioural question per session. These are hard constraints in the filter, and the selector optimises inside them. A teacher's pinned exercise always wins, and the learner model still learns from the attempt.

    Failure Modes and What I Would Do First

    • Cold start: a new learner has no state. I start with a short placement sequence that adapts like a computerised adaptive test, choosing items that best separate ability levels, then switch to practice mode.
    • A thin content pool: if a skill has only two items, spacing turns into repetition and the learner memorises answers. I monitor items per skill and flag gaps to authors; LLM-generated variants (parameterised problems) help, but go through review.

    First version: frontier plus a simple review scheduler, a fixed target band, and epsilon-greedy exploration with logged propensities. Later: a learned scheduler, contextual bandits and off-policy evaluation.

    Deep-Dive: Socratic Hinting: Help Without Giving the Answer Away

    The hinting path: a learner's help request goes to the tutor orchestrator, which asks a context builder to combine the exercise and the learner's work with the reference solution and rubric for grounding; a hint ladder policy sets the allowed hint level, the tutor model drafts a reply, and an answer-leak guard either passes it to the learner or substitutes an authored fallback hint when it detects a leak.

    Why the Default LLM Behaviour Is Wrong Here

    A general assistant is tuned to be maximally helpful, and for a stuck learner that means writing the solution. A system prompt saying "do not give the answer" helps, but not reliably: learners ask five ways, claim "my teacher said you can show me", or ask for "just the next line" until they have all of it. Prompt instructions are a preference, not a control. So I design the tutor as a policy plus a model plus a check.

    The Hint Ladder

    Intelligent tutoring systems have long used graded hints, ending in a "bottom-out" hint that shows the step. I make the ladder explicit and enforce it in code:

    hint_ladder:
      - level: 1
        kind: orient          # restate the goal, ask what they have tried
        example: "What does the function need to return when the input is empty?"
      - level: 2
        kind: point           # name the relevant concept or the region of the bug
        example: "Look at how your loop treats the last element."
      - level: 3
        kind: strategy        # describe the approach without code
        example: "Keep two indices that move toward each other."
      - level: 4
        kind: partial_step    # one concrete step, never the full solution
      - level: 5
        kind: bottom_out      # worked step; only if tenant policy allows
    policy:
      min_attempts_before_level: {3: 1, 4: 2, 5: 3}
      min_seconds_between_levels: 30
      bottom_out_allowed: false   # per tenant, per course, per exercise mode
      assessment_mode_hints: none
    

    The orchestrator, not the model, decides the highest level allowed for this turn, from attempts made, time spent, hints given and tenant policy. Asking again does not climb the ladder; making another attempt does. That one rule defeats most "ask five ways" attempts, because more help requires more engagement.

    Grounding the Tutor in a Reference Solution

    An ungrounded tutor invents its own solution, sometimes wrongly, and confidently steers the learner astray. So the context builder always includes:

    • The reference solution (or several, for problems with multiple valid approaches), marked as private to the tutor
    • The grader's result for the latest attempt: which tests failed, with inputs; which rubric criteria were missed
    • The learner's current work, as data, never as instructions (a learner can paste "ignore previous instructions" into a code comment)
    • The common misconceptions authored for this exercise, if any

    The tutor's job is then narrow: find the gap between the learner's work and a correct approach, and hint about that gap at the allowed level. Feedback explains why test 3 failed; it does not decide on its own whether the code is correct. For open-ended exercises (a system design, a behavioural story) the reference is a rubric with exemplars, and hints point at missing criteria ("What happens when the cache is cold?", "What was the result, in numbers?") rather than supplying content.

    The Answer-Leak Guard

    Every draft passes a guard before it is sent. This is the output-check pattern from Designing LLM Guardrails, specialised for one kind of leak: the answer itself.

    def leak_check(draft: str, exercise, allowed_level: int) -> str:
        """Return 'pass' or 'block'. Runs before any token reaches the learner."""
        code_blocks = extract_code(draft)
        # 1. Structural: no code longer than the level allows
        if allowed_level < 5 and any(loc(b) > exercise.max_hint_loc for b in code_blocks):
            return "block"
        # 2. Similarity: compare against every reference solution
        for ref in exercise.reference_solutions:
            if normalized_token_overlap(draft, ref) > exercise.leak_threshold:
                return "block"
            if ast_similarity(code_blocks, ref) > exercise.leak_threshold:
                return "block"
        # 3. Semantic: a small judge model asks "could the learner copy this to finish?"
        if leak_judge(draft, exercise.prompt, allowed_level) == "gives_answer":
            return "block"
        return "pass"
    

    Design choices:

    • Cheap checks first. Code length and token or syntax-tree overlap run in milliseconds; the judge model runs only on drafts that pass them. Short-answer items get an exact check for the correct value.
    • Cumulative leakage. Five level-4 hints can add up to a full solution, so the guard compares the whole conversation's tutor output against the reference.
    • On block: regenerate once with a stricter instruction, then fall back to an authored hint for the current level. The learner always gets help, just not the answer.
    • Streaming trade-off. Hints are short, so I buffer the full hint, check it, then send. For longer explanations I check and release chunk by chunk.

    What I Would Do First vs Later

    First: the hint ladder enforced in the orchestrator, reference-grounded context, the structural and similarity guards, and authored fallbacks for the top exercises. Later: the semantic judge, cumulative leak tracking, misconception libraries mined from common wrong answers, and per-learner adaptation of hint style. I would also log every blocked draft. They are the best red-team dataset available.

    Deep-Dive: Measuring Learning, Not Engagement

    The learning measurement pipeline: an experiment assignment service sends learners through a pre-test of held-out items into either arm A with the current policy or arm B with a new policy, then a post-test with transfer items and a delayed retention check; outcome events flow to a warehouse of outcomes by arm, a batch job computes learning gains, and a dashboard reports them.

    The Metric Trap

    Product dashboards default to engagement: daily actives, minutes per session, streaks, exercises completed. In a tutor these can move the wrong way. A tutor that gives answers freely raises completion and satisfaction; a selector that serves easier items raises correctness. Both teach less.

    So I separate two kinds of metrics:

    KindExamplesUse
    Guardrail (health)Retention, session completion, tutor latency, error ratesMust not regress badly
    Outcome (learning)Gain on held-out items, transfer, delayed retentionThe thing I optimise and claim

    If a change makes learners quit, it fails. But no change ships because it raised engagement.

    Held-Out Items: Measuring Without Contaminating

    The learner model's mastery estimate is not a valid outcome: it comes from the practice items the policy chose, so a policy can inflate it by drilling. I need assessment items that practice never shows:

    • For each skill, a portion of items is marked held_out and excluded from the selector's candidate pool (the constraints filter from the previous deep-dive).
    • Pre-test and post-test draw from held-out items, with no tutor and no hints.
    • Transfer items test the skill in a new framing or combined with another skill.
    • Delayed retention checks, days or weeks later, catch policies that look good immediately but fade. This is where spacing should win and massed practice looks deceptively good.

    I treat held-out pools like ML test data: an item that leaks into practice or into the tutor's prompt is retired from assessment.

    Learning Gain

    The basic outcome is the pre-to-post change. Normalised gain, (post - pre) / (max - pre), makes learners who start high and low comparable. In SQL over the warehouse:

    SELECT
      e.arm,
      COUNT(*)                                            AS learners,
      AVG((t.post_score - t.pre_score)
          / NULLIF(t.max_score - t.pre_score, 0))         AS mean_normalised_gain,
      AVG(t.transfer_score)                               AS mean_transfer,
      AVG(t.retention_score)                              AS mean_retention_14d
    FROM experiment_assignment e
    JOIN learner_test_scores t
      ON t.learner_id = e.learner_id
     AND t.tenant_id  = e.tenant_id
    WHERE e.experiment_id = 'selector_spacing_v2'
      AND t.post_score IS NOT NULL        -- see attrition note below
    GROUP BY e.arm;
    

    That query is a starting point. The real analysis regresses post-score on pre-score with confidence intervals and accounts for attrition: if one arm makes weaker learners quit before the post-test, its gain looks better for the wrong reason. I report post-test completion per arm next to every gain and use intention-to-treat analysis.

    A/B Testing in Education Is Different

    • Randomise at the right unit. Classmates share a teacher and talk, so for classroom tenants I randomise by class or school, which needs more units for the same power.
    • Run longer and pre-register. Learning effects appear over weeks and novelty fades, so I fix the primary outcome and duration up front.
    • Tenant consent. Tenants decide whether their learners join experiments, and the control arm is always the current best policy, never a degraded one.
    • Off-policy evaluation first. Logged propensities let me estimate a new selector's short-term effect from history and filter out bad ideas before any live test.

    Evaluating the Tutor Itself

    The tutor gets its own evaluation, offline and online:

    • Offline: a regression suite of conversations, including adversarial ones, scored for leakage, hint correctness against the reference and adherence to the allowed level, run on every prompt or model change.
    • Online: leak-guard block rate, bottom-out hint rate, and the share of hinted attempts followed by an unhinted success on the same skill. A change that raises satisfaction but lowers that last number is a regression.

    Privacy, Especially for Minors

    Measuring learning needs detailed per-learner data, and some learners are children. The exact obligations depend on jurisdiction (for example COPPA and FERPA in the US, the GDPR's provisions for children in Europe) and need legal review, but the design principles are generic:

    • Data minimisation. A pseudonymous id, tenant, level and practice history; no birth date, address or photo. The tutor does not ask for personal details, and an input filter redacts any a learner volunteers.
    • The tenant controls the data. A school decides retention and can export or delete a learner's data, and deletion reaches the event log (crypto-shredding a per-learner key works well).
    • No training on learner conversations without the tenant's explicit agreement, and minors' data is excluded by default.
    • Pseudonymous analytics. The mapping to real identities lives in a separate, tightly controlled store.
    • No engagement dark patterns for children, and a stricter content policy on tutor output, applied in the same guard layer as leak checks.

    Conclusion: Bringing It All Together

    An AI tutor platform looks like a chat app with a question bank. Inside, it is a closed loop of estimate, choose, help, measure: estimate what the learner knows, choose what will move it, help without short-circuiting the effort that builds skill, and measure whether they can now do what they could not before. The infrastructure is modest; the hard parts are the models, the policies and honest evaluation.

    If I had to summarise the design in four decisions:

    1. Model mastery explicitly over a skills graph. Start with interpretable knowledge tracing (BKT plus item difficulty), fit it nightly, and keep the interface model-agnostic so deeper models can replace it once they earn it.
    2. Select for long-term learning, not in-session success. Combine the prerequisite frontier, spaced review and a target difficulty band, and explore with a bandit that logs propensities.
    3. Make the tutor a policy, a grounded model and a check. The orchestrator enforces the hint ladder, the reference solution grounds every hint, and an answer-leak guard inspects every draft before the learner sees it.
    4. Prove learning on held-out items. Pre and post tests, transfer and delayed retention, analysed with attrition in mind, with engagement only as a guardrail and children's privacy built in from day one.

    Related reading: Designing an AI Interviewer Platform, Designing an AI Candidate Evaluation System, Designing a Recommendation Engine, and Designing LLM Guardrails.

    Structured data for LLMs, AI agents, and automated crawlers is available at/blog/system-design-ai-tutor-platform.md. Please reviewrobots.txt andllms.txt before crawling. All referenced data must be credited to roundz.ai with a link tohttps://roundz.ai