Ladder-Graded Spacing: execution-aware scheduling for code practice

DeepForge is a practice platform where learners implement machine-learning and math building blocks in Python from scratch and receive immediate, execution-verified feedback in the browser. Every solved problem enters a spaced review queue: on a due date the learner re-solves the problem from scratch, and the scheduler decides when it returns. That queue is the one place where the platform applies memory science on a schedule of its own rather than the learner's.

The queue inherits from SM-2. A session is graded by qualityFromRun, collapsed to one of 0/3/4/5, and turned into an ease factor and an interval stored as a single record per problem. The collapse throws away signals the platform already owns: how many test runs failed before the pass (failedRuns), which hint tier was revealed (hintTier in 0..3), and whether the code buffer was reset before the pass (resetBeforePass). In the current path the reset flag is not even passed from the problem view on an ordinary solve, so a hint-heavy session after three failures and a clean first-try solve can land on nearly the same ease and interval.

The motivation is not that self-report is dishonest — the platform never asks the learner to self-grade a solve. It is that an execution ladder is objective evidence about difficulty and effort, and the three-value collapse discards it. Failed test runs are observable errors, a stronger signal than the response-time proxies now entering scheduling research, and hint tiers are earned by a budget rather than offered on request: tier 2 requires one failed run or three minutes of struggle, tier 3 requires two failed runs or six minutes. A hint that deep in the session records how hard retrieval actually was.

This paper specifies Ladder-Graded Spacing (LGS), the technique invented in DeepForge research wave 40, and reports its measured behavior. LGS keeps the ladder and changes the model around it: a ladder observation turns the raw signal into an effective failure count phi; phi updates an observed difficulty D and an effort EWMA; success growth uses the FSRS-6 long-term stability formula; failure uses the FSRS-6 post-lapse stability instead of resetting to one day; and an effort-driven cap bounds the interim interval for repeatedly struggling items. Queue order and the readiness curve move to the same quantity, predicted retrievability R(t, S) under a power-law forgetting curve.

• Execution evidence: define phi from failed runs, hint tier, and reset, with explicit clamping for junk input.
• Fixed-parameter memory model: FSRS-6 defaults, no per-user fitting, no network, no randomness, no new dependencies.
• Ladder channel: phi updates D and effort; in the shipped variant it does not multiply stability.
• Migration guard: existing SM-2 records upgrade in place with due dates byte-identical and idempotent output.
• Simulation: a deterministic, two-run-identical protocol with paired-by-learner bootstrap gates, plus edge-learner cohorts.

Spacing is one of the most replicated effects in learning research, but the shape of the schedule is contested. Cepeda et al. (2006), across 839 assessments, found roughly a 15% average benefit for long retention intervals and a non-monotonic lag function: for a given retention interval there is an optimal inter-study gap, and the optimal gap grows with the retention interval. Expanding versus fixed is not settled in favor of expanding: Karpicke & Bauernschmidt (2011) found absolute spacing drove large gains while expanding, equal, and contracting relative schedules were indistinguishable; Latimier et al. (2021) found no expanding-versus-uniform difference (g = 0.034, n.s.) across 54 effect sizes; Kang et al. (2014) found expanding equal at an eight-week final test; Storm et al. (2010) found expanding wins when interference is high. Task complexity moderates the effect: Donovan & Radosevich (1999) report d = 0.46 overall but near-zero for high-complexity tasks (0.07-0.11) versus 0.97 for psychomotor tasks, so spacing should help a long interview problem, but by less than the flashcard literature implies.

FSRS models difficulty, stability, and retrievability (DSR) and schedules to an explicit desired retention. FSRS-6 (21 weights, optimizable forgetting-curve decay) ships in Anki since 25.07 (Anki PR 3929); FSRS-7 (35 weights, fractional intervals, same-day predictions) merged into the benchmark in March 2026, and the fsrs-rs integration PR merged on 2026-09-08. The original algorithm paper is Ye et al., KDD 2022, and a browser-ready TypeScript implementation (ts-fsrs) exists for the web stack. Calibration gains are large and replicated: on about 350 million held-out reviews, FSRS-6 log loss is 0.346 versus 0.436 for FSRS v3; on the earlier 20,000-collection benchmark, SM-2 scored 0.73 log loss and 0.21 RMSE(bins) versus FSRS v4 at 0.39 and 0.046 (Expertium breakdown), and the corrected FSRS-7 benchmark figure is 0.3401 against 0.3620 for default parameters. The benchmark authors themselves caution that SM-2 never predicted probabilities, so converting its intervals to probabilities requires assumptions; the comparison is directionally strong, not perfectly fair.

Those numbers are flashcard recall, and execution-verified multi-minute problems are out of distribution for them. Community simulations report 20-30% fewer reviews for the same retention, and per-user optimization beats default parameters for roughly 84% of collections, but the Anki manual warns optimization needs at least a few hundred reviews to generalize. DeepForge stores one state row per problem and has no per-review log, which is exactly what an optimizer needs. FSRS's own benchmark shows default parameters give up much of the edge of optimized ones, so LGS is honest about being fixed-parameter and must be validated by simulation, not assumed.

Attempt-level frequency models support the shape LGS borrows. DASH and DAS3H count correct and attempted responses with log transforms — diminishing returns, the power law of practice — and beat IRT baselines on tutoring data. HLR predicts recall about 45% better than Leitner, but needs hundreds of traces per item, which DeepForge does not have. Bayesian and deep knowledge tracing are rejected here for the same data reason (Corbett & Anderson 1994; Piech et al. 2015): per-step tracing needs step logs or large samples per skill, and LGS stays at review-event level. The lesson taken is narrower: more failures should earn less credit, monotonically, with a fixed floor.

Objective outcome cues are entering scheduling research. ARTS used response time plus accuracy and beat fixed spacing at both immediate and delayed tests; SSP-MMC++ reports a 22.8% improvement in target-half-life achievement and a 16.8% reduction in retention cost. But step-level response-time propensities are ambiguous and proficiency-moderated, and DeepForge does not persist time per problem anyway. LGS therefore uses errors and hints — both persisted and both objective — and explicitly avoids wall-clock features.

Unsuccessful retrieval followed by immediate feedback teaches something: Kornell, Hays & Bjork (2009) and Hays et al. (2013) found that failed retrieval with immediate feedback enhances later learning. Retrieval practice itself is strong evidence (Rowland 2014 reports g around 0.50, stronger for effortful recall), but difficulty has a floor: it helps only when the learner has the prior knowledge to succeed, otherwise it is an undesirable difficulty (Bjork & Bjork 2020). In-app feedback is immediate and execution-verified, so a struggled pass is discounted by LGS but never treated as a lapse. This is also why the ladder is legitimate memory evidence rather than a punishment.

Assistance is not free. Hint shortcuts correlate with worse learning (Stamper, LAK 2026), hint use improves help-need prediction (Maniktala 2022), and the assistance dilemma (Koedinger & Aleven 2007) is well documented. The evidence is weak in one specific place, and LGS states it plainly: nothing published maps hint tiers or failed runs to a stability multiplier. The hintDebit values in LGS are chosen, swept in simulation, and first to cut if calibration fails.

Item difficulty from outcomes is usually estimated with Elo-style updates that approximate IRT quality (Pelánek 2016). LGS uses a single exponential update because there is one local user and no matchmaking; that is weaker than a fitted IRT model and is flagged as such. Finally, leech thresholds are lore: Anki practice tags and suspends after a configurable lapse count (Anki manual), but no controlled trial validates the threshold or the actions, and FSRS-era statistical leech detection remains a design discussion rather than shipped behavior.

LGS is a pure function of stored state, an execution signal, and an injected clock. It has no dependence on wall-clock time, randomness, the network, or any new package, so the same inputs produce identical output on every run. The grade-time signal is passed flag, failed runs (an integer, at least zero), hint tier in 0..3, and a reset flag.

The ladder observation reduces the signal to an effective failure count phi. Failed runs and a reset form the retrieval part, with the reset worth 0.5 of a failure because a reset may be refactoring rather than forgetting. The hint part adds a per-tier debit. Every input is clamped before it touches arithmetic: hint tier is floored and clamped to 0..3, failed runs are non-negative integers, and non-finite values read as zero, so junk input can never propagate NaN into a due date.

phi = min(fMax, f + (reset ? wReset : 0)) + hintDebit[clamp(0, 3, floor(hintTier))]

Memory is represented by a power-law forgetting curve with FSRS-6's default decay and a stability S measured in days at a 0.9 retrievability target. The factor F is derived, not fitted, so that R(S, S) = 0.9 exactly; the interval that returns to 0.9 is exactly S, which keeps the target retention semantic and makes intervals and stabilities interchangeable.

R(t, S) = (1 + F * t / S)^(-d), F = 0.9^(-1/d) - 1 = 0.9803464944134797, d = 0.1542; I(r, S) = (S / F) * (r^(-1/d) - 1)

On a pass, stability grows with the FSRS-6 long-term formula. This is the shipped variant, called noE: the ladder is deliberately absent from the stability multiplication. On a failure, stability uses the FSRS-6 post-lapse formula, which preserves part of a mature item's interval rather than resetting to one day, while still scheduling weak items soon. Both branches then update difficulty and effort from phi.

S' = S * (1 + exp(w8) * (11 - D) * S^(-w9) * (exp(w10 * (1 - rPred)) - 1))

S' = w11 * D^(-w12) * ((S + 1)^(w13) - 1) * exp(w14 * (1 - rPred))

Difficulty is an observed value: the model computes what difficulty the execution ladder implies and moves the stored D toward it with a fixed learning rate. In the noE variant this is the ladder's main channel on passes. Effort is an EWMA of normalized struggle that persists across reviews: a failed review contributes a larger increment than a pass, and any clean pass halves the accumulator. When accumulated effort reaches the cap, the next interval is bounded to one week, so repeated struggle cannot buy a long absence. The cap is entered by lapses or repeated high-phi reviews, not by a single struggled pass: from effort = 0, even a maximal phi = 16 review only reaches effort' = 0.5.

D_obs = pass ? 3 + 7 * phi / (phi + 6) : 10; D' = clamp(1, 10, D + eta * (D_obs - D)); e = clamp(0, 1, (phi + (pass ? 0 : wForget)) / 8); effort' = 0.5 * effort + 0.5 * e

Scheduling then writes the standard fields. Elapsed days come from calendar date keys, never wall-clock deltas: the difference from lastReviewedAt when present, otherwise the stored interval plus any days past due. This matters for determinism across time zones and daylight-saving transitions; the specification's original elapsedDays was undefined and changed results by roughly a factor of three until it was pinned to date keys.

S' = clamp(0.1, 36500, S'); interval = clamp(1, 36500, round(I(0.9, S'))); if effort' >= 0.6 interval = min(interval, 7); due = addDays(todayKey, interval); reps' = pass ? reps + 1 : 0; lapses' = pass ? lapses : lapses + 1

Queue order and the readiness curve read the same model. Due items sort by predicted retrievability ascending, so the most-forgotten item is first, with the existing weak-category, due, difficulty, and id tiebreaks after that, then the unchanged category interleaving and a cap of ten. Readiness uses R(t, S) for scheduled items and R(t, S0) with S0 = 2.3065 (the FSRS-6 initial Good stability) for unresolved items; the 0.7 retained threshold is unchanged.

Existing records upgrade through a guarded migration that runs on every read but only derives what is missing. Stability, difficulty, and effort are derived from the legacy fields when the record is not version 2, or when the specific field is missing, non-finite, or out of range; valid version-2 fields are copied verbatim. Everything else — due date, interval, reps, lapses, lastGrade, lastReviewedAt, and the frozen ease rollback field — is preserved exactly. The output key order is fixed, so migrating twice is byte-identical to migrating once, and a stale version-1 record that wins a merge is re-migrated on the next read. No storage key, backup inventory, or merge rule changes; the new fields travel atomically with the record under the existing last-write-wins rule, and field-level merging of S, D, or effort is explicitly forbidden because max and average are wrong for a sequential update.

S = reps > 0 ? clamp(0.1, 36500, interval) : 1; D = clamp(1, 10, 5 + (2.5 - ease) * 2); effort = {0: 0.9, 3: 0.6, 4: 0.3, 5: 0.1, null: 0}[lastGrade]

const phi = ladderPhi(signal, params);
const rPred = lgsRetrievabilityAt(elapsed, S, params);
const rawS = signal.passed
  ? S * (1 + Math.exp(w8) * (11 - D) * Math.pow(S, -w9) * (Math.exp(w10 * (1 - rPred)) - 1))
  : w11 * Math.pow(D, -w12) * (Math.pow(S + 1, w13) - 1) * Math.exp(w14 * (1 - rPred));
const nextS = clamp(0.1, 36500, rawS);
const dObs = signal.passed ? 3 + (7 * phi) / (phi + 6) : 10;
const nextD = clamp(1, 10, D + eta * (dObs - D));
const e = clamp(0, 1, (phi + (signal.passed ? 0 : wForget)) / 8);
const nextEffort = 0.5 * effort + 0.5 * e;
let interval = clamp(1, 36500, Math.round(lgsIntervalFor(nextS)));
if (nextEffort >= effortCap) interval = Math.min(interval, capInterval);
return { ...next, due: addDays(todayKey, interval), S: nextS, D: nextD, effort: nextEffort };

Because there is no per-review log and no external dataset of execution-graded code practice, LGS was validated in a deterministic simulation rather than on user data. The cohort is 100 learners, 40 items, and a 365-day horizon, extendable once to 400 learners if a result is inconclusive. True ability is drawn a ~ N(0, 0.7^2), true item difficulty delta ~ N(0, 1), and true initial stability S* ~ lognormal(0.8, 0.5) clipped to [0.3, 20]. A synthetic hint policy mirrors the real one: tier 2 after one failure, tier 3 after two. Attempts are Bernoulli with a per-session probability, up to four failures per session.

The protocol was corrected during verification. The original ground truth used the full session pass probability in the memory update, which made the literal protocol degenerate: every arm retained 4000/4000 items and the comparison carried no information. The corrected truth model updates memory from memory retrievability alone, while the session pass probability multiplies in an ability offset, p = Rmem * sigma(a - delta). Acceptance uses the memory-only truth model with ability-offset interpolation at first-attempt recall 0.5; the literal numbers are reported for continuity but are not the basis for shipping.

Three arms are compared. The lgs arm is the shipped noE scheduler. The sm2 arm is the exact current code path: qualityFromRun plus the hint penalty plus the existing grade function. The oracle arm knows the true stability S* and serves as a ceiling, not a competitor. The primary metric is reviews per item still retrievable at day 365. Each arm's pre-review prediction is also scored with Brier loss; SM-2's interval is converted to a probability by the same curve, because SM-2 never emitted probabilities. Secondary metrics are first-attempt recall, the overdue-failure rate, and p95 reviews per day.

• Cohort: L = 100 learners (extend to 400 once), M = 40 items, 365 days.
• Truth: memory-only update with ability-offset session probability; hints tier 2 after one failure, tier 3 after two.
• Arms: lgs (noE), sm2 (current path), oracle (true S*, ceiling only).
• Stopping: paired-by-learner bootstrap, 2,000 fixed-seed resamples; stop when the 95% CI of LGS minus SM-2 excludes zero, else extend once, else declare inconclusive.
• Gates: at least 10% fewer reviews at matched recall, Brier no worse than SM-2 by more than 0.01, no edge cohort above 1.10x SM-2 reviews, and two runs byte-identical.

The default spec with the E multiplier failed the primary gate: 3.10% fewer reviews, 0 of 30 seeds at or above 10%. Lowering the tier-3 hint debit to 1.5 passed (14.31% across seeds), and the ablation attributed the E cost to +27% reviews with no calibration gain. That is why the chosen variant drops E: it clears the gates broadly, improves calibration, and keeps every edge cohort inside the 1.10x bound.