Behavioral Delta Ledger: Per-Edit No-Op Detection on a Hidden Basis Derived from an Exercise's Own Tests

On a practice platform, pressing Run on an auto-graded exercise produces one bit of information and destroys the process that produced it. The learner sees pass or fail; the browser keeps only the latest source. A learner on their fifth edit cannot tell which of the five changed anything, whether the last one was a rename, or whether an earlier failing case quietly regressed. The platform cannot tell either, because nothing ever compares two of the learner's own programs.

The Behavioral Delta Ledger (BDL) makes the comparison executable. For a problem with reference f and shipped tests T, the Ledger builds a deterministic probe basis from the exercise's own test inputs (one argument perturbed at a time, exact shipped inputs excluded), executes the submission on that basis under a fresh-copy protocol, and reduces the run to a ternary signature: agree, wrong value, raise-or-timeout. After each Run it diffs the new signature against the previous one and reports counts only: how many hidden checks changed, whether the signature is byte-identical (a behavioral no-op edit), and whether the visible tests changed. The basis is hidden; the reference is recomputed locally per run; nothing is graded.

This paper reports the measurement program behind that instrument: a full-corpus census of 88,357 sampled single-edit mutants on 5,682 analyzable exercises, a study of 699 simulated edit walks under three policies, a leakage-controlled re-measurement of the in-sample concordance result, and a formal reconciliation of the cold-set object that the first draft measured incorrectly. It also reports what failed: four pre-registered predictions did not survive, and the failures removed copy from the product rather than being smoothed in the text.

• Per-edit hidden signature: a deterministic basis derived from the exercise's own tests, ternary states under a fresh-copy protocol, diffed between two consecutive learner programs.
• Census: 5,682 analyzable exercises, 88,357 sampled single-edit mutants, 10.23% invisible on the 48-probe basis, 16.45% passing all shipped tests, 45.23% of those hidden-visible, and 44.40% of problems carrying at least one visible silent slip.
• Walk study: 699 simulated walks (climb / random / revert) with exact-signature ghost rates of 48.37% / 51.16% / 33.98% after the corrected no-op definition, replacing a definition that missed wrong-to-wrong state flips.
• Held-out collapse: 84.00% (AUC 0.829) and 82.23% (AUC 0.797) under an odd-test-basis, even-test-scoring protocol, a drop of 3.78 and 1.17 points from the like-for-like full-sign in-sample rates of 87.78% and 83.40%; directional copy is cut.
• Formal cold-set reconciliation: the measured object (walk-long) and the defined object (last k = 3) disagree, H6b is dead under both, and the residual routing story is cut from the product.
• Product decision: a count-only, opt-in, standalone route that never states direction and never touches grading, review, certificates, or sync.

The paper also states a limit that applies to every learner-facing number below: there is no human data. Every walk, ghost rate, and improvement number comes from simulated edit policies or synthetic cohorts on the real corpus. Whether real learners produce no-op edits at these rates, and whether the count card helps them, is unmeasured. The first human experiment proposed here is a local shadow log of consecutive signature pairs.

Differential testing (McKeeman, 1998) runs two comparable programs on shared inputs and treats divergence as the oracle. BDL is a differential test in miniature, but three things differ: the setting is a learner's own consecutive submissions rather than independent implementations; the input coordinate system is a deterministic basis derived from the exercise's own tests relative to the shipped reference; and the product is a count of changed checks with a no-op statement, never a verdict. Regression testing supplies the fix/break intuition; the measured held-out concordance bounds how far that intuition can be carried, and the Ledger's copy stops short of it.

Mutation testing is the parent methodology. DeMillo, Lipton, and Sayward (1978) framed test data selection around seeded simple faults; Jia and Harman (2011) and Papadakis et al. (2019) surveyed the field; Just et al. (2014) gave the strongest evidence that mutants correlate with real faults. BDL samples the same single-edit families as the wave-41 census, but its object is not a mutation score: it computes the behavioral delta of an edit rather than the survival of a mutant, and it is relative to the learner's previous program, not to an absolute reference. Inozemtseva and Holmes (2014) showed coverage is a weak proxy for suite effectiveness, which is why the Ledger reports counts over a basis instead of a score. DSpot (Danglot et al., 2019) amplifies developer tests, and the position that the individual surviving mutant matters more than an aggregate adequacy threshold is also the position this paper reaches independently, by measuring the estimator rather than the score. The Ledger is adjacent to both but never modifies a shipped suite or gates a learner.

On the education side, Jadud (2006) characterized novice compilation behaviour and flailing from error streams; the Ledger measures futility behaviorally, judging an edit by whether it changed observable behavior rather than by its text, error count, or timing. Examplar (Prasad et al., 2024) uses curated chaff mutants to test student-authored examples; BugSpotter (Padurean et al., 2025), When AI Is Wrong on Purpose (2026), FPPgen (Caraco et al., 2024), and Katabench all generate buggy code as practice. Those bugs fail by construction or are findable by writing tests; BDL's subject is the learner's own edit, which may change nothing at all. Errorful learning (Kornell, Hays, and Bjork, 2009; Metcalfe, 2017) and desirable difficulties (Bjork and Bjork, 2020) motivate the opt-in replay shelf, but the pedagogy is old and the paper makes no learning-outcome claim. Spacing references (Cepeda et al., 2006; Karpicke and Bauernschmidt, 2011; the FSRS algorithm) describe the scheduler the Ledger deliberately does not touch.

Delta debugging (Zeller and Hildebrandt, 2002) minimizes a known-failing input; a hidden probe exists where no shipped input fails. Machine teaching (Goldman and Kearns, 1995; Zhu, 2015) asks which examples identify a concept; BDL asks a different question, whether an edit changed anything on a fixed hidden sample. STING (2026) augments benchmarks with tests that kill surviving program variants; BDL's wave-41 neighbor, the closure result, shares that goal, while BDL itself produces a feedback count and no new tests at all.

A problem is a triple p = (S, f, T): a statement S, a Python reference f, and a shipped test set T with three to six cases. The basis B(p) is built from the first three shipped inputs by perturbing exactly one argument at a time using the frozen perturbation set (empty, singleton, reversed, duplicated, extended, shrunk, negated, sorted, incremented; matrices are also transposed and row-perturbed). Exact shipped inputs and duplicates are excluded; the candidate list is shuffled with a seed derived from the problem id and the first 48 (research) or 24 (product runtime) probes with a reference answer are kept. Probes where the reference itself raises or times out define no answer and are dropped from comparisons.

sigma_j(g) = 0 if g(args_j) returns and deep_eq(g(args_j), f(args_j)); 1 if g returns a different value; 2 if g raises or times out. The reference state x means f has no answer at that probe.

Execution follows the fresh-copy protocol: every call runs on a fresh deep copy of the probe arguments. This is not a stylistic choice. A dedicated pass over the corpus found 8 of 5,682 references (0.14%) mutate their argument objects; a shared-object protocol flaked on 9 problems and disagreed with fresh-copy on 8, while the fresh-copy protocol flaked on 0. The same pass exposed a second bug in the transform check: consistent local renaming initially reported churn on two nested-function problems because nonlocal declarations were not renamed with the variable. After the fix, rename churn is 0 on 4,680 of 4,680 applied renames and cosmetic churn is 0 on 5,682 of 5,682.

Churn(sigma, sigma') = |{ j : sigma_j != sigma'_j }| over comparable dims; ghost = text_changed and sigma = sigma' exactly and the visible test bitmask is unchanged.

The walk study used three policies over a stride-24 sample: climb (edit maximizing tests passed), random (uniform single edit), and revert (alternating edit and undo), each 233 walks over 2,330 / 2,330 / 2,796 steps. Walk RNG seeds are derived from the problem id and the policy index. Sub-basis sufficiency was measured on a stride-12 sample (477 problems) against the 48-probe research basis and on a 600-problem stratified sample against a 96-probe reference. The leakage protocol is separate: the basis is built from odd-index shipped tests only and the delta is scored on even-index tests over independent walks (n = 18,191 steps). The clean corpus run used a truncating writer with no resume path, one fresh record per problem.

# p = (id, f, T); f is the reference, T the shipped tests
B = perturb_one_argument(first_three_inputs(T))   # frozen perturbation set
B = dedupe(exclude_shipped(B))
B = shuffle(B, seed=int(md5(id)[:8], 16))          # product seeds with FNV-1a instead
B = [b for b in B[:cap] if reference_answers(f, b)]  # cap 48 research / 24 product

sig(g) = [state(g, b) for b in B]                  # 0 agree, 1 wrong, 2 raise/timeout

churn = count(sig_before[j] != sig_after[j] for j in comparable)
ghost = text_changed and sig_before == sig_after and mask_before == mask_after
fix   = count(sig_before[j] != 0 and sig_after[j] == 0)
break_ = count(sig_before[j] == 0 and sig_after[j] != 0)

The clean run processed all 5,730 exercises; 48 have fewer than five reference-evaluable probes and are skipped, leaving 5,682 analyzable problems. Mutant sampling generated 88,357 single-edit mutants, of which 14,534 (16.45%) pass every shipped test. The hidden basis sees 6,574 of those passers (45.23%) and misses the other 54.77%; 9,041 mutants (10.23%) are invisible on the basis, and 2,523 problems (44.40%) carry at least one test-passing, hidden-visible slip. The test-passing rate is close to wave 41's 16.50% (16.45% here), which is a useful coarse consistency check but not a replication: the two runs sample mutants under different caps (at most 24 per problem here, at most 36 there) over slightly different analyzable sets (5,682 against 5,721), so the two percentages are not estimates of the same quantity and their 0.05-point proximity should not be read as agreement.

Family structure reverses the pre-registered expectation. Comparison swaps are the blindest family: 24.75% of cmp mutants are invisible, because changing <= to < or == to != often changes nothing on the generated inputs; deletion families are the least blind (notdel 1.47%, notins 1.62%). int, the other family predicted to be nearly fully visible, sits at 16.30%. The prediction H2b is falsified, and per-family labels were removed from every learner surface as a consequence: family rates are a corpus fact about this edit model, not a diagnosis of a learner.

Two denominator corrections matter for reading the tables. 55 of 5,682 problems generated no evaluable mutant at all, and problem-level rates condition on the 5,627 that did; the all-sampled-mutants-visible share is therefore 2,818 / 5,627 = 50.08%, not the 49.6% printed in the first draft with the wrong denominator. Within problems that do have mutants, sensitivity (the mean fraction of a problem's sampled mutants that change the signature) is p10 0.283, median 0.551, p90 0.838.

A smaller basis still sees most of what the research basis sees. Against the 48-probe universe, a 16-probe basis recovers 81.2% of the visible silent slips and a 24-probe basis 91.8%. Against a stricter 96-probe reference on a 600-problem stratified sample, the 48-probe basis sees 96.12% (669/696) of the reference's test-passing slips, the 24-probe basis 86.93%, and the 16-probe basis 76.72%. Sign-level agreement is stronger than coverage: at the step level the 16- and 24-probe bases preserve the sign of the 48-probe impact on 96.19% and 97.83% of steps in the test-positivity sense, and on 92.40% and 95.87% in the full-sign sense; the 24-probe basis estimates the 48-probe agreement rate within +/-0.10 for 88.4% of steps (median error 0.024). The product uses 24 probes because these bounds suffice for a count, not a verdict.

The walk study simulates a learner editing a broken program for ten to twelve steps under three policies: climb, random, and revert. Exact-signature ghost edits, after the corrected definition, are 48.37% of climb steps, 51.16% of random steps, and 33.98% of revert steps. Almost all ghosts are the same wrong behavior repeated: the walks start broken and the edits keep the program broken in exactly the same way. The old Churn = |Fix| + |Break| definition reported higher rates and, worse, miscounted 6.86-10.44% of its own 'ghosts' - edits that actually flipped a probe from one wrong state to another. Only the corrected definition is used anywhere in this paper or the product.

The sign of a hidden delta agrees with the sign of the graded test delta far more often than chance - but only in-sample. On the full-sign statistic used throughout this section, in-sample concordance is 87.78% for climb and 83.40% for random; the positive-only in-sample rate of the first draft (90.20% climb, AUC 0.903) conditions on steps with a nonzero test delta, is a different statistic on a different denominator, and is not used as a reported result. Under the leakage-controlled protocol, with the basis built only from odd-index tests and the delta scored on even-index tests, held-out concordance is 84.00% (AUC 0.829) for climb and 82.23% (AUC 0.797) for random over 18,191 independent steps. The like-for-like drop is 3.78 points for climb and 1.17 for random. The correct reading is that the basis is a genuinely informative proxy for the graded contract, and not informative enough to state a direction to a learner. That is precisely why the product reports whether the signature changed and how many dims changed, and never whether the edit improved the program.

The next-edit version of the question is where the forecasting story dies. Breaking hidden dimensions raises the risk of a next-attempt test regression by 1.14x (95% CI [0.88, 1.47]) under climb and 1.36x ([1.04, 1.79]) under random; the random arm's interval excludes 1, but both are far below the pre-registered 2x threshold. The next-improvement AUC is below chance (0.308 and 0.359) and the next-regression AUC is weak (0.667 and 0.626). The honest use of the delta is retrospective attribution - this edit did change or did not change the measured behavior - not a prediction of what happens next.

RR = P(test regression at next attempt | hidden dims broken) / P(test regression | hidden dims not broken) = 1.14 [0.88, 1.47] (climb); 1.36 [1.04, 1.79] (random).

The first draft defined the cold set on the last k = 3 attempts but measured it over every version of each walk. Those are different objects, and the difference is large. Under the walk-long object, 49.79% of climb walks and 74.68% of random walks end with at least four cold dims; under the formal last k = 3 object the shares rise to 85.84% and 93.56%, because a three-attempt window is easier to keep fully wrong than a ten-step history.

The pre-registered family-coherence prediction H6b is dead under both objects. A single perturbation family covering at least 60% of the cold set with at least 1.5x excess over its basis share occurred in only 1.72% (walk-long) and 1.50% (formal) of climb walks, and 1.15% / 0.46% of random walks, against a predicted 40%. The aggregate enrichment is real but post hoc and modest: under the shipped within-walk permutation null (300 draws, seed 12345) on the formal object, empty is under-indexed at z = -11.8, zero1 at -3.8, and zero at -3.7, while reverse (+3.1), inc (+2.5), inc2 (+2.5), sort (+1.9), dup (+1.8), and extend (+1.7) are over-indexed. The walk-long recompute points the same way (empty -11.7, inc +4.6, reverse +4.1). The old z-values (empty -20.5, inc +8.2, reverse +7.1) came from a null that was never shipped and are withdrawn.