Alibi Distance: measuring and closing the silent-failure aperture of a verified exercise corpus

DeepForge is a practice platform whose catalog is machine-verified: 5,730 Python exercises, each with a reference solution that provably passes its shipped tests under real execution, a 1e-6 deep-equality harness, and a per-call time budget. That verification is a claim about the catalog, not about the tests. A shipped suite has three to six cases, so passing it is a finite condition. A program that agrees with the reference on every shipped case and disagrees somewhere else is wrong in the only sense a grader can see, and the grader cannot see it at all. We call such a program an alibi: it was at the scene, but the evidence cannot convict it.

This paper measures how close the nearest alibi sits. The alibi distance alpha(p) is the minimum number of single-edit mutations of the reference that produces a program passing every shipped test and diverging from the reference somewhere in a deterministic, per-problem probe bank. alpha = 1 means a one-line slip passes the entire suite and is behaviorally wrong somewhere the suite never looks. This is a radius/weakest-link framing: it asks not how many faults a suite kills on average, but how far the nearest silent nearest-miss is.

The measurement is a build-time operation. It needs the reference, the tests, real Python execution, and a fixed seed; it needs no network, no model, and no telemetry about learners. That is a deliberate fit for a server-free, LLM-free product: the same resources that make the catalog verifiable make its aperture measurable. The measurement reported here was constructed in DeepForge research wave 41 and was run over the whole corpus by two separately written engines, which is why almost every number below arrives twice. The components it uses — single-edit mutants, the surviving-mutant set, a witness plus its reference output, greedy set cover over witnesses, and ordinary cross-validation — are standard mutation-analysis and differential-testing practice; the assembled procedure and the minimum-radius aggregation are what this paper describes.

The paper is candid about what is new and what is not. Closing a suite against a surviving mutant is textbook mutation-test augmentation; the witness plus the reference output is the generated test, and no generation step is needed. What we contribute is: (i) a census of radius-1 silent survivors over this platform's own corpus of 5,730 verified exercises, computed with real execution and cross-checked by a second, separately written engine — not a priority claim, because survival of CS autograding suites has been measured in the published literature for over a decade and the aggregation, not the phenomenon, is the difference; (ii) the radius/weakest-link metric as a corpus statistic rather than a per-program adequacy score; (iii) two clean negative results that stop a plausible next step before it is built; and (iv) two audits of ten alibis each, one sampled blind and one not, that give a first estimate of how often the mined objects are defects, with a wide interval at that sample size. The contributions in full:

• Census: 106,081 sampled mutants over 5,730 exercises; 17,502 pass all shipped tests (16.50%); 7,727 of those diverge in the probe bank; 46.08% of analyzable problems admit a radius-1 alibi. These are records of a mining run whose programs are not in the release (Section 9), so they are not independently reproducible from a checkout.
• Second engine: a from-scratch engine reports 105,425 mutants, 16.57% pass, 45.94% affected, 73.25% best-probe kill; across Table 7 the census-rate rows agree to 0.14 percentage points, the length-quartile rows to 0.32, and the largest single gap is 0.44 on cross-validated held-out kill. Both engines are by this paper's author and neither is shipped, so this is an internal agreement record rather than external replication.
• Closure accounting: one oracle-chosen witness kills 73.22% of alibis and fully closes 59.71% of affected problems; a cross-validated witness kills 48.35% of held-out alibis, making the deployable estimate 23.5-23.8% still affected, not the in-sample 18.6%.
• Negative result: feature-guided probe policies transfer no better than random (63.3% random vs 59.5% guided conviction within ten probes).
• Negative result: best <=2-literal predicates reach 0.9 balanced accuracy for only 32.2% of alibis, against a permutation control of 0.881.
• Audits: a blind audit of ten alibis sampled independently of the author's first ten found 10/10 plainly wrong, judged by this paper's author; a separate non-blind author audit found 8 wrong, 1 immaterial, 1 subtle. Neither audit record is committed.
• Artifact: a frozen, machine-verified set of 96 mined alibis that resists an uncapped union lazy suite and an independently constructed held-out suite, shipped as the Silent Bug Hunt practice route, with no grading, review, or certificate integration.

Mutation testing is the parent technique. DeMillo, Lipton, and Sayward framed test data selection around deliberately seeded simple faults; Jia and Harman surveyed the field's first decades; Papadakis et al. surveyed the following decade and the equivalent-mutant problem that dominates its practice. An alibi is a surviving mutant under a fixed operator family and a deterministic execution probe; what differs here is that the survival set is used to build a minimum-distance statistic over an exercise corpus rather than a mutation score for program p under test.

The closest published system is STING (2026), which generates surviving variants of a reference patch on SWE-bench and synthesizes targeted regression tests to kill them, reporting that 77% of instances admit at least one surviving variant. The goal is the same as witness closure here; the mechanism differs in two ways. STING generates tests with a model and validates them against behavior-preserving transformations, while our closure appends the witness input and the reference's own output directly, so the added test cannot reject the reference and no test synthesis is involved. STING targets agent benchmarks; this paper targets a verified education corpus and reports the closure's generalization honestly, including its loss. Its aperture and ours are not comparable and should not be lined up: its 77% is over SWE-bench Verified instances under LLM-based mutation, while 46.08% is over exercises under eleven fixed operator families with no generated variants, and the denominators are different objects. The ordering of the two numbers says nothing about either study.

Education-side work shares the motivating phenomenon but inverts the object. Examplar and conceptual mutation testing (Prasad, Greenman, Nelson, and Krishnamurthi, 2024) check whether student-authored examples pin a problem down using curated chaff mutants; our mining checks whether the platform-shipped suite pins the problem down and produces a corpus metric, not a student-facing exercise. BugSpotter (Padurean, Denny, and Singla, SIGCSE 2025) and When AI Is Wrong on Purpose (2026) generate buggy code and have students produce failing tests. Their bugs fail the suite by construction and are findable; an alibi passes the suite, which is exactly the silent case a finite test set can miss. FPPgen (Caraco, Lojo, Verdicchio, and Fox, 2024) and Katabench both ship mutant-as-practice: students write tests graded by whether planted faults are caught. Those products are practice surfaces; neither reports a corpus census or a minimum-distance measure.

On the testing side, Just et al. (2014) is the strongest available evidence that mutants correlate with real faults, which is the coupling assumption this paper keeps at arm's length: their result is about fault detection correlation in Java programs with real fault data, while this corpus has no human false-pass data at all. Inozemtseva and Holmes (2014) showed coverage is a weak proxy for suite effectiveness, which is the same warning this paper's mutant-count-vs-length confound echoes in miniature. DSpot (Danglot et al., 2019) amplifies developer-written tests, and differential testing (McKeeman, 1998) runs two comparable programs against the same inputs and treats divergence as the oracle. The alibi duel in the shipped practice route is differential testing in miniature: reference and ghost run locally, and the divergence fact is the verdict.

Delta debugging (Zeller and Hildebrandt, 2002) minimizes a known-failing input; there, an input is known to fail, while an alibi witness exists precisely because no shipped input fails. The minimal pair in linguistics is the conceptual ancestor of the witness, and machine teaching (Goldman and Kearns, 1995; Zhu, 2015) asks which examples identify a concept, but that literature concerns example selection for a learner rather than measuring the aperture of a shipped suite. The engineering position this paper takes — act on the individual surviving mutant rather than on an aggregate score threshold — is also the position Section 5 reaches independently, and for a different reason: the score's apparent strength is partly in-sample, which is a statement about the estimator and not about the aggregate. Finally, near-miss exposure is a plausible practice object under the desirable-difficulties account (Bjork and Bjork, 2020), but this paper makes no learning-outcome claim.

A problem is a triple p = (S, f, T): a natural-language statement S, a partial reference function f in Python, and a shipped test set T = {(x_i, y_i)} with y_i = f(x_i) and, in this corpus, three to six cases. The platform supplies two step functions: eval_f(x), which runs f under the harness semantics (1e-6 relative-or-absolute deep equality, a 0.25 s per-call alarm, first def as entry point), and pass(g, T), which runs g against every shipped case. Nothing about S is machine-readable; that fact bounds everything below.

The edit model E is a set of AST rewrites, each applied once. The census used eleven families. One direction applies per family for cmp, bool, bin, aug, func, notins, notdel, and negdel; int, idx, and lenshift are two-directional per target node. The number of concrete rewrites for a given problem therefore depends on the reference AST, and no fixed rewrite count is claimed. Distance d(f, g) is the minimum number of family applications taking f to g; it is slip distance, the size of the human error class being modeled, not semantic distance.

The probe domain D(p) is deterministic and per-problem. The first three shipped inputs are perturbed one argument at a time: empty, singleton, reversed, duplicated, extended, shrunk, negated, sorted, and incremented variants; matrices are also transposed and row-perturbed. Duplicates are removed, shipped inputs come first, and the bank is shuffled with seed int(md5(problem_id)[:8], 16) and capped at 48 entries. Across the corpus the mean bank holds 39.05 probes, of which 89.22% are reference-evaluable: only probes on which the reference succeeds can serve as witnesses, because the comparison needs a reference output to diverge from. A 300-problem robustness rerun with the cap raised to 120 moved the aperture from 45.3% to 46.7% (4 problems gained, 0 lost), so the 48-probe cap is not what produces the headline rate.

Mining samples at most 36 single-edit mutants per problem, at most 6 per family, with the target order shuffled by seed int(md5(problem_id)[:6], 16). A control step first requires ast.unparse(ast.parse(f)) to pass T, so formatting round-trips cannot inflate survival. The pass set, the divergent programs, and the metric are then defined as follows, where D_eval is the reference-evaluable part of the bank and A1(p) is the mined radius-1 alibi set.

Pass(p) = { g in E(f) : g compiles and pass(g, T) } ; D_eval(p) = { x in D(p) : eval_f(x) succeeds } ; Diverge(g) = { x in D_eval : eval_g(x) != eval_f(x) or eval_g(x) fails/times out } ; Alibi(p) = { g in Pass(p) : Diverge(g) != empty } ; alpha(p) = min { d(f, g) : g in Alibi(p) }

# input: p = (id, f, T)
assert pass(unparse(parse(f)), T)                      # round-trip control
D  = probe_bank([x for x, _ in T], seed=int(md5(id)[:8], 16), cap=48)
O  = { i: eval_f(D[i]) for i in range(len(D)) if ok(eval_f(D[i])) }
M  = sample_mutants(f, cap=36, per_family=6, seed=int(md5(id)[:6], 16))
P  = [m for m in M if compiles(m) and pass(m, T)]       # survivors
W  = { m: [x for x in O if diverges(eval_m(x), O[x])] for m in P }
A1 = { m: W[m] for m in W if W[m] }                     # radius-1 alibis
best_probe = argmax_x |{ m : x in W[m] }|
closure = greedy_set_cover([set(W[m]) for m in A1])     # 1-closure probes

Cost on the real corpus: 106,081 mutants generated, 17,502 survivors, 7,727 alibis, roughly 4.5-5 million Python function calls, about 27 CPU-minutes, and about 4 minutes wall-clock on 8 CPython workers. This is a build-time content pipeline, not runtime work. Seeds are deterministic throughout: the problem id seeds the mutant target order and the probe shuffle, and the resampling and audit scripts use the fixed seeds 11, 5, 7, 99, 2026, 3, and 41.

Two scope facts must be stated before any guarantee is read. First, mining samples mutants: no statement below is about the full single-edit space, only about the sampled family M. Second, the probe bank is rebuilt whenever the shipped test set changes, so closure statements are relative to a fixed (D, M). An early draft claimed monotonicity across test sets; that claim is false and is withdrawn. The corrected statement holds only when the bank and the sample are held fixed.

For T subseteq T' with D and M fixed: Alibi_{D,M}(p') subseteq Alibi_{D,M}(p) and alpha_{D,M}(p') >= alpha_{D,M}(p)

The author engine analyzed 5,721 of the 5,730 problems: five have fewer than five reference-evaluable probes and four raised during mining (ds-038, ds-235, ds-236, ds-237). The second engine analyzed 5,725: it handled the four raising problems and skipped the same five for the same probe reason. Over its analyzable set the author engine sampled 106,081 mutants; 17,502 pass every shipped test, a 16.50% survival rate; 7,727 of those survivors diverge somewhere in the probe bank, so they are alibis, which is 7.28% of all sampled mutants and 44.15% of survivors. The second engine found 105,425 mutants, 17,471 passers (16.57%), and 7,704 alibis (7.31% of mutants, 44.10% of survivors).

At the problem level, 2,636 of 5,721 analyzable problems admit at least one radius-1 alibi, 46.08%; the second engine found 2,630 of 5,725, 45.94%. That agreement is the load-bearing cross-implementation check in the paper, and it is not the only one: the singleton-witness share is 22.78% (author) against 22.83% (second engine); the best-probe corpus kill is 73.22% against 73.25%; the fully-closed share is 59.71% against 59.77%. Row by row in Table 7, the five census-rate rows differ by at most 0.14 percentage points, the closure and derived rows by at most 0.44 (cross-validated held-out kill, 48.35% against 48.79%), and the four length-quartile aperture rows by at most 0.32 (56.66% against 56.34%). Two qualifications belong with that number. Both engines were written by this paper's author, in the same project, against the same corpus and the same seed discipline, so the check establishes that two implementations of one specification agree and not that the measurement has been reproduced by anyone else. And neither engine, its inputs, or its output is in this repository, so the agreement cannot be re-checked from a checkout by an outside reader. The remaining problems include 1,051 (18.37%) whose sampled survivors never diverged anywhere in the bank; the second engine counted 1,054 (18.41%). Those probe-equivalent survivors are reported as unresolved, not as alibis: their true status is unknown, and an alibi without a witness is a claim this method cannot make.

Family structure is the clearest qualitative result. The classic boundary and arithmetic slips dominate: cmp survives at 35.26% and int at 25.93%, while the destructive rewrites nearly always die: notins at 2.26% and notdel at 1.90%. Which is the most that can be said without human false-pass data: the ordering is consistent with a slip-relevance reading of the families, but a family-ordering argument cannot separate a plausible-slip effect from a rewrite that is simply mechanically destructive, and no human false-pass data exists here to separate them.

The aperture rises with both difficulty and length. Easy problems sit at 35.77%, Medium at 49.39%, and Hard at 57.48%. By line quartile the share runs 20.24%, 41.45%, 56.66%, and 69.95%, with per-mutant survival rising from 11.45% to 18.57% (1.62x), the same direction independently (11.50% to 18.63%). Test count, read marginally, is associated with aperture (35.35% at three tests, 43.63% at four, 57.62% at five), but that association is largely length: holding the solution to 5-8 lines, the shares are 36.73%, 42.98%, and 41.83%, and per-mutant survival is actually higher at five tests (15.76%) than at three (12.06%). More shipped tests is not a proxy for a tighter suite in this corpus.