Verifying 5,050 problems with real Python

DeepForge is a practice platform: you get a function signature and write the implementation from scratch, in the browser, against real Python. That means the product is only as good as its problem bank, and the bank is large — 5,050 problems, 21,165 test cases, 15 categories. Every single solution is asserted by execution before it ships.

1.Structural checks are cheap, and not enough

The first gate is shape. scripts/verify-problems.ts walks the TypeScript dataset and checks that every problem has an id matching ^[a-z]+-\d{3,4}$ with a known category prefix, that ids are unique, that the category and difficulty are valid, that there are three to five test cases, that inputs are arrays and expected values are present, and that the function in starterCode has the same name as the function in solution.

This pass runs in milliseconds over the entire dataset and catches the boring failures: a copy-pasted id, a renamed helper, a missing expected. What it cannot do is run the code. A solution can satisfy every structural rule and still be wrong — an off-by-one in a sliding window, a transposed matrix, a return type that is a list of lists where a list of tuples was expected. Structure is necessary; execution is the proof.

2.The contract: one function, 3–5 cases, JSON in

The data model is deliberately boring. A problem is one function, a handful of cases, and metadata — nothing the verifier has to interpret.

interface Problem {
  id: string;                    // la-001, al-015, dl-321
  title: string;
  category: Category;            // one of 15
  difficulty: "Easy" | "Medium" | "Hard";
  description: string;
  starterCode: string;           // the function the learner fills in
  solution: string;              // our reference, pure stdlib Python
  testCases: TestCase[];         // 3-5: { input: any[], expected: any }
  hint?: string;
}

Test cases are serialized to JSON and read back with Python's json module. Numbers stay numbers, lists stay lists, booleans stay booleans, and nothing is evaluated. There is no eval, no pickle, no fixture format to document. The input array is splatted positionally; the expected value is compared. That is the entire protocol.

3.The Python harness

scripts/py_verify.py receives JSON and, for each problem, compiles and executes the solution in a fresh namespace, extracts the first def, calls it once per test case, and records reprs for anything that failed. The TypeScript side prints the first 60 failures with their inputs, then the per-category counts, then a one-line summary. A run either ends in ALL GREEN or with the exact case that broke.

The interesting part is the equality semantics, because they are shared: the same function body lives in the browser harness in src/lib/pyodide.ts. If CI grades a solution with 1e-6 tolerance, the browser does too — there is no second opinion.

def _deep_eq(a, b, tol=1e-6):
    if isinstance(a, bool) or isinstance(b, bool):
        return a == b
    if isinstance(a, (int, float)) and isinstance(b, (int, float)):
        try:
            return math.isclose(float(a), float(b), rel_tol=tol, abs_tol=tol)
        except (ValueError, OverflowError):
            return float(a) == float(b)
    if isinstance(a, (list, tuple)) and isinstance(b, (list, tuple)):
        return len(a) == len(b) and all(_deep_eq(x, y, tol) for x, y in zip(a, b))
    if isinstance(a, dict) and isinstance(b, dict):
        return a.keys() == b.keys() and all(_deep_eq(a[k], b[k], tol) for k in a)
    if a is None or b is None:
        return a is b
    return a == b

Three choices matter. Booleans compare exactly and before numbers, so True is not 1. Numbers get both relative and absolute tolerance. Sequences compare itemwise and ignore list-versus-tuple — a learner returning a list where the reference returned a tuple is not the bug we are hunting. Everything is recursive, so nested structures get the same treatment at every depth.

4.Seeded randomness is a problem contract

Randomness is the classic way to make a verifier flaky, so the dataset treats it as part of the problem statement. 104 of the 5,050 solutions import random, and every one of them takes an explicit seed argument: 67 call random.seed(seed) first, and 37 use a local random.Random(seed) instance. The verifier itself seeds nothing.

Determinism is the problem's responsibility, not the runner's convenience. A bootstrap percentile interval, a Monte Carlo estimate, a Gibbs sweep, or inverted dropout is still a pure function of its inputs when the seed is an input. The same reasoning gives the browser the same numbers: a learner who re-runs the tests sees identical results, and so does CI.

5.Failure modes the suite catches

In practice the harness finds three families of bugs. Structural failures — unknown id prefixes, duplicate ids, a starter function named differently from the solution, fewer than three cases — are caught before Python even starts. Runtime failures surface on edge cases: empty lists, single elements, division by zero, a dict lookup that only exists for the happy path. Equality failures show up as floating-point drift outside 1e-6 or as a container mismatch.

Dataset invariants Python cannot see live in tests/data.test.ts: unique titles across all 15 categories, counts per difficulty, category totals. More than one batch landed with duplicate titles (softmax jacobian, variance reduction, effective sample size) and was cleaned up in a follow-up pass.

6.What it still cannot catch

The uncomfortable truth is that the verifier proves agreement, not correctness. If an expected value was authored with the same misunderstanding as the solution, the two agree and the suite is green with a wrong problem in it. Test cases are the spec, and the spec is written by the same batch that writes the code. For numeric problems the 1e-6 window is a useful lie detector; for algorithmic problems the cases themselves are the only definition of right.

Coverage is the second limit. Three to five fixed cases cannot prove a function over its input space; boundary discipline — empty inputs, single elements, negative numbers, duplicate keys — is manual work. And nothing statically bans an unseeded random call; it would surface as an intermittent failure eventually. We rely on the seed convention and review, which is cheaper than an AST rule across 5,050 snippets.

7.Running it

The whole bank verifies in well under a second on a laptop, which is the strategic point: the check is cheap enough that it is never optional.^[1] It runs alongside the TypeScript compiler, ESLint, and 252 unit tests, and a red run prints the exact case, input, actual, and expected for the first 60 failures.

$ bun run scripts/verify-problems.ts
Loaded 5050 problem(s).
...
Structural errors: 0   Runtime failures: 0   Passed: 5050/5050
ALL GREEN

If you want to see the bank without running anything, browse the problems — every one of them is the artifact this pipeline produces.