CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction
Ask a model to predict what a function outputs (or to invent an input that produces a given output) in plain language, and code turns into a scalable supply of verified reasoning practice.
arXiv:2502.07316Where this sits
A research satellite rather than a step on the main lineage: it uses DeepSeek-V2.5 as a synthesizer and targets the same bottleneck R1 attacks from the RL side, namely where diverse reasoning training data comes from. There is no honest single predecessor to point at, so this entry has no 'from' edge.
- Turns arbitrary executable code into a reasoning corpus by prediction rather than by code generation, decoupling reasoning structure from syntax.
- Scales to 3.5M training instances derived from 454.9K raw code files, with verified ground truth for every input-output pair.
- Introduces CodeI/O++, which verifies predictions by re-execution and asks the model to revise its own wrong answers, keeping the data instead of filtering it.
- Shows balanced gains across 14 benchmarks for four different base models, where math-only or code-only datasets tend to help some tasks and hurt others.
What it is
CodeI/O is a data-construction method, not a new architecture. It harvests hundreds of thousands of real functions, converts each into an executable form, generates input-output pairs by running the code, and has a strong model (DeepSeek-V2.5) write natural-language chains of thought that predict the answer. Because the ground truth comes from execution, every prediction can be checked, and wrong predictions can be revised instead of discarded. Training on the result improves symbolic, logic, math, scientific, and commonsense benchmarks, not just code tasks.
1Theory from first principles
Reasoning data is scarce because it is expensive
Math and code have abundant structured supervision: final answers and unit tests. Most other reasoning domains (logic, state tracking, commonsense chains) have nothing comparable, and manually writing step-by-step solutions does not scale. Code is an interesting middle ground: it is abundant and executable, but the reasoning inside it is implicit and entangled with syntax. Reading code does not automatically teach you to reason; predicting what code does does.
Input-output prediction as reasoning condensation
The transformed task is simple to state. Show the model a function and a textual query, then either give an input and ask for the output, or give an output and ask for a feasible input. The answer must be a natural-language chain of thought, not code: trace the flow, track variables, follow branches and loops. That is procedural reasoning with no syntax to hide behind. Because the reference function was executed to make the pair, the prediction has a ground truth, which is rare in open-ended reasoning data.
Keep the mistakes: verification and revision
A tempting move is rejection sampling: drop every response whose prediction was wrong. The paper tested this and it backfired. Filtering removes about half the data, and the surviving set scored worse on average than a random subset of the same size (56.5 versus 56.7 on the reported average). Instead, CodeI/O++ appends execution feedback as a second turn and asks the model to try again; the revised response is concatenated with the original and the feedback. About 10% of incorrect first-turn predictions are fixed this way, and the resulting dataset beats CodeI/O on every reported model. The lesson is that error correction is more informative than error deletion.
def check_prediction(fn, mode, given, predicted):
"""mode is 'output' (given an input, predicted the output)
or 'input' (given an output, predicted a candidate input)."""
if mode == "output":
return fn(given) == predicted
if mode == "input":
try:
return fn(predicted) == given
except Exception:
return False
raise ValueError("unknown mode")
# doubles = lambda x: x * 2
# check_prediction(doubles, "output", 21, 42) -> True
# check_prediction(doubles, "input", 42, 21) -> True- Output prediction is easy to verify by calling the function once.
- Input prediction verifies by executing the candidate input inside a try, because a wrong input can crash the function.
- This same check is what powers the second-turn revision prompt.
Why the schedule is two-stage
The CodeI/O dataset has far more samples than the general instruction-tuning set. Mixing them in one stage would let prediction samples swamp instruction data, so training runs as code-prediction first (making the base model a stronger reasoner) and general instruction tuning second (making it a usable assistant). The paper compares alternatives and reports that two-stage training beats mixing in every configuration tested, and that first-stage data helps even when the second stage is identical to the single-stage baseline.
What the results show
Averaged over 14 benchmarks, Qwen2.5-Coder-7B improves from 54.8 (instruction tuning only) to 57.2 with CodeI/O and 57.7 with CodeI/O++. CRUXEval-O, the benchmark that literally asks for output prediction, rises from 60.0 to 64.9 and then 67.9. LLaMA 3.1 8B goes from 49.3 to 51.2 and 52.1, DeepSeek Coder v2 Lite from 51.6 to 53.6, and Gemma 2 27B from 59.5 to 60.9 and 61.5. Gains also appear on MATH, BBH, and KorBench. Continual pretraining on raw code (7.7M samples, more data than CodeI/O) barely helps, which the authors read as evidence that task design, not data volume, is doing the work.
The honest limits
The dataset is built with a strong proprietary-quality synthesizer (DeepSeek-V2.5), and the paper shows synthesis-model quality matters. Leakage is handled with an n-gram audit; overlap is low on most benchmarks, with LeetCode-O at 21.5% and KorBench at 5.1%, and the paper shows gains persist on the non-leaked subsets. Input prediction is only verifiable when the function accepts many valid inputs; deterministic, JSON-serializable outputs are required, so randomness and side effects are excluded. And the benchmark suite is the authors' selection, not an external standard.
2Inside the paper
Problem the paper sets out to solve
Reasoning training data outside math and code is sparse and fragmented. Raw code is abundant but its reasoning signal is implicit, and training on text-to-code generation is constrained by syntax.
Key idea to look for while reading
Reformulate code learning as input-output prediction in natural language. Section 2 covers the four-stage data pipeline, Section 3 the main results against strong data baselines, and Section 4 the ablations that justify each design choice, especially the rejection-sampling result.
Evidence the paper musters
3.5M instances from 454.9K raw code files, split roughly 50/50 between input and output prediction. CodeI/O and CodeI/O++ lead on average across four base models and 14 benchmarks, beating OpenMathInstruct2, WebInstruct, OpenCoder-SFT-Stage-1, and raw code. The Qwen2.5-Coder-7B average moves 54.8 to 57.2 and 57.7; CRUXEval-O moves 60.0 to 64.9 and 67.9; MATH moves 71.6 to 71.9 and 72.1. A DeepSeek-V2.5 re-synthesis of the WebInstruct baseline still trails CodeI/O, which isolates task design from synthesizer strength.
Limits the authors admit
LeetCode-O overlap is measured at 21.5% before manual inspection, and the authors rely on subset analysis to argue the gains are real. The domain is restricted to executable, deterministic Python with JSON-serializable inputs and outputs. The instruction-tuning stage uses an in-house dataset, so exact reproduction is impossible from the paper alone.
Practitioner takeaway
An execution engine is a data generator. When you have code, turn it into prediction problems, let execution grade the answers, and revise the wrong ones rather than deleting them. Train this before instruction tuning, expect broad rather than code-specific gains, and treat the synthesizer model as a first-class hyperparameter.
3Implementation check
0 of 5 questions answered
1.Why does CodeI/O ask for input prediction in addition to output prediction?
2.The paper tried rejection sampling (dropping wrong predictions) and abandoned it. Why?
3.What does execution-based verification buy over an LLM judge?
4.Why does the paper train in two stages instead of mixing CodeI/O with instruction data?
5.Raw-code continual pretraining (7.7M samples) underperforms CodeI/O (3.5M samples). What does that suggest?
4Build it yourself
Condensed Features Beat Surface Matching
Starter2-3 hoursTrain nearest-neighbor predictors on the same (input, output) pairs and change only the distance: raw tuples against a multiset representation of the computation. The condensed view generalizes to held-out inputs while surface matching cannot.
Milestones
- Generate all 256 length-4 tuples of digits 1-4; the target output is the sorted tuple.
- Keep 150 inputs as (input, sorted output) training pairs and hold out the other 106.
- Implement surface_distance as position mismatches and multiset_distance as the number of elements that must move.
- Implement knn_predict with a majority vote over the k nearest training outputs.
- Compare accuracies and read the mispredictions to see why surface matching fails.
- Vary k and the split and record where each representation saturates.
Success criteria
- Multiset kNN accuracy is above 0.9 while surface kNN stays below 0.1 on the same split.
- Every test input whose multiset appears in training is predicted exactly.
- Shuffling the training outputs drops both accuracies toward chance, confirming the pairing matters.
Stretch goals
- Use distance-weighted voting instead of a plain majority and report the change.
- Swap sorting for a different transformation such as reverse or dedupe and design a representation that generalizes.
- Add revision: on a wrong prediction, add the corrected pair to training and re-measure.