Nonlinear Regression Chase
The data follows a parabola plus noise. A straight line leaves the curve on the table — add curvature and drive MSE down.
Your goal
Beat the linear regression baseline — score lower MSE than 0.4432 on the hidden 25 test rows. The first winning run records 25 points in this browser.
- Train
- 50 × 1
- Hidden test
- 25 rows
- Metric
- MSE · lower
- Baseline score
- 0.4432
Data
One feature x drawn uniformly from [-1, 1]. The label is 2x² - x + 0.5 plus independent uniform noise in [-0.2, 0.2]. 50 train rows and 25 hidden test rows from a single LCG stream (seed 7), two draws per row (x then noise).
x: feature 1 · y: label
Each dot is one training row: the single feature on x, its label on y.
Method & theory
read before you submitA straight line through a parabola is the classic underfitting picture. This challenge makes the gap between linear and quadratic explicit in MSE.
The generator behind the curve
The training split has 50 rows and the hidden split 25. Each row draws x uniformly from [-1, 1], then y = 2x^2 - x + 0.5 plus independent uniform noise in [-0.2, 0.2] (variance 0.4^2/12, about 0.0133). The curve bottoms out at x = 0.25 with y = 0.375 and climbs to 3.5 at x = -1, so a straight line can follow the average slope but never the bend.
Why the line leaves most of the error on the table
Least squares on a single linear term records MSE 0.4432 on the hidden rows. Since the noise floor is only about 0.0133, roughly 97% of that error is bias - the constant price of insisting the world is straight. That gap is the entire opportunity of the challenge.
Three unknowns, three normal equations
Fit y = c0 + c1*x + c2*x^2. The normal equations are symmetric: the 3x3 matrix holds n, sum(x), sum(x^2) in the first row, sum(x), sum(x^2), sum(x^3) in the second, and sum(x^2), sum(x^3), sum(x^4) in the third, with the right-hand side sum(y), sum(x*y), sum(x^2*y). Gaussian elimination on the 50 training rows recovers about (0.51, -1.01, 2.05), close to the true (0.5, -1, 2), and scores MSE 0.019 on the hidden rows.
Where to stop
The irreducible error is the label noise, about 0.0133, and a good quadratic lands just above it. Adding x^3 or x^4 gives the fit enough rope to chase individual noisy points: training MSE falls while hidden MSE rises, because 50 rows and noise standard deviation 0.115 cannot identify more than three coefficients reliably.
Pitfalls
- Building the x^2 column in the normal equations but predicting with c0 + c1*x + c2*x. The third coefficient then multiplies the raw feature, and the model is still a line.
- Assembling the design matrix in one column order and predicting in another (for example fitting [1, x^2, x] but evaluating [1, x, x^2]), which silently swaps the two slope coefficients.
- Ordering the normal equations by hand without keeping the matrix symmetric - the sum of x^3 must appear in both the (1,2) and (2,1) positions - which flips signs during elimination.
Takeaway. Feature engineering is model choice here: handing the model x^2 is the whole fix, and the normal equations turn it into a 3x3 linear solve.
Hint
Fit y = c0 + c1·x + c2·x² with the normal equations: build the 3×3 system and solve it with Gaussian elimination. Pure Python is enough — no libraries needed.