Skip to content
DeepForgeby svx
Advancedoptimizationgradient-descentadamlearning-rate-schedules

Optimization Mastery

30 problems·~9h·4 stages

How models actually learn: gradients, momentum, adaptive methods, and the schedules that decide whether training converges. You will implement gradient descent, Adam, and learning rate policies, then reach the quasi-Newton methods used in serious solvers. Take it after ML From Scratch or Deep Learning Essentials.

Your progress

0/30 · 0%

Path readiness 0%Easy 0% · Medium 0% · Hard 0%

Continue path · Exponential LR Schedule

What you will learn

  • Tune training with warmup, decay, clipping, and weight-decay schedules
  • Implement SGD, momentum, Nesterov, Adam, AdamW, and Lion updates
  • Handle constraints and stochastic gradients with projected methods
  • Reach quasi-Newton solvers: BFGS, L-BFGS, Gauss-Newton, and trust regions

Before you start

1Schedules and Stability

0/7

The learning-rate policy decides whether training converges. Implement exponential, step, warmup, and polynomial schedules, plus value and norm gradient clipping.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepNoisy QuadraticMedium

2First-Order Optimizers

0/7

Follow the gradient: full-batch and mini-batch descent, momentum, Nesterov acceleration, Adam, AdamW, and the sign-based Lion update.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepNoisy QuadraticMedium

3Adaptive and Constrained Methods

0/9

Variants that adapt per parameter and methods that respect constraints: Adagrad, RMSProp, stochastic regression, AMSGrad, projected gradient, conjugate gradients, natural gradient, and Frank-Wolfe.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepNoisy QuadraticMedium

4Second-Order and Quasi-Newton

0/7

When first-order steps stall, use curvature: Hessian-vector products, finite-difference Newton steps, BFGS, L-BFGS, Gauss-Newton, Levenberg-Marquardt, and trust-region updates.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepNoisy QuadraticMedium

Capstone

Collection

Close the path with one real artifact from the catalog.

Optimizers Deep Dive

A focused set that revisits every update rule this path builds, from SGD to the adaptive methods.