Optimizers Deep Dive

SGD, momentum, AdaGrad, RMSProp, Adam, and AdamW, implemented update by update. Covers weight decay, bias correction, and the optimizer state memory costs that decide what fits on a GPU.

12 problems