Skip to content
DeepForgeby svx
Advanceddeep-learningtransformersquantizationtraining-systems

Deep Learning Advanced

27 problems·~14h·4 stages

Beyond the basics: modern activations, normalization variants, efficient attention, and quantization. You will implement GELU, group norm, multi-query attention, int8 quantization, and BPTT by hand, then compute the memory and FLOPs budgets of real models. Take it once you are comfortable with backprop.

Your progress

0/27 · 0%

Path readiness 0%Easy 0% · Medium 0% · Hard 0%

Continue path · SiLU (Swish)

What you will learn

  • Implement modern activations and normalization variants
  • Build efficient attention: multi-query, Bahdanau, and masked decoder attention
  • Quantize models and budget their compute with int8 and FLOP accounting
  • Work through VAE, DPO, and diffusion update steps

Before you start

1Modern Activations and Normalization

0/7

The non-linearities and normalizers in current architectures: SiLU, softplus, exact GELU, group norm, instance norm, layer scale, and per-channel quantization scales.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepXOR BoundaryHard

2Attention Variants

0/6

Attend efficiently: multi-query KV repeat, peephole LSTM, bidirectional RNN concatenation, additive Bahdanau attention, masked decoder attention, and positional-embedding interpolation.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepXOR BoundaryHard

3Compression and Training Systems

0/8

Ship models under budget: int8 scale, quantization and dequantization, checkpoint averaging, BPTT gradients, ZeRO memory accounting, transformer parameter counts, and inference FLOP estimates.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepXOR BoundaryHard

4Generative and Alignment Objectives

0/6

The losses behind modern generative systems: MAE patch masking, ViT class tokens, DETR set loss, the VAE ELBO, DPO loss, and a DDIM sampling step.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.
LabRecommended next stepXOR BoundaryHard

Capstone

Collection

Close the path with one real artifact from the catalog.

Quantization Basics

Take the quantization stage from scales and packing to serving-size memory savings.