Skip to content
DeepForgeby svx
Advancedinformation-theoryentropykl-divergencesource-coding

Information Theory

29 problems·~11h·4 stages

How much does a message tell you? Build entropy and mutual information from scratch, compress with real source codes, and finish with KL divergence, channel capacity, and the limits of processing. You will implement Huffman and Shannon codes, Kraft's inequality, arithmetic coding, and the data-processing gap. Perfect for the mathematically curious.

Your progress

0/29 · 0%

Path readiness 0%Easy 0% · Medium 0% · Hard 0%

Continue path · Self-Information

What you will learn

  • Quantify information with entropy, cross-entropy, and perplexity
  • Compute mutual information, KL, Jensen-Shannon, and Hellinger divergences
  • Design source codes with Huffman, Shannon, and Kraft's inequality
  • Understand channel capacity and data-processing bounds

Before you start

1Information and Entropy

0/8

Surprise and uncertainty: self-information, entropy of distributions, binary entropy, cross-entropy, total variation, perplexity, entropy in nats, and BSC capacity.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.

2Joint and Mutual Information

0/7

Relationships between variables: entropy from counts, conditional entropy, the chain rule, mutual information from joints and marginals, KL divergence, and Jensen-Shannon divergence.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.

3Source Coding

0/8

Compress messages down to their entropy: Huffman code lengths and average length, Shannon code lengths, expected code length, Kraft's inequality, Markov-source rate, LZ78 parsing, and arithmetic coding.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.

4Divergences and Fundamental Limits

0/6

Distances between distributions and the limits of processing: Gaussian KL, Hellinger distance, BSC mutual information, maximum entropy, Blahut-Arimoto capacity, and the data-processing gap.

Stage checkpoint

0/2 to pass

A checkpoint is a short retrieval test drawn from this stage: solve 2 of 3 from memory to mark the stage complete. The stage stays open either way.

Tried this set and did not pass? Record it — the next set swaps in fresh problems from the same stage.

Capstone

Pen and paper

Close the path with one real artifact from the catalog.

Entropy of a Bernoulli(0.25)

Compute a Shannon entropy by hand: the move every bound in this path rests on.