Information Theory problems
Entropy, KL divergence, mutual information. How much does a message tell you?
360 problems130 Easy159 Medium71 Hard
50 problems
Self-Information
info-001
Entropy of a Distribution
info-002
Entropy from Counts
info-003
Binary Entropy Function
info-004
Joint Entropy of a Distribution
info-005
Conditional Entropy H(X|Y)
info-006
Chain Rule Entropy Check
info-007
Mutual Information from Joint
info-008
Mutual Information from Marginals
info-009
Pointwise Mutual Information
info-010
Variation of Information
info-011
Cross-Entropy Between Distributions
info-012
KL Divergence (Discrete)
info-013
KL Divergence Asymmetry Check
info-014
Jensen-Shannon Divergence
info-015
KL Divergence of Gaussians
info-016
Total Variation Distance
info-017
Hellinger Distance
info-018
Chi-Square Distance
info-019
Perplexity from Entropy
info-020
Perplexity from Cross-Entropy
info-021
Perplexity from Log-Likelihood
info-022
Uniform Distribution Entropy
info-023
Maximum Entropy with Mean Constraint
info-024
Entropy of a Geometric Distribution
info-025
Bernoulli Entropy in Nats
info-026
Huffman Code Lengths
info-027
Huffman Average Code Length
info-028
Shannon Code Length
info-029
Expected Code Length
info-030
Kraft Inequality Check
info-031
Rate of a Markov Source
info-032
Binary Symmetric Channel Capacity
info-033
BSC Mutual Information
info-034
Binary Erasure Channel Capacity
info-035
Channel Capacity via Blahut-Arimoto
info-036
Data Processing Inequality Gap
info-037
Fano's Inequality Lower Bound
info-038
Mutual Information of Independent Variables
info-039
Conditional Mutual Information
info-040
Interaction Information
info-041
Entropy Rate of a Markov Chain
info-042
KL of Markov Stationary to Uniform
info-043
Cross-Entropy of a Language Model
info-044
Lempel-Ziv 78 Phrase Count
info-045
Huffman Tree Total Cost
info-046
Canonical Code Words
info-047
Shannon-Fano Split
info-048
Arithmetic Coding Interval Update
info-049
Final Interval Width
info-050
Showing 50 of 360 problems.