Expected Calibration Error by Probability Bins

Problem

Compute the expected calibration error with equal-width bins. Assign p to bin min(int(p * num_bins), num_bins - 1), then return the sum over non-empty bins of (count/n) * |mean_true - mean_pred|.

Empty input returns 0.0.

Hint

Each bin contributes its share of samples times the gap between average confidence and accuracy.

Test cases

case 1

in: [[0,0,0,1,1,1],[0.1,0.2,0.3,0.7,0.8,0.9],3]

expected: 0.20000000000000007

case 2

in: [[1,1,0,0],[1,1,0,0],2]

expected: 0

case 3

in: [[0,1],[0.3,0.7],1]

expected: 0

case 4

in: [[],[],3]

expected: 0

Starter codePython

def expected_calibration_error(y_true, probs, num_bins):
    # Your code here
    pass

4 test cases · run your code against them in the editor