Expected Calibration Error for Confidence Scores

Problem

Compute expected calibration error with equal-width bins. Bin index is int(conf * num_bins) clamped to the last bin, each bin contributes (count / N) * |avg_confidence - avg_accuracy|.

The signature is expected_calibration_error(confidences, correct, num_bins=10). confidences are floats in [0, 1] and correct holds 0/1 values.

Hint

Within each bin compare the average confidence with the observed accuracy.

Test cases

case 1

in: [[0.9,0.8,0.6],[1,0,1],5]

expected: 0.3666666667

case 2

in: [[0.5,0.5],[1,0],10]

expected: 0

case 3

in: [[0.1,0.9],[0,1],10]

expected: 0.1

case 4

in: [[1,1],[1,0],2]

expected: 0.5

Starter codePython

def expected_calibration_error(confidences, correct, num_bins=10):
    # Your code here
    pass

4 test cases · run your code against them in the editor