Expected Calibration Error

Problem

Bin confidences into num_bins equal-width bins using int(p * num_bins) clipped to the final bin, then return the sample-weighted average of |mean_confidence - mean_accuracy| over non-empty bins. correct holds 0/1 correctness labels. Return 0.0 when there are no samples or num_bins <= 0.

Hint

Weight each bin gap by the fraction of samples that landed in it.

Test cases

case 1

in: [[0.9,0.8,0.3,0.1],[1,1,0,0],5]

expected: 0.17499999999999996

case 2

in: [[0.5,0.5],[1,0],2]

expected: 0

case 3

in: [[],[],5]

expected: 0

case 4

in: [[0.2,0.4,0.6,0.8],[0,0,1,1],4]

expected: 0.3

Starter codePython

def expected_calibration_error(confidences, correct, num_bins):
    # Your code here
    pass

4 test cases · run your code against them in the editor