Calibration Gap by Equal-Frequency Bins

Problem

Compute the worst calibration gap over equal-frequency probability bins. Sort samples by probability, split them into num_bins contiguous bins (earlier bins take the remainder first), and return the maximum absolute difference between the mean label and mean probability inside a bin. Empty input or num_bins <= 0 returns 0.0.

Hint

Equal-frequency bins each hold the same number of samples, unlike equal-width probability bins.

Test cases

case 1

in: [[0,0,1,1],[0.1,0.2,0.8,0.9],2]

expected: 0.15000000000000002

case 2

in: [[1,0,1,0],[0.9,0.8,0.7,0.6],2]

expected: 0.3500000000000001

case 3

in: [[1,0],[0.5,0.5],3]

expected: 0.5

case 4

in: [[],[],3]

expected: 0

case 5

in: [[1],[0.5],0]

expected: 0

Starter codePython

def calibration_gap_equal_freq(y_true, probs, num_bins):
    # Your code here
    pass

5 test cases · run your code against them in the editor