Calibration Bins

Problem

Bin probability predictions and report calibration statistics.

The bin index is min(int(p * num_bins), num_bins - 1); for each bin return [count, mean predicted probability, mean true label], using [0, 0.0, 0.0] for empty bins. Empty input returns num_bins empty bins.

Hint

Accumulate count, probability sum, and label sum per bin, then divide by the count.

Test cases

case 1

in: [[0,0,1,1],[0.05,0.15,0.85,0.95],2]

expected: [[2,0.1,0],[2,0.8999999999999999,1]]

case 2

in: [[1,1,0],[0.1,0.9,0.5],3]

expected: [[1,0.1,1],[1,0.5,0],[1,0.9,1]]

case 3

in: [[1,0],[0.2,0.8],5]

expected: [[0,0,0],[1,0.2,1],[0,0,0],[0,0,0],[1,0.8,0]]

case 4

in: [[],[],3]

expected: [[0,0,0],[0,0,0],[0,0,0]]

Starter codePython

def calibration_bins(y_true, probs, num_bins):
    # Your code here
    pass

4 test cases · run your code against them in the editor