Reliability Slope

Problem

Return the reliability slope: the OLS slope of the true labels regressed on the predicted probabilities, equal to cov(y, p) / var(p) with population statistics. A slope above 1 means predictions are under-confident and below 1 means over-confident. Return 0.0 when the input is empty or the probabilities are constant.

Hint

Reliability slope rescales predicted probabilities so that a perfectly calibrated model has slope 1.

Test cases

case 1

in: [[0,0,1,1],[0.1,0.2,0.8,0.9]]

expected: 1.4

case 2

in: [[0,1],[0,1]]

expected: 1

case 3

in: [[1,1],[0.5,0.5]]

expected: 0

case 4

in: [[],[]]

expected: 0

Starter codePython

def reliability_slope(y_true, probs):
    # Your code here
    pass

4 test cases · run your code against them in the editor