Pairwise Reward Accuracy

Problem

Measure the pairwise accuracy of a reward model.

Score 1.0 when r_pref > r_rej, 0.5 on a tie, and 0.0 otherwise, then return the mean over pairs. An empty batch returns 0.0.

Hint

Ties count as half a win because the model failed to separate the pair.

case 1

in: [[2,1],[1,2]]

expected: 0.5

case 2

in: [[1,1],[0,1]]

expected: 0.75

case 3

in: [[3],[3]]

expected: 0.5

case 4

in: [[],[]]

expected: 0

Starter codePython

def pairwise_accuracy(r_pref, r_rej):
    # Your code here
    pass

4 test cases · run your code against them in the editor