Annotator Agreement Kappa

Problem

Compute Cohen's kappa between two annotators.

po is the fraction of matching labels and pe = sum over labels of (count_a / n) * (count_b / n); return (po - pe) / (1 - pe). Return 0.0 when the batch is empty or chance agreement pe equals 1.

Hint

Kappa corrects raw agreement for how often the annotators would agree by chance.

Test cases

case 1

in: [[0,1,1,0],[0,1,0,0]]

expected: 0.5

case 2

in: [[0,0,1,1],[0,0,1,1]]

expected: 1

case 3

in: [[0,1,0,1],[0,0,0,0]]

expected: 0

case 4

in: [[],[]]

expected: 0

Starter codePython

def cohen_kappa(labels_a, labels_b):
    # Your code here
    pass

4 test cases · run your code against them in the editor