Self-Consistency Majority Gain

Problem

For each item, compare greedy accuracy (the first sample) with majority accuracy (the most frequent sample, ties broken by the lexicographically smallest label). Return majority accuracy minus greedy accuracy averaged over all items. Return 0.0 when samples is empty, and count empty sample lists as incorrect for both strategies.

Test cases

case 1

in: [[["a","a","b"],["b","b","a"]],["a","a"]]

expected: 0

case 2

in: [[["a","b","b"]],["b"]]

expected: 1

case 3

in: [[[],["a"]],["x","a"]]

expected: 0

case 4

in: [[],[]]

expected: 0

Starter codePython

def self_consistency_gain(samples, gold):
    # Your code here
    pass

4 test cases · run your code against them in the editor