CQL Hinge Penalty

Problem

Compute the per-state CQL hinge penalty against out-of-distribution actions.

For each state take the maximum Q over the other actions and add max(0, max_other_q - q_data + margin), then average over states. An empty batch returns 0.0.

Hint

The margin forces dataset actions to stay ahead of the best unseen action.

Test cases

case 1

in: [[1,2],[[0,3],[1,1.5]],0.5]

expected: 1.25

case 2

in: [[5],[[1,2]],1]

expected: 0

case 3

in: [[],[],1]

expected: 0

case 4

in: [[0,0],[[1],[2]],0.25]

expected: 1.75

Starter codePython

def cql_hinge_penalty(q_data, q_other, margin):
    # Your code here
    pass

4 test cases · run your code against them in the editor