CQL Penalty Lite

Problem

Compute the CQL conservative penalty for one state.

The penalty is the log-sum-exp of the Q values minus the Q value of the action in the dataset, pushing down out-of-distribution action values. Return the difference.

Hint

CQL lowers Q on actions the dataset never took.

Test cases

case 1

in: [[1,2,3],0]

expected: 2.4076059644443806

case 2

in: [[0,0],1]

expected: 0.6931471805599453

case 3

in: [[5,1,2],0]

expected: 0.06588390375742925

case 4

in: [[10,10,10],2]

expected: 1.0986122886681091

Starter codePython

import math
def cql_penalty_lite(q_values, data_action):
    # Your code here
    pass

4 test cases · run your code against them in the editor