CQL Conservative Penalty

Problem

Compute the batch CQL conservative penalty.

For each state, penalize log-sum-exp of the Q values minus the Q value of the dataset action, then scale by alpha and average over the batch. An empty batch returns 0.0, and log-sum-exp must be computed with the max-subtraction trick.

Hint

Pushing down unseen action values is what keeps offline Q-learning conservative.

Test cases

case 1

in: [[[1,2,3],[0,0]],[0,1],1]

expected: 1.550376572502163

case 2

in: [[[5,1,2]],[0],0.5]

expected: 0.032941951878714626

case 3

in: [[],[],1]

expected: 0

Starter codePython

import math
def cql_conservative_penalty(q_batch, data_actions, alpha):
    # Your code here
    pass

3 test cases · run your code against them in the editor