SAC Soft Twin-Q Target

Problem

Compute the soft Bellman target used by SAC with twin critics.

If done, return reward. Otherwise return reward + gamma * (min(q1, q2) - alpha * log_prob), where the subtracted entropy term rewards next actions with low probability. gamma, alpha, and the probabilities follow the usual SAC conventions.

Hint

The entropy term equals alpha times the negative log-probability of the next action.

Test cases

case 1

in: [1,false,0.99,2,3,-0.5,0.2]

expected: 3.079

case 2

in: [1,true,0.99,2,3,-0.5,0.2]

expected: 1

case 3

in: [0,false,0.5,-1,2,-1,1]

expected: 0

case 4

in: [2,false,0.9,4,4,0,0.5]

expected: 5.6

Starter codePython

def sac_soft_twin_q_target(reward, done, gamma, q1, q2, log_prob, alpha):
    # Your code here
    pass

4 test cases · run your code against them in the editor