SAC Log-Alpha Update Step

Problem

Take one gradient step on the SAC temperature parameterized as log_alpha.

With alpha = exp(log_alpha) and mean_lp the mean action log-probability, the loss is J = -alpha * (mean_lp + target_entropy), so dJ/d(log_alpha) = -alpha * (mean_lp + target_entropy). Return exp(log_alpha - lr * dJ/dlog_alpha), clamped below at min_alpha. Empty log_probs returns exp(log_alpha).

Hint

The temperature rises when the policy entropy falls below the target entropy.

Test cases

case 1

in: [-1.6094379124341003,[-1,-2],-2,0.01,0.01]

expected: 0.19860488858664707

case 2

in: [0,[-4],-1,1,0.01]

expected: 0.01

case 3

in: [-0.6931471805599453,[],-1,0.1,0]

expected: 0.5

case 4

in: [-1.6094379124341003,[-0.5],-2,0.5,0]

expected: 0.155760156614281

Starter codePython

import math
def sac_log_alpha_step(log_alpha, log_probs, target_entropy, lr, min_alpha):
    # Your code here
    pass

4 test cases · run your code against them in the editor