Imagined Actor Gradient Step

Problem

Apply one actor update averaged over a batch of imagined states.

For each parameter take the mean over imagined steps of advantage_t * policy_grads[t][i], scale by lr, and add it to params[i]. An empty advantage list returns the parameters unchanged.

Hint

In Dreamer the actor learns entirely from imagined trajectories, not real transitions.

Test cases

case 1

in: [[1],[[2],[4]],[1,1],0.5]

expected: [2.5]

case 2

in: [[0,1],[[1,0],[0,-1]],[2,-1],0.1]

expected: [0.1,1.05]

case 3

in: [[1],[],[],1]

expected: [1]

case 4

in: [[2,3],[[1,2]],[0],5]

expected: [2,3]

Starter codePython

def imagined_actor_step(params, policy_grads, advantages, lr):
    # Your code here
    pass

4 test cases · run your code against them in the editor