Actor-Critic With Eligibility Traces

Problem

Apply one actor-critic update with accumulating eligibility traces.

Compute delta = r + gamma * V(phi_next) - V(phi_s), accumulate both traces as gamma*trace + phi_s, then update the critic by alpha*delta*critic_trace and the actor by beta*delta*actor_trace. Return [critic_w, actor_w, critic_trace, actor_trace].

Hint

Both the critic and the actor reuse the same TD error, scaled by their own traces.

Test cases

case 1

in: [[0,0],[0,0],[0,0],[0,0],[1,0],1,[0,1],0.1,0.2,0.9]

expected: [[0.1,0],[0.2,0],[1,0],[1,0]]

case 2

in: [[1,1],[0.5,0.5],[1,0],[0,1],[1,1],0,[1,0],0.2,0.1,0.5]

expected: [[0.5499999999999999,0.7],[0.35,0.27499999999999997],[1.5,1],[1,1.5]]

case 3

in: [[0.2,0.2],[1,-1],[0.1,0.1],[0.3,0],[0,1],1,[1,1],0.1,0.05,0.9]

expected: [[0.21044000000000002,0.32644000000000006],[1.01566,-0.942],[0.09000000000000001,1.09],[0.27,1]]

Starter codePython

def actor_critic_trace_update(critic_w, actor_w, critic_trace, actor_trace, phi_s, r, phi_next, alpha, beta, gamma):
    # Your code here
    pass

3 test cases · run your code against them in the editor