SARSA(Lambda) Backward

Problem

Apply one backward-view SARSA(lambda) update with accumulating traces.

First compute delta = r + gamma * Q[s_next][a_next] - Q[s][a]. Then increment trace[s][a], update Q by alpha * delta * trace, and decay the trace by gamma * lam. Return [new_Q, new_trace] as deep copies with the decay applied.

Hint

Unlike Q(lambda), the TD error uses the action actually taken next.

Test cases

case 1

in: [[[0,0],[0,0]],[[0,0],[0,0]],0,0,1,1,1,0.1,0.9,0.5]

expected: [[[0.1,0],[0,0]],[[0.45,0],[0,0]]]

case 2

in: [[[1,2],[3,4]],[[0,1],[0,0]],0,1,0,0,0,0.5,0.5,0.5]

expected: [[[1,0.5],[3,4]],[[0,0.5],[0,0]]]

case 3

in: [[[0]],[[0]],0,0,-2,0,0,0.25,0.9,0]

expected: [[[-0.5]],[[0]]]

Starter codePython

def sarsa_lambda_backward(Q, trace, s, a, r, s_next, a_next, alpha, gamma, lam):
    # Your code here
    pass

3 test cases · run your code against them in the editor