Q(Lambda) Accumulate

Problem

Apply a backward-view Q(lambda) update with accumulating traces.

Increment trace[s][a] by 1, update every Q[i][j] by alpha * delta * trace[i][j], then decay the trace by gamma * lam. Return [new_Q, new_trace] as deep copies with the decay applied.

Hint

The returned trace is already decayed, so it is ready for the next update.

Test cases

case 1

in: [[[0,0],[0,0]],[[0,0],[0,0]],0,0,1,0.1,0.9,0.5]

expected: [[[0.1,0],[0,0]],[[0.45,0],[0,0]]]

case 2

in: [[[1,2],[3,4]],[[0.5,0],[0,0.25]],1,1,2,0.5,0.5,0.8]

expected: [[[1.5,2],[3,5.25]],[[0.2,0],[0,0.5]]]

case 3

in: [[[0]],[[0]],0,0,-1,0.2,1,0]

expected: [[[-0.2]],[[0]]]

Starter codePython

def q_lambda_accumulate(Q, trace, s, a, delta, alpha, gamma, lam):
    # Your code here
    pass

3 test cases · run your code against them in the editor