True Online TD(Lambda) Lite

Problem

Take one true online TD(lambda) step with Dutch traces.

After computing delta = r + gamma * w.phi_next - w.phi_s, decay the trace to z = gamma*lam*trace, compute scale = 1 - alpha*gamma*lam*(z.phi_s), set z = z + scale*phi_s, and return [w + alpha*delta*z, z].

Hint

The scale factor corrects the trace so the algorithm is exactly online-equivalent to TD(lambda).

Test cases

case 1

in: [[0,0],[1,0],[0,1],1,[0,0],0.1,0.9,0.5]

expected: [[0.1,0],[1,0]]

case 2

in: [[1,1],[1,1],[1,0],0,[0.5,0.5],0.2,0.5,0.1]

expected: [[0.69265,0.69265],[1.0245,1.0245]]

case 3

in: [[0.5,-0.5],[1,1],[0.5,0.5],1,[0.2,0.2],0.1,0.9,0.9]

expected: [[0.6135756,-0.3864244],[1.135756,1.135756]]

Starter codePython

def true_online_td_lambda(weights, phi_s, phi_next, r, trace, alpha, gamma, lam):
    # Your code here
    pass

3 test cases · run your code against them in the editor