Gradient TD One Step

Problem

Take one GTD(0) gradient-TD step with a secondary weights vector.

With delta = r + gamma*w.phi_next - w.phi_s and xv = v.phi_s, update w by alpha*delta*phi_s - alpha*gamma*xv*phi_next and v by beta*delta*phi_s. Return [new_w, new_v].

Hint

Gradient TD stays stable off-policy because it follows the true gradient of the projected error.

Test cases

case 1

in: [[1,0],[0.5,0.5],[1,0],[0,1],1,0.1,0.05,0.9]

expected: [[1,-0.045000000000000005],[0.5,0.5]]

case 2

in: [[0,0],[0,0],[1,1],[1,0],2,0.2,0.1,0.5]

expected: [[0.4,0.4],[0.2,0.2]]

case 3

in: [[0.5,0.5],[1,-1],[0,1],[1,0],0.5,0.05,0.2,0.95]

expected: [[0.5475,0.52375],[1,-0.905]]

Starter codePython

def gtd_one_step(weights, v, phi_s, phi_next, r, alpha, beta, gamma):
    # Your code here
    pass

3 test cases · run your code against them in the editor