Semi-Gradient TD Update

Problem

Apply a semi-gradient TD(0) update for linear function approximation.

Compute v_s = w . phi_s, v_next = w . phi_next, and delta = r + gamma * v_next - v_s. The update uses only phi_s because the bootstrap target is treated as constant: return w + alpha * delta * phi_s.

Hint

Ignoring the gradient of the bootstrap target is what makes it semi-gradient.

Test cases

case 1

in: [[1,2],[1,0],[0,1],1,0.1,0.9]

expected: [1.18,2]

case 2

in: [[0,0],[1,1],[1,1],2,0.5,1]

expected: [1,1]

case 3

in: [[0.5,-1],[2,0],[0,2],0,0.25,0.5]

expected: [-0.5,-1]

Starter codePython

def semi_gradient_td_update(weights, phi_s, phi_next, r, alpha, gamma):
    # Your code here
    pass

3 test cases · run your code against them in the editor