TD(Lambda) Forward View

Problem

Compute the TD(lambda) forward-view return from time 0 using the backward recursion.

values[i] is the value of the state reached after i rewards, so values[-1] bootstraps the end. Starting from g = values[-1], iterate backwards: g = reward[t] + gamma * ((1 - lam) * values[t + 1] + lam * g). If the episode is empty return values[0].

Hint

This backward recursion gives the same weighted average of n-step returns as the forward view.

Test cases

case 1

in: [[1,1],[0,2,3],0.9,0.5]

expected: 3.565

case 2

in: [[1,1],[0,2,3],0.9,1]

expected: 4.33

case 3

in: [[1,1],[0,2,3],0.9,0]

expected: 2.8

case 4

in: [[2],[1,4],0.5,0.8]

expected: 4

case 5

in: [[],[5],0.9,0.5]

expected: 5

Starter codePython

def td_lambda_forward(rewards, values, gamma, lam):
    # Your code here
    pass

5 test cases · run your code against them in the editor