Truncated GAE Lambda Value

Problem

Compute generalized advantage estimates from a list of TD residuals with a truncation horizon.

A[t] = sum over l from 0 to horizon - 1 of (gamma * lam)^l * deltas[t + l], stopping at the end of the episode. A non-positive horizon yields all zeros, and an empty list returns [].

Hint

Truncating the GAE sum bounds variance at the cost of some bias.

Test cases

case 1

in: [[1,1,1],0.9,0.5,2]

expected: [1.45,1.45,1]

case 2

in: [[1,1,1],0.9,0.5,3]

expected: [1.6524999999999999,1.45,1]

case 3

in: [[1,1,1],0.9,1,3]

expected: [2.71,1.9,1]

case 4

in: [[1,2],0.5,0.5,0]

expected: [0,0]

case 5

in: [[],0.9,0.5,2]

expected: []

Starter codePython

def truncated_gae(deltas, gamma, lam, horizon):
    # Your code here
    pass

5 test cases · run your code against them in the editor