N-Step Advantage Value

Problem

Compute an n-step bootstrapped return minus a state-value baseline.

Accumulate sum gamma^t * rewards[t], add gamma^n * next_value when done is False, then subtract value0 to get the n-step advantage. An empty reward list uses gamma^0 = 1 for the bootstrap.

Hint

Subtracting the baseline turns the n-step return into a variance-reduced advantage.

Test cases

case 1

in: [[1,1],0.9,5,2,false]

expected: 3.950000000000001

case 2

in: [[1,1],0.9,100,2,true]

expected: -0.10000000000000009

case 3

in: [[],0.9,3,1,false]

expected: 2

case 4

in: [[2],0.5,4,0,false]

expected: 4

Starter codePython

def n_step_advantage_value(rewards, gamma, next_value, value0, done):
    # Your code here
    pass

4 test cases · run your code against them in the editor