PPO Advantage Estimate

Problem

Compute PPO value targets by adding GAE advantages back to the value estimates.

Build the GAE vector backwards with delta = r + gamma*V_{t+1} - V_t and A_t = delta + gamma*lam*A_{t+1}, then return A_t + V_t for every step. values has one more entry than rewards.

Hint

The critic regresses toward advantage plus value, which is a variance-reduced return.

Test cases

case 1

in: [[1,1],[0,2,3],0.9,0.5]

expected: [3.565,3.7]

case 2

in: [[0,1,2],[0,0,0,0],0.9,0.9]

expected: [2.1222000000000003,2.62,2]

case 3

in: [[2],[1,4],0.5,1]

expected: [4]

Starter codePython

def ppo_value_targets(rewards, values, gamma, lam):
    # Your code here
    pass

3 test cases · run your code against them in the editor