PPO Value Clip

Problem

Compute the clipped PPO value-function loss over a batch.

For each sample clip the new value to [old - clip_eps, old + clip_eps] and take the max of the clipped and unclipped squared errors against the return. Return the mean; an empty batch gives 0.0.

Hint

The pessimistic max keeps the value from moving too far in one update.

Test cases

case 1

in: [[2],[1],[3],0.2]

expected: 3.24

case 2

in: [[1.1],[1],[1.1],0.2]

expected: 0

case 3

in: [[5],[1],[1],0.5]

expected: 16

case 4

in: [[2,2],[1,1],[3,0],0.5]

expected: 3.125

Starter codePython

def ppo_value_loss(values, old_values, returns, clip_eps):
    # Your code here
    pass

4 test cases · run your code against them in the editor