PPO Clip Objective Value

Problem

Compute the PPO clipped surrogate objective over a batch.

For each sample take min(ratio * advantage, clip(ratio, 1 - clip_eps, 1 + clip_eps) * advantage) and return the mean. An empty batch returns 0.0.

Hint

This is the objective PPO maximizes, so the value is a reward-like score rather than a loss.

case 1

in: [[1],[2],0.2]

expected: 2

case 2

in: [[1.5],[1],0.2]

expected: 1.2

case 3

in: [[0.5],[-1],0.2]

expected: -0.8

case 4

in: [[1.5,0.5],[1,-1],0.2]

expected: 0.2

case 5

in: [[],[],0.2]

expected: 0

Starter codePython

def ppo_clip_objective(ratios, advantages, clip_eps):
    # Your code here
    pass

5 test cases · run your code against them in the editor