PPO Clipped Objective

Problem

Compute the PPO clipped surrogate objective: mean of min(ratio * advantage, clip(ratio, 1 - clip_eps, 1 + clip_eps) * advantage), where ratio = exp(logp_new - logp_old).

The signature is ppo_clipped_objective(logp_new, logp_old, advantages, clip_eps). Return a scalar.

Hint

The min prevents updates that move the policy too far in either direction.

Test cases

case 1

in: [[0,0],[0,0],[1,-1],0.2]

expected: 0

case 2

in: [[0.5],[0],[1],0.2]

expected: 1.2

case 3

in: [[0.5],[0],[-1],0.2]

expected: -1.6487212707

case 4

in: [[-0.5],[0],[1],0.2]

expected: 0.6065306597

case 5

in: [[-0.5],[0],[-1],0.2]

expected: -0.8

Starter codePython

import math
def ppo_clipped_objective(logp_new, logp_old, advantages, clip_eps):
    # Your code here
    pass

5 test cases · run your code against them in the editor