GRPO Objective Value

Problem

Compute the GRPO clipped surrogate objective over one group.

For each sample form the ratio exp(log_probs[i] - old_log_probs[i]) and use rewards[i] as the group-relative advantage. Return the mean of min(ratio * advantage, clip(ratio, 1 - clip_eps, 1 + clip_eps) * advantage); an empty group returns 0.0.

Hint

The pessimistic min keeps the update honest when the ratio leaves the trust region.

Test cases

case 1

in: [[1,-1],[0,0],[0,0],0.2]

expected: 0

case 2

in: [[1],[-0.1],[-0.5],0.2]

expected: 1.2

case 3

in: [[-1],[-0.1],[-0.5],0.2]

expected: -1.4918246976412703

case 4

in: [[],[],[],0.2]

expected: 0

Starter codePython

import math
def grpo_objective(rewards, log_probs, old_log_probs, clip_eps):
    # Your code here
    pass

4 test cases · run your code against them in the editor