PPO-RLHF Ratio

Problem

Compute the per-token PPO ratio used in RLHF: exp(log_probs_new - log_probs_old) for every token. An empty batch returns [].

Hint

Ratios above 1 mean the new policy assigns more probability to the sampled token.

case 1

in: [[-1,-2],[-1,-2]]

expected: [1,1]

case 2

in: [[-1],[-2]]

expected: [2.718281828459045]

case 3

in: [[-2],[-1]]

expected: [0.36787944117144233]

case 4

in: [[],[]]

expected: []

Starter codePython

import math
def ppo_rlhf_ratio(log_probs_new, log_probs_old):
    # Your code here
    pass

4 test cases · run your code against them in the editor