V-Trace Clipped Ratio

Problem

Compute the clipped importance ratios used by IMPALA V-trace.

For each step take ratio = target_probs[t][action] / behavior_probs[t][action], clip it to rho_bar to get the value ratio, and clip it to c_bar to get the trace coefficient. Return the list of [rho, c] pairs. Behavior probabilities are positive.

Hint

Separate clipping bounds keep the target variance low while allowing longer traces.

Test cases

case 1

in: [[[0.5,0.5],[0.8,0.2]],[[0.25,0.75],[0.5,0.5]],[0,1],1,1]

expected: [[1,1],[0.4,0.4]]

case 2

in: [[[0.5,0.5],[0.8,0.2]],[[0.25,0.75],[0.5,0.5]],[0,1],0.5,1]

expected: [[0.5,1],[0.4,0.4]]

case 3

in: [[],[],[],1,1]

expected: []

case 4

in: [[[0.9,0.1]],[[0.3,0.7]],[1],3,0.9]

expected: [[0.14285714285714288,0.14285714285714288]]

Starter codePython

def vtrace_clipped_ratio(target_probs, behavior_probs, actions, rho_bar, c_bar):
    # Your code here
    pass

4 test cases · run your code against them in the editor