Relaxed Off-Policy Correction

Problem

Compute a relaxed off-policy correction factor for a trajectory.

Multiply (target_probs[t][action_t] / behavior_probs[t][action_t]) ** eta over all steps. An empty trajectory returns 1.0.

Hint

An exponent eta between 0 and 1 trades bias for much lower variance.

case 1

in: [[[0.4,0.6]],[[0.2,0.8]],[0],1]

expected: 2

case 2

in: [[[0.4,0.6]],[[0.2,0.8]],[0],2]

expected: 4

case 3

in: [[[0.5,0.5],[0.5,0.5]],[[0.5,0.5],[0.5,0.5]],[0,1],1]

expected: 1

case 4

in: [[],[],[],1]

expected: 1

Starter codePython

def relaxed_off_policy_factor(target_probs, behavior_probs, actions, eta):
    # Your code here
    pass

4 test cases · run your code against them in the editor