Off-Policy Corrected Return

Problem

Compute the per-decision importance-sampling estimate of the return.

At each step t multiply the running ratio by target_probs[t][a_t] / behavior_probs[t][a_t], then add ratio * gamma^t * rewards[t] to the total. Return the accumulated total.

Hint

The running product reweights each reward by the probability of the partial trajectory.

Test cases

case 1

in: [[1,1,1],[[0.5,0.5],[0.5,0.5],[0.5,0.5]],[[0.25,0.75],[0.5,0.5],[0.8,0.2]],[0,1,0],0.9]

expected: 4.8125

case 2

in: [[2,0],[[0.5,0.5],[0.5,0.5]],[[0.5,0.5],[0.5,0.5]],[1,0],0.5]

expected: 2

case 3

in: [[1],[[0.9,0.1]],[[0.3,0.7]],[1],0.99]

expected: 0.14285714285714288

Starter codePython

def off_policy_corrected_return(rewards, target_probs, behavior_probs, actions, gamma):
    # Your code here
    pass

3 test cases · run your code against them in the editor