Doubly Robust OPE

Problem

Compute the doubly robust off-policy estimate. For each sample add the target policy's expected value sum_a pi(a)*q(a) plus the importance-weighted correction pi(a_taken)/behavior * (reward - q(a_taken)).

Return the mean over samples; empty input returns 0.0.

Hint

The model-based term reduces variance while the importance weight corrects its bias.

Test cases

case 1

in: [[1,2],[0,1],[0.5,0.5],[[0.7,0.3],[0.4,0.6]],[[1,0],[0,2]]]

expected: 0.95

case 2

in: [[1],[1],[0.25],[[0.5,0.5]],[[1,3]]]

expected: -2

case 3

in: [[],[],[],[],[]]

expected: 0

Starter codePython

def dr_ope(rewards, actions, behavior_probs, target_probs, q_values):
    # Your code here
    pass

3 test cases · run your code against them in the editor