FQE Estimate Lite

Problem

Compute a fitted Q evaluation estimate with importance weights.

For each transition weight the Q value of the taken action by target_probs[i][a] / behavior_probs[i][a] and return the mean over the batch. An empty batch returns 0.0.

Hint

FQE reweights logged actions toward the target policy before averaging values.

Test cases

case 1

in: [[[2,4],[3,1]],[[0.5,0.5],[0.5,0.5]],[[0.25,0.75],[0.5,0.5]],[0,1]]

expected: 2.5

case 2

in: [[[2,4]],[[0.25,0.75]],[[0.5,0.5]],[1]]

expected: 6

case 3

in: [[],[],[],[]]

expected: 0

Starter codePython

def fqe_estimate_lite(q_values, target_probs, behavior_probs, actions):
    # Your code here
    pass

3 test cases · run your code against them in the editor