Per-Decision IS

Problem

Return the per-decision importance-sampling terms of an off-policy episode.

At each step multiply the running ratio by target_probs[t][a_t] / behavior_probs[t][a_t] and record ratio * gamma^t * reward_t. Return the list of terms.

Hint

Per-decision IS only reweights rewards from the step of each ratio onward.

Test cases

case 1

in: [[1,1,1],[[0.5,0.5],[0.5,0.5],[0.5,0.5]],[[0.25,0.75],[0.5,0.5],[0.8,0.2]],[0,1,0],0.9]

expected: [2,1.8,1.0125000000000002]

case 2

in: [[2,0],[[0.5,0.5],[0.5,0.5]],[[0.5,0.5],[0.5,0.5]],[1,0],0.5]

expected: [2,0]

case 3

in: [[1,2],[[0.6,0.4],[0.2,0.8]],[[0.3,0.7],[0.5,0.5]],[1,0],1]

expected: [0.5714285714285715,0.45714285714285724]

Starter codePython

def per_decision_is(rewards, target_probs, behavior_probs, actions, gamma):
    # Your code here
    pass

3 test cases · run your code against them in the editor