Retrace Lite

Problem

Compute a one-step Retrace target with a truncated importance ratio.

c = min(c_bar, target_probs[a] / behavior_probs[a]) and V = target_probs . q_next; return reward + gamma * c * V. A done episode returns just the reward.

Hint

Truncating the ratio bounds the variance Retrace can inherit from a bad behavior policy.

Test cases

case 1

in: [1,false,0.9,[0.5,0.5],[0.25,0.75],0,[2,4],1]

expected: 3.7

case 2

in: [1,false,0.9,[0.5,0.5],[0.05,0.95],0,[2,4],0.2]

expected: 1.54

case 3

in: [0,true,0.9,[0.5,0.5],[0.5,0.5],1,[2,4],1]

expected: 0

Starter codePython

def retrace_lite(reward, done, gamma, target_probs, behavior_probs, action, q_next, c_bar):
    # Your code here
    pass

3 test cases · run your code against them in the editor