Tree Backup Lite

Problem

Compute the one-step tree-backup target.

If the episode is done return the reward; otherwise return reward + gamma * (policy_probs . q_next). The policy probabilities come from the target policy, not the behavior policy.

Hint

Tree backup never uses importance ratios, so it is safe off-policy.

Test cases

case 1

in: [1,false,0.9,[0.5,0.5],[2,4]]

expected: 3.7

case 2

in: [1,true,0.9,[0.5,0.5],[2,4]]

expected: 1

case 3

in: [0,false,0.5,[1,0],[6,1]]

expected: 3

Starter codePython

def tree_backup_lite(reward, done, gamma, policy_probs, q_next):
    # Your code here
    pass

3 test cases · run your code against them in the editor