Model-Based Value Expansion

Problem

Average the model-based value estimates over horizons 1 through H.

model_rewards gives the H imagined rewards and values[k] is the value estimate after k imagined steps. For each horizon k, V_k = sum of gamma^t * r_t for t < k plus gamma^k * values[k]. Return the mean of V_1 through V_H.

Hint

Each horizon bootstraps with the value at that horizon, then the horizons are averaged.

Test cases

case 1

in: [[1,1],[0,2,3],0.9]

expected: 3.565

case 2

in: [[2],[0,4],0.5]

expected: 4

case 3

in: [[1,1,1],[0,0,0,0],0.9]

expected: 1.8699999999999999

case 4

in: [[0],[0,5],1]

expected: 5

Starter codePython

def mbve_value(model_rewards, values, gamma):
    # Your code here
    pass

4 test cases · run your code against them in the editor