First-Visit vs Every-Visit MC

Problem

Compare first-visit and every-visit Monte Carlo value estimates for a target state.

Find every index where states equals target and compute the discounted return from each occurrence to the end of the episode. Return [first_visit_return, mean_of_all_visit_returns], or [0.0, 0.0] if the state never occurs.

Hint

Every-visit averages over all occurrences while first-visit keeps only the earliest.

Test cases

case 1

in: [["A","B","A"],[1,2,3],"A",0.5]

expected: [2.75,2.875]

case 2

in: [["A","A"],[1,1],"A",1]

expected: [2,1.5]

case 3

in: [["A","B"],[1,2],"C",0.9]

expected: [0,0]

case 4

in: [["B"],[5],"B",0.9]

expected: [5,5]

Starter codePython

def mc_first_vs_every(states, rewards, target, gamma):
    # Your code here
    pass

4 test cases · run your code against them in the editor