Expected SARSA Update

Problem

Apply one Expected SARSA update using the full policy distribution over next actions.

expected = sum_b probs[b] * Q[s_next][b], then Q[s][a] += alpha * (r + gamma * expected - Q[s][a]). Return a deep copy of Q.

Hint

Replace the sampled next action with its expectation under the policy.

case 1

in: [[[1,2],[3,4]],0,0,1,1,[0.25,0.75],0.1,0.9]

expected: [[1.3375,2],[3,4]]

case 2

in: [[[1,2],[3,4]],1,1,-1,0,[1,0],0.5,0.5]

expected: [[1,2],[3,1.75]]

case 3

in: [[[1,2]],0,1,0,0,[0.5,0.5],1,1]

expected: [[1,1.5]]

Starter codePython

def expected_sarsa_update(Q, s, a, r, s_next, probs, alpha, gamma):
    # Your code here
    pass

3 test cases · run your code against them in the editor