Double Q-Learning Update

Problem

Apply one Double Q-learning update to one of two Q tables.

If update_first is true, select the greedy action from Q1[s_next] and evaluate it with Q2[s_next], updating Q1[s][a]; otherwise select from Q2 and evaluate with Q1, updating Q2. Break argmax ties by smallest index and return [Q1, Q2] as deep copies.

Hint

One table chooses the action, the other table scores it, which fights maximization bias.

Test cases

case 1

in: [[[1,2],[0.5,3]],[[4,1],[2,2.5]],0,1,1.5,1,0.1,0.9,true]

expected: [[[1,2.175],[0.5,3]],[[4,1],[2,2.5]]]

case 2

in: [[[1,2],[0.5,3]],[[4,1],[2,2.5]],0,1,1.5,1,0.1,0.9,false]

expected: [[[1,2],[0.5,3]],[[4,1.32],[2,2.5]]]

case 3

in: [[[0,0],[3,3]],[[1,1],[5,1]],0,0,0,1,0.5,1,true]

expected: [[[2.5,0],[3,3]],[[1,1],[5,1]]]

Starter codePython

def double_q_learning_update(Q1, Q2, s, a, r, s_next, alpha, gamma, update_first):
    # Your code here
    pass

3 test cases · run your code against them in the editor