Double DQN Bias Check

Problem

Measure how much the standard DQN target overestimates relative to the Double DQN target.

standard = reward + gamma * max(target_next) and double = reward + gamma * target_next[argmax(online_next)], using the smallest index for ties. Return standard - double.

Hint

The gap is positive exactly when the max target action is not the online-greedy action.

Test cases

case 1

in: [1,0.9,[1,3,2],[5,1,2]]

expected: 3.6

case 2

in: [0,0.5,[2,2],[3,3]]

expected: 0

case 3

in: [2,1,[1],[0]]

expected: 0

case 4

in: [-1,0.8,[0,1],[2,4]]

expected: 0

Starter codePython

def double_dqn_bias_check(reward, gamma, online_next, target_next):
    # Your code here
    pass

4 test cases · run your code against them in the editor