RNN BPTT One Step

Problem

One backward step through a vanilla RNN cell. Given the upstream gradient dh for the hidden state h = tanh(Wx x + Wh h_prev + b), first compute dpre = dh * (1 - h^2), then dWx = outer(dpre, x), dWh = outer(dpre, h_prev), db = dpre and dh_prev = Wh^T dpre.

The signature is rnn_bptt_step(dh, x, h_prev, h, Wx, Wh). Return [dWx, dWh, db, dh_prev].

Hint

The tanh derivative is 1 - h^2; propagate through Wh to reach the previous hidden state.

Test cases

case 1

in: [[1,0],[1,0],[0,0],[0.5,0],[[1,0],[0,1]],[[1,0],[0,1]]]

expected: [[[0.75,0],[0,0]],[[0,0],[0,0]],[0.75,0],[0.75,0]]

case 2

in: [[1,1],[1,2],[0.5,0.5],[0.5,0.5],[[1,1],[0,1]],[[0.5,0],[0,0.5]]]

expected: [[[0.75,1.5],[0.75,1.5]],[[0.375,0.375],[0.375,0.375]],[0.75,0.75],[0.375,0.375]]

case 3

in: [[2],[1],[0.5],[0.25],[[1]],[[1]]]

expected: [[[1.875]],[[0.9375]],[1.875],[1.875]]

case 4

in: [[-1,0.5],[0,-1],[1,0],[-0.5,0],[[0.5,0],[0,2]],[[1,0],[0,1]]]

expected: [[[0,0.75],[0,-0.5]],[[-0.75,0],[0.5,0]],[-0.75,0.5],[-0.75,0.5]]

Starter codePython

def rnn_bptt_step(dh, x, h_prev, h, Wx, Wh):
    # Returns [dWx, dWh, db, dh_prev]
    # Your code here
    pass

4 test cases · run your code against them in the editor