Long-Term Short-Term Memory ABECON 11.09.2015 LESZEK Rybicki

Outline 1. Memory Game 2. Recurrent Neural Networks 3. Memory Game 4. Long-Term Short-Term Memory Networks 5. Memory Game 6. ...one more thing

DAY 2: -3 tasks done, +2 new DAY 3: +3 new tasks, -1 done DAY 4: did -3 tasks, +4 new tasks DAY 5: +1 new task, did -5! Memory Game DAY 1: +5 new tasks What is the name of the intern? Imagine that you are an intern

Feedforward Network: XOR 0 xor 0 = 0 1 xor 0 = 1 0 xor 1 = 1 1 xor 1 = 0 0.6 0.4 0.6 0.4 0.5 -0.5 -1 -1

Feedforward Network in a box x output= (W 2 · hidden) W 1 W 2 hidden= (W 1 · x + [-1,-1]) 0.6 0.4 0.6 0.4 0.5 -0.5 -1 -1

Recurrent Network: XOR [0, 0] = [?, 0] [1, 0] = [?, 1] [0, 1] = [?, 1] [1, 1] = [?, 0] t-1

Recurrent Network: unfolded x t+1 x t+2 x t+3 t-1 x t h t h t+1 h t+2 h t+3

Trouble with recurrent networks - short attention span - not good with distractions - unstable when training - training takes a long time - short attention span

No content

Jürgen Schmidhuber “YOU AGAIN” To avoid long time lag problems of gradient-based approaches we may simply randomly initialize all network weights until the resulting net happens to classify all training sequences correctly. In fact, recently we discovered that simple weight guessing solves many of the problems faster than the algorithms proposed therein. This does not mean that weight guessing is a good algorithm. It just means that the problems are very simple. -- Jürgen Schmidhuber LONG SHORT-TERM MEMORY Neural Computation 9(8):1735{1780, 1997}

LSTM Unit ✕ tanh ✕ ✕ ta + ✕ ✕ ✕ tanh tanh + x t h t h t-1 x t+1 C t h t-1 C t-1 h t

Symbols ✕ neural network layer pointwise operation + concatenate two vectors clone vector ✕ [x,y,z] [a,b,c] [ax,by,cz] + [x,y,z] [a,b,c] [a+x, b+y, c+z] tanh

Cell state ✕ ✕ tanh tanh x t h t C t h t-1 C t-1 h t + ✕

Forget gate ✕ ✕ tanh tanh + x t h t C t h t-1 C t-1 ✕ h t ✕ x,y,z a,b,c ax,by,cz

Update gate ✕ tanh x t h t C t h t-1 C t-1 ✕ ✕ tanh + h t + x,y,z a,b,c a+x, b+y, c+z

Select gate x t h t C t h t-1 C t-1 ✕ ✕ tanh + ✕ tanh h t tanh

LSTM unit ✕ ✕ ✕ tanh tanh + x t h t C t h t

READ ROMAJI ONLY Memory Game J た Ü ロ 駅R ऌ G E N 止

End-to-end people detection in crowded scenes Russell Stewart, Mykhaylo Andriluka ● Mechanical Turk ● GoogLeNet ● LSTM ● Hungarian loss algorithm

Understanding LSTM Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Prof. Schmidhuber http://people.idsia.ch/~juergen/ LSTM in the browser with Synaptic.js http://synaptic.juancazala.com/#/dsr Read more