Long Term Short Term Memory

Transcript

1. Long-Term Short-Term Memory ABECON 11.09.2015 LESZEK Rybicki
2. Outline 1. Memory Game 2. Recurrent Neural Networks 3. Memory Game 4. Long-Term Short-Term Memory Networks 5. Memory Game 6. ...one more thing
3. DAY 2: -3 tasks done, +2 new DAY 3: +3 new tasks, -1 done DAY 4: did -3 tasks, +4 new tasks DAY 5: +1 new task, did -5! Memory Game DAY 1: +5 new tasks What is the name of the intern? Imagine that you are an intern
4. Feedforward Network: XOR 0 xor 0 = 0 1 xor 0 = 1 0 xor 1 = 1 1 xor 1 = 0 0.6 0.4 0.6 0.4 0.5 -0.5 -1 -1
5. Feedforward Network in a box x output= (W 2 · hidden) W 1 W 2 hidden= (W 1 · x + [-1,-1]) 0.6 0.4 0.6 0.4 0.5 -0.5 -1 -1
6. Recurrent Network: XOR [0, 0] = [?, 0] [1, 0] = [?, 1] [0, 1] = [?, 1] [1, 1] = [?, 0] t-1
7. Recurrent Network: unfolded x t+1 x t+2 x t+3 t-1 x t h t h t+1 h t+2 h t+3
8. Trouble with recurrent networks - short attention span - not good with distractions - unstable when training - training takes a long time - short attention span
10. Jürgen Schmidhuber “YOU AGAIN” To avoid long time lag problems of gradient-based approaches we may simply randomly initialize all network weights until the resulting net happens to classify all training sequences correctly. In fact, recently we discovered that simple weight guessing solves many of the problems faster than the algorithms proposed therein. This does not mean that weight guessing is a good algorithm. It just means that the problems are very simple. -- Jürgen Schmidhuber LONG SHORT-TERM MEMORY Neural Computation 9(8):1735{1780, 1997}
11. LSTM Unit ✕ tanh ✕ ✕ ta + ✕ ✕ ✕ tanh tanh + x t h t h t-1 x t+1 C t h t-1 C t-1 h t
12. Symbols ✕ neural network layer pointwise operation + concatenate two vectors clone vector ✕ [x,y,z] [a,b,c] [ax,by,cz] + [x,y,z] [a,b,c] [a+x, b+y, c+z] tanh
13. Cell state ✕ ✕ tanh tanh x t h t C t h t-1 C t-1 h t + ✕
14. Forget gate ✕ ✕ tanh tanh + x t h t C t h t-1 C t-1 ✕ h t ✕ x,y,z a,b,c ax,by,cz
15. Update gate ✕ tanh x t h t C t h t-1 C t-1 ✕ ✕ tanh + h t + x,y,z a,b,c a+x, b+y, c+z
16. Select gate x t h t C t h t-1 C t-1 ✕ ✕ tanh + ✕ tanh h t tanh
17. LSTM unit ✕ ✕ ✕ tanh tanh + x t h t C t h t
18. READ ROMAJI ONLY Memory Game J た Ü ロ 駅R ऌ G E N 止
19. End-to-end people detection in crowded scenes Russell Stewart, Mykhaylo Andriluka • Mechanical Turk • GoogLeNet • LSTM • Hungarian loss algorithm
20. Understanding LSTM Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Prof. Schmidhuber http://people.idsia.ch/~juergen/ LSTM in the browser with Synaptic.js http://synaptic.juancazala.com/#/dsr Read more