two-dimensional vectors) This approach surprisingly works well in practice, but cannot distinguish different word orders (“John loves Mary” vs “Mary loves John”) loves (1,0) Mary (0,1) John (0.25,-0.25) John loves Mary (1.25, 0.75)

Martens, G Hinton. 2011. Generating text with recurrent neural networks. Proc. of ICML, pp. 1017–1024. John loves ℎ 4 𝑦 ℎℎ Mary ℎℎ much ℎℎ softmax Word embeddings Represent a word with a vector ∈ ℝ ℎ ℎ ℎ 1 2 3 1 2 3 4 Recurrent computation Compose a hidden vector from an input word and the hidden vector −1 at the previous timestep = (ℎ + ℎℎ−1) Fully-connected layer for a task Make a prediction from the hidden vector 4 , which are composed from all words in the sentence, by using a fully-connected layer and softmax 0 = 0 ℎℎ ☺ The parameters ℎ , ℎℎ , 𝑦 are shared over the entire sequence They are trained by the supervision signal 1 , … , 4 , using backpropagation

Forward Backward Concatenate the last hidden vectors of the both directions Fully-connected layer for a task The same as unidirectional RNNs ☺ A Graves, A Mohamed and G Hinton. 2013. Speech Recognition with Deep Recurrent Neural Networks. Proc. of ICASSP, pp. 6645-6649.

, 2 , … , of length Include interactions from the past Neural network is deep in time direction Share parameters of ℎ and ℎℎ over sequence Trained by backpropagation on unfolded graph This is called backpropagation through time (BPTT) RNN 1 , 2 , … , 1 , 2 , … , Unfold RNN 1 1 RNN 2 2 RNN

49, 53, 33, 42], 17], [[22, 46, 43, 42], 17], [[14, 33], 17], …… ] Find alphabet (X) and a set of country names (Y) Build an associative array to map a letter/country into an integer ID Convert letters and countries into integer IDs by using the associative arrays https://github.com/chokkan/deeplearning/blob/master/notebook/rnn.ipynb

input and activation function), = −1 After steps, this is equivalent to multiplying = 0 When has an eigenvalue decomposition, = diag()−1 We can compute as, = diag −1 = diag −1 The eigenvalues are multiplied times When < 1, → 0 (gradient vanishing) When > 1, → ∞ (gradient exploding) Computing in this way is similar to the power method will be close to the eigenvector for the largest eigenvalue of , regardless of the vector 0 I Goodfellow, Y Bengio, A Courville. 2016. Deep Learning, page 286, MIT Press.

(Pascanu+ 2013) Gradient vanishing Activation function: tanh to ReLU Long Short-Term Memory (LSTM) Gated Recurrent Unit (GRU) Residual Networks (Pascanu+ 2013) When the norm of gradients is above the threshold, scale down the gradients R Pascanu, T Mikolov, Y Bengio. 2013. On the difficulty of training recurrent neural networks. Proc. of ICML, pp. 1310-1318.

cuts among states Memory cells do not suffer from zero gradients caused by activation functions (tanh and ReLU) Memory cells are connected without activation functions Information in −1 can flow when a forget gate is wide opened ( = 1) The input from each state ( ∗ ) has no effect in computing −1 + * −1 + * +1 +1 ∗ +1 ∗ +1

(∗ denotes elementwise product): Hidden state: ℎ = ∗ ℎ−1 + 1 − ∗ New hidden state: = tanh ℎ + ℎℎ ( ∗ ℎ−1 ) Reset gate: = 𝑟𝑟 + 𝑟 ℎ−1 Update gate: = 𝑧𝑧 + 𝑧 ℎ−1 Motivated by LSTM unit But much simpler to compute and implement K Cho, van B Merrienboer, C Gulcehre, D Bahdanau, F Bougares, H Schwenk, Y Bengio. 2014. Learning phrase representations using RNN encoder– decoder for statistical machine translation. Proc. of EMNLP, pp. 1724–1734.

character-level language modeling (predicting subsequent characters) LSTMs and GRUs significantly outperform RNNs RNNs seem to learn different embeddings from those of LSTMs and GRUs A Karpathy, J Johnson, and L Fei-Fei. 2016. Visualizing and Understanding Recurrent Networks. Proc. of ICLR Workshop 2016.

E Huang, A Ng, and C Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. Proc. of EMNLP, pp. 151-161. movie good very ( × 2) ・ ・ very good very good movie ( × 2) Compose a phrase vector = , = , ∈ ℝ: constituent vectors ∈ ℝ: phrase vector ∈ ℝ×2: parameter : activation function Recursively compose vectors along the phrase structure (parse tree) of a sentence

word has a semantic vector and composition matrix Compose a phrase vector and matrix recursively = , , = , = [; ] = , = [; ] R Socher, B Huval, C Manning and A Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. Proc. of EMNLP, pp. 1201-1211.

too many parameters to train, assigning every word with a composition matrix Transform a word vector into a composition matrix by using a tensor R Socher, A Perelygin, J Wu, J Chuang, C Manning, A Ng and C Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. Proc. of EMNLP, pp. 1631-1642.

Socher, C D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proc. of ACL- IJCNLP, pp. 1556–1566.

into phrase structures. Each node in a parse tree has a sentiment value (--, -, 0, +, ++) assigned by three annotators. R Socher, A Perelygin, J Wu, J Chuang, C Manning, A Ng and C Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. Proc. of EMNLP, pp. 1631-1642.

Tai, R Socher, C D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proc. of ACL- IJCNLP, pp. 1556–1566.

Convolutional neural networks for sentence classification. Proc. of EMNLP, pp. 1746-1751. It is a very good movie indeed :+ ・ ・ ・ ・ ・ ・ = max 1<<−+1 , Max pooling: each dimension is the maximum number of the values , over timesteps softmax (𝑦𝑦) ☺

P Blunsom. 2014. A convolutional neural network for modelling sentences. Proc. of ACL, pp. 655-665. Max pooling = max 1<<−+1 , Average pooling = 1 − + 1 � =1 −+1 , -max pooling Taking the -max values (instead of 1-max) Dynamic -max pooling Change the value of adaptively based on the length () of an input

best of all (1) (2) (3) (4) Max Pooling Use these vectors (e.g., concatenation of these vectors) as the input to the fully-connected layer for classification H Zhao, Z Lu, P Poupart. 2015. Self-Adaptive Hierarchical Sentence Model. Proc. of IJCAI, pp. 4069-4076.