DNN for Structural Data Naoaki Okazaki School of Computing, Tokyo Institute of Technology [email protected] PowerPoint template designed by https://ppt.design4u.jp/template/

Embeddings for phrases and sentences 1  Word embeddings represent words with real-valued vectors  Is it possible to consider embeddings for phrases and sentences? John loves Mary John loves Mary

The baseline: additive composition 2 (Example with word embeddings of two-dimensional vectors) This approach surprisingly works well in practice, but cannot distinguish different word orders (“John loves Mary” vs “Mary loves John”) loves (1,0) Mary (0,1) John (0.25,-0.25) John loves Mary (1.25, 0.75)

Summary  Various NN architectures that can leverage structures  Recurrent Neural Networks (RNNs)  Long Short-Term Memories (LSTMs)  Gated Recurrent Units (GRU)  Recursive Neural Networks (Recursive NNs)  Convolutional Neural Networks (CNNs) 3

Recurrent Neural Networks (RNNs) 4

Recurrent Neural Networks (RNNs) (Sutskever+ 2011) 5 I Sutskever, J Martens, G Hinton. 2011. Generating text with recurrent neural networks. Proc. of ICML, pp. 1017–1024. John loves ℎ 4 𝑦 ℎℎ Mary ℎℎ much ℎℎ softmax Word embeddings Represent a word with a vector ∈ ℝ ℎ ℎ ℎ 1 2 3 1 2 3 4 Recurrent computation Compose a hidden vector from an input word and the hidden vector −1 at the previous timestep = (ℎ + ℎℎ−1) Fully-connected layer for a task Make a prediction from the hidden vector 4 , which are composed from all words in the sentence, by using a fully-connected layer and softmax 0 = 0 ℎℎ ☺ The parameters ℎ , ℎℎ , 𝑦 are shared over the entire sequence They are trained by the supervision signal 1 , … , 4 , using backpropagation

RNN in math 6 ℎ ℎℎ + −1 tanh ℎ ℎℎ + +1 tanh +1 +1 = RNN , −1 = ℎ + ℎℎ −1 , ∈ ℝ, ∈ ℝ, ℎ ∈ ℝ×, ℎℎ ∈ ℝ× Typical activation functions are tanh and ReLU RNN RNN

Multi-layer RNNs 7 ℎ (1) ℎℎ (1) + −1 (1) tanh (1) ℎ (1) ℎℎ (1) + +1 (0) tanh +1 (1) (0) RNN(1) RNN(1) ℎ (2) ℎℎ (2) + −1 (2) tanh (2) ℎ (2) ℎℎ (2) + tanh +1 (2) +1 (2) (2) RNN(2) RNN(2) = = +1

Forward and backward RNNs 8 ℎ ℎℎ + −1 tanh RNN ℎ ℎℎ + −1 tanh RNN Forward RNNs = RNN , −1 = ℎ + ℎℎ −1 Backward RNNs −1 = RNN , = ℎ + ℎℎ

Bidirectional RNNs (Graves+ 2013) 9 John loves Mary much softmax Forward Backward Concatenate the last hidden vectors of the both directions Fully-connected layer for a task The same as unidirectional RNNs ☺ A Graves, A Mohamed and G Hinton. 2013. Speech Recognition with Deep Recurrent Neural Networks. Proc. of ICASSP, pp. 6645-6649.

Unfolded Recurrent Neural Network 10  Process a sequence 1 , 2 , … , of length  Include interactions from the past  Neural network is deep in time direction  Share parameters of ℎ and ℎℎ over sequence  Trained by backpropagation on unfolded graph  This is called backpropagation through time (BPTT) RNN 1 , 2 , … , 1 , 2 , … , Unfold RNN 1 1 RNN 2 2 RNN

Example: RNN for nationality prediction 11 G o ℎ 4 𝑦 ℎℎ t ℎℎ o ℎℎ softmax ℎ ℎ ℎ 1 2 3 1 2 3 4 ∈ ℝ18 0 = 0 ℎℎ ∈ ℝ55 (one-hot vector) (55 = |[A-Za-z .,;']|) ∈ ℝ128

Preprocess the data 12 [ [ "Nguyen", "Vietnamese“ ], [ "Tron", "Vietnamese“ ], [ "Le", "Vietnamese“ ], …… ] https://github.com/chokkan/deeplearning/blob/master/notebook/rnn.ipynb

Convert the data into numerical data 13 [ [[16, 35, 49, 53, 33, 42], 17], [[22, 46, 43, 42], 17], [[14, 33], 17], …… ] Find alphabet (X) and a set of country names (Y) Build an associative array to map a letter/country into an integer ID Convert letters and countries into integer IDs by using the associative arrays https://github.com/chokkan/deeplearning/blob/master/notebook/rnn.ipynb

Bare implementation of RNN states 14 = ℎ + ℎℎ −1 = (ℎ [ ; −1 ]) https://github.com/chokkan/deeplearning/blob/master/notebook/rnn.ipynb

Sequential RNN module 15 https://github.com/chokkan/deeplearning/blob/master/notebook/rnn.ipynb

Mini-batch RNN 16 https://github.com/chokkan/deeplearning/blob/master/notebook/rnn.ipynb

Long-term dependency 17  Consider a simplified RNN (without an input and activation function), = −1  After steps, this is equivalent to multiplying = 0  When has an eigenvalue decomposition, = diag()−1  We can compute as, = diag −1 = diag −1  The eigenvalues are multiplied times  When < 1, → 0 (gradient vanishing)  When > 1, → ∞ (gradient exploding)  Computing in this way is similar to the power method  will be close to the eigenvector for the largest eigenvalue of , regardless of the vector 0 I Goodfellow, Y Bengio, A Courville. 2016. Deep Learning, page 286, MIT Press.

Gradient vanishing/exploding problem 18  Gradients vanish or explode over time  More detailed explanations:  Why are deep neural networks hard to train? http://neuralnetworksanddeeplearning.com/chap5.html  ニューラルネットワークを訓練するのはなぜ難しいのか https://nnadl-ja.github.io/nnadl_site_ja/chap5.html  Recurrent Neural Networks LSTMs and Vanishing & Exploding Gradients - Fun and Easy Machine Learning https://www.youtube.com/watch?v=2GNbIKTKCfE RNN 1 1 RNN 2 2 RNN RNN −2 −2 RNN −1 −1

Addressing gradient vanishing/exploding 19  Gradient exploding  Gradient clipping (Pascanu+ 2013)  Gradient vanishing  Activation function: tanh to ReLU  Long Short-Term Memory (LSTM)  Gated Recurrent Unit (GRU)  Residual Networks (Pascanu+ 2013) When the norm of gradients is above the threshold, scale down the gradients R Pascanu, T Mikolov, Y Bengio. 2013. On the difficulty of training recurrent neural networks. Proc. of ICML, pp. 1310-1318.

Long Short-Term Memory (LSTM) 20

Long Short-Term Memory (Hochreiter+ 1997) 21  Consist of (∗ denotes elementwise product):  Hidden state: ℎ = ∗ tanh  Memory cell: = ∗ −1 + ∗ tanh 𝑔𝑔 + 𝑔 ℎ−1  Input gate: = 𝑖𝑖 + 𝑖 ℎ−1  Output gate: = 𝑜𝑜 + 𝑜 ℎ−1  Forget gate: = 𝑓𝑓 + 𝑓 ℎ−1  The architecture looks complicated, but LSTMs are also a neural network  LSTMs can be also trained by the standard procedure of backpropagation S Hochreiter, J Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780.

LSTM in math and diagram 22 𝑓𝑓 𝑓 𝑖𝑖 𝑖 𝑔𝑔 𝑔 𝑜𝑜 𝑜 + + + tanh + * + * tanh * ℎ−1 −1 ℎ ℎ Memory cell Hidden state Forget gate Input gate Output gate = 𝑓𝑓 + 𝑓 ℎ−1 = + ℎ ℎ−1 = + ℎ ℎ−1 = tanh 𝑔𝑔 + 𝑔 ℎ−1 = ∗ −1 + ∗ ℎ = ∗ tanh

Sequential LSTM in pytorch 23  Replace torch.nn.RNN to torch.nn.LSTM  Change the shape of an initial state

Implementation of LSTM cell in pytorch 24 𝑓𝑓 𝑓 𝑖𝑖 𝑖 𝑔𝑔 𝑔 𝑜𝑜 𝑜 + + + tanh + * + * tanh * ℎ−1 −1 ℎ ℎ Memory cell Hidden state Forget gate Input gate Output gate def LSTMCell(x, hidden, w_x, w_h, b_x=None, b_h=None): h_prev, c_prev = hidden gates = F.linear(x, w_x, b_x) + F.linear(h_prev, w_h, b_h) ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1) ingate = F.sigmoid(ingate) forgetgate = F.sigmoid(forgetgate) cellgate = F.tanh(cellgate) outgate = F.sigmoid(outgate) ct = (forgetgate * c_prev) + (ingate * cellgate) ht = outgate * F.tanh(ct) return ht, ct

Implementation of LSTM cell in pytorch 25 𝑓𝑓 𝑖𝑖 𝑔𝑔 𝑜𝑜 + + + + (x) w_x = [𝑓𝑓 ; 𝑖𝑖 ; 𝑔𝑔 ; 𝑜𝑜 ] def LSTMCell(x, hidden, w_x, w_h, b_x=None, b_h=None): h_prev, c_prev = hidden gates = F.linear(x, w_x, b_x)

Implementation of LSTM cell in pytorch 26 𝑓𝑓 𝑓 𝑖𝑖 𝑖 𝑔𝑔 𝑔 𝑜𝑜 𝑜 + + + + ℎ−1 (h_prev) (x) w_x = [𝑓𝑓 ; 𝑖𝑖 ; 𝑔𝑔 ; 𝑜𝑜 ] w_h = [ℎ ; ℎ ; ℎ ; ℎ ] def LSTMCell(x, hidden, w_x, w_h, b_x=None, b_h=None): h_prev, c_prev = hidden gates = F.linear(x, w_x, b_x) + F.linear(h_prev, w_h, b_h) gates

Implementation of LSTM cell in pytorch 27 gates = F.linear(x, w_x, b_x) + F.linear(h_prev, w_h, b_h) ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1) ingate = F.sigmoid(ingate) forgetgate = F.sigmoid(forgetgate) cellgate = F.tanh(cellgate) outgate = F.sigmoid(outgate) tanh gates ingate forgetgate cellgate outgate

Implementation of LSTM cell in pytorch 28 outgate = F.sigmoid(outgate) ct = (forgetgate * c_prev) + (ingate * cellgate) ht = outgate * F.tanh(ct) return ht, ct * + * tanh * −1 (c_prev) ℎ ℎ ingate forgetgate cellgate outgate

LSTM remedies vanishing gradients 29  Memory cells provide short cuts among states  Memory cells do not suffer from zero gradients caused by activation functions (tanh and ReLU)  Memory cells are connected without activation functions  Information in −1 can flow when a forget gate is wide opened ( = 1)  The input from each state ( ∗ ) has no effect in computing −1 + * −1 + * +1 +1 ∗ +1 ∗ +1

Gated Recurrent Units (GRUs) 30

Gated Recurrent Unit (GRU) (Cho+ 2014) 31  Consist of (∗ denotes elementwise product):  Hidden state: ℎ = ∗ ℎ−1 + 1 − ∗  New hidden state: = tanh ℎ + ℎℎ ( ∗ ℎ−1 )  Reset gate: = 𝑟𝑟 + 𝑟 ℎ−1  Update gate: = 𝑧𝑧 + 𝑧 ℎ−1  Motivated by LSTM unit  But much simpler to compute and implement K Cho, van B Merrienboer, C Gulcehre, D Bahdanau, F Bougares, H Schwenk, Y Bengio. 2014. Learning phrase representations using RNN encoder– decoder for statistical machine translation. Proc. of EMNLP, pp. 1724–1734.

GRU in math and diagram 32 𝑟𝑟 𝑟 ℎ ℎℎ 𝑧𝑧 𝑧 + + + * * tanh ℎ−1 ℎ Reset gate * 1 − + Update gate = tanh ℎ + ℎℎ ( ∗ ℎ−1 ) ℎ = ∗ ℎ−1 + 1 − ∗ = 𝑟𝑟 + 𝑟 ℎ−1 = + ℎ ℎ−1

Sequential GRU in PyTorch 33  Replace torch.nn.RNN to torch.nn.GRU  The shape of an initial state is unchanged

Implementation of GRU cell in PyTorch 34 𝑟𝑟 𝑟 ℎ ℎℎ 𝑧𝑧 𝑧 + + + * tanh ℎ−1 ℎ = 𝑟𝑟 + 𝑟 ℎ−1 Reset gate - + Update gate * = 𝑧𝑧 + 𝑧 ℎ−1 This is more computationally efficient = tanh ℎ + ℎℎ ( ∗ ℎ−1 ) = tanh ℎ + ∗ ℎℎ ℎ−1 ℎ = ∗ ℎ−1 + 1 − ∗ = + ∗ (ℎ−1 − )

Implementation of GRU cell in PyTorch 35 𝑟𝑟 𝑟 ℎ ℎℎ 𝑧𝑧 𝑧 + + + * tanh ℎ−1 ℎ Reset gate - + Update gate * def GRUCell(x, hidden, w_x, w_h, b_x=None, b_h=None): gx = F.linear(input, w_x, b_x) gh = F.linear(hidden, w_h, b_h) x_r, x_i, x_n = gi.chunk(3, 1) h_r, h_i, h_n = gh.chunk(3, 1) resetgate = F.sigmoid(x_r + h_r) inputgate = F.sigmoid(x_i + h_i) newgate = F.tanh(x_n + resetgate * h_n) hy = newgate + inputgate * (hidden - newgate) return hy

Implementation of GRU cell in PyTorch 36 𝑟𝑟 ℎ 𝑧𝑧 ℎ−1 def GRUCell(x, hidden, w_x, w_h, b_x=None, b_h=None): gx = F.linear(input, w_x, b_x)

Implementation of GRU cell in PyTorch 37 𝑟𝑟 𝑟 ℎ ℎℎ 𝑧𝑧 𝑧 ℎ−1 def GRUCell(x, hidden, w_x, w_h, b_x=None, b_h=None): gx = F.linear(input, w_x, b_x) gh = F.linear(hidden, w_h, b_h)

Implementation of GRU cell in PyTorch 38 𝑟𝑟 𝑟 ℎ ℎℎ 𝑧𝑧 𝑧 ℎ−1 h_r h_n x_i x_r x_n h_i def GRUCell(x, hidden, w_x, w_h, b_x=None, b_h=None): gx = F.linear(input, w_x, b_x) gh = F.linear(hidden, w_h, b_h) x_r, x_i, x_n = gi.chunk(3, 1) h_r, h_i, h_n = gh.chunk(3, 1)

Implementation of GRU cell in PyTorch 39 𝑟𝑟 𝑟 𝑧𝑧 𝑧 + + ℎ−1 resetgate inputgate resetgate = F.sigmoid(x_r + h_r) inputgate = F.sigmoid(x_i + h_i) = 𝑟𝑟 + 𝑟 ℎ−1 = 𝑧𝑧 + 𝑧 ℎ−1

Implementation of GRU cell in PyTorch 40 𝑟𝑟 𝑟 ℎ ℎℎ 𝑧𝑧 𝑧 + + + * tanh ℎ−1 ℎ - + * hy newgate inputgate hidden resetgate newgate = F.tanh(x_n + resetgate * h_n) hy = newgate + inputgate * (hidden - newgate) = 𝑟𝑟 + 𝑟 ℎ−1 = 𝑧𝑧 + 𝑧 ℎ−1

Comparison of RNNs (Karpathy+ 2016) 41 (Karpathy+ 2016)  Task: character-level language modeling (predicting subsequent characters)  LSTMs and GRUs significantly outperform RNNs  RNNs seem to learn different embeddings from those of LSTMs and GRUs A Karpathy, J Johnson, and L Fei-Fei. 2016. Visualizing and Understanding Recurrent Networks. Proc. of ICLR Workshop 2016.

Observing LSTM cells (Karpathy+ 2016) 42 (Karpathy+ 2016) A Karpathy, J Johnson, and L Fei-Fei. 2016. Visualizing and Understanding Recurrent Networks. Proc. of ICLR Workshop 2016.

RNNs over tree 43

Recursive Neural Network (Socher+ 2011) 44 R Socher, J Pennington, E Huang, A Ng, and C Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. Proc. of EMNLP, pp. 151-161. movie good very ( × 2) ・ ・ very good very good movie ( × 2)  Compose a phrase vector = , =  , ∈ ℝ: constituent vectors  ∈ ℝ: phrase vector  ∈ ℝ×2: parameter  : activation function  Recursively compose vectors along the phrase structure (parse tree) of a sentence

Matrix-Vector Recursive Neural Network (MV-RNN) (Socher+ 2012) 45  Each word has a semantic vector and composition matrix  Compose a phrase vector and matrix recursively = , , = , = [; ] = , = [; ] R Socher, B Huval, C Manning and A Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. Proc. of EMNLP, pp. 1201-1211.

Recursive Neural Tensor Network (Socher+ 2013) 46  MV-RNN has too many parameters to train, assigning every word with a composition matrix  Transform a word vector into a composition matrix by using a tensor R Socher, A Perelygin, J Wu, J Chuang, C Manning, A Ng and C Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. Proc. of EMNLP, pp. 1631-1642.

Tree-structured LSTM (Tai+ 2015) 47 https://pdfs.semanticscholar.org/bd19/c394931257c1901a940ba8388366c35a3e33.pdf K S Tai, R Socher, C D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proc. of ACL- IJCNLP, pp. 1556–1566.

Stanford Sentiment Treebank (Socher+ 2013) 48 Movie reviews are parsed into phrase structures. Each node in a parse tree has a sentiment value (--, -, 0, +, ++) assigned by three annotators. R Socher, A Perelygin, J Wu, J Chuang, C Manning, A Ng and C Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. Proc. of EMNLP, pp. 1631-1642.

Comparison on Stanford Sentiment Treebank (Tai+ 2015) 49 K S Tai, R Socher, C D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proc. of ACL- IJCNLP, pp. 1556–1566.

Convolutional Neural Networks (CNNs) for Text 50

Convolutional Neural Network (CNN) (Kim 2014) 51 Y Kim. 2014. Convolutional neural networks for sentence classification. Proc. of EMNLP, pp. 1746-1751. It is a very good movie indeed :+ ・ ・ ・ ・ ・ ・ = max 1<<−+1 , Max pooling: each dimension is the maximum number of the values , over timesteps softmax (𝑦𝑦) ☺

Various pooling operations (Kalchbrenner+ 2014) 52 N Kalchbrenner, E Grefenstette, P Blunsom. 2014. A convolutional neural network for modelling sentences. Proc. of ACL, pp. 655-665.  Max pooling = max 1<<−+1 ,  Average pooling = 1 − + 1 � =1 −+1 ,  -max pooling  Taking the -max values (instead of 1-max)  Dynamic -max pooling  Change the value of adaptively based on the length () of an input

Hierarchical CNN includes Recursive NN 53 The movie was the best of all (1) (2) (3) (4) (5) (6)

Hierarchical CNN includes Recursive NN 54 The movie was the best of all (1) (2) (3) (4) (5) (6) PP NP VP NP S

Hierarchical CNN (AdaSent) (Zhao+ 2015) 55 The movie was the best of all (1) (2) (3) (4) Max Pooling Use these vectors (e.g., concatenation of these vectors) as the input to the fully-connected layer for classification H Zhao, Z Lu, P Poupart. 2015. Self-Adaptive Hierarchical Sentence Model. Proc. of IJCAI, pp. 4069-4076.

Summary  Various NN architectures that can leverage structures  Recurrent Neural Networks (RNNs)  Long Short-Term Memories (LSTMs)  Gated Recurrent Units (GRU)  Recursive Neural Networks (Recursive NNs)  Convolutional Neural Networks (CNNs)  Next question Can we generate a sentence from neural networks? 56