Character level LSTM Recurrent Neural Networks for language and music modeling Δημήτρης Σπαθής Εξαμηνιαία εργασία – 05 / 2016 Ανάλυση & περιγραφή πολυμεσικών δεδομένων Καθ.: Αναστάσιος Τέφας 

Eισαγωγή Η ανάπτυξη στην έρευνα των deep neural networks τα τελευταία χρόνια, σε συνδυασμό με τη χρήση των GPU και την πρόσβαση σε μεγάλα δεδομένα εκπαίδευσης, έχουν οδηγήσει σε νέα state-of-the-art αποτελέσματα σε πολλά δύσκολα προβλήματα εικόνας και ακολουθιών. Η παρούσα δουλειά εστιάζει στα Recurrent Neural Networks μακράς βραχυπρόθεσμης μνήμης (LSTM) για τη μοντελοποίηση γλώσσας και μουσικής. 

“we cut our transcription errors by 49%” SPEECH RECOGNITION 

“the LSTM did not have difficulty on long sentences” MACHINE TRANSLATION 

char-RNN GENERATIVE MODELS 

Tα Recurrent Neural Networks διαθέτουν βρόγχους 

To πρόβλημα των long term dependencies 

Ένα απλό RNN class RNN: def step(self, x): # update the hidden state self.h = np.tanh(np.dot(self.W_hh, self.h) + np.dot(self.W_xh, x)) # compute the output vector y = np.dot(self.W_hy, self.h) return y 

Oι πράξεις στο εσωτερικό ενός RNN 

RNN LSTM 

Cell state & gates 

Forget gate layer 

Input gate layer 

Update cell state 

Filtered output 

“none of the variants can improve upon the standard LSTM architecture” LSTM VARIANTS ANALYSIS 

Πειράματα 

Βιβλιοθήκες & frameworks ΠΕΡΙΒΑΛΛΟΝ Python python.org Tensorflow tensorflow.org Docker docker.com LIBRARIES Numpy import numpy 

Datasets για εκπαίδευση DATA SIZE TRAINING TIME TRAINING LOSS English folk music 14 MB 12 MB 27 hrs 92 hrs 0.6 0.85 Shakespeare 4.4 MB 2 hrs 1.2 Game of Thrones books 10.3 MB 8 hrs 1.1 

Training output 8600/33992 (epoch 2), train_loss = 1.209, time/batch = 2.264 8601/33992 (epoch 2), train_loss = 1.189, time/batch = 2.205 8602/33992 (epoch 2), train_loss = 1.198, time/batch = 2.482 8603/33992 (epoch 2), train_loss = 1.276, time/batch = 2.410 8604/33992 (epoch 2), train_loss = 1.213, time/batch = 2.367 8605/33992 (epoch 2), train_loss = 1.193, time/batch = 2.264 8606/33992 (epoch 2), train_loss = 1.218, time/batch = 2.291 8607/33992 (epoch 2), train_loss = 1.208, time/batch = 2.323 8608/33992 (epoch 2), train_loss = 1.195, time/batch = 2.336 8609/33992 (epoch 2), train_loss = 1.156, time/batch = 2.378 8610/33992 (epoch 2), train_loss = 1.236, time/batch = 2.468 8611/33992 (epoch 2), train_loss = 1.193, time/batch = 2.214 8612/33992 (epoch 2), train_loss = 1.222, time/batch = 2.368 8613/33992 (epoch 2), train_loss = 1.241, time/batch = 2.595 8614/33992 (epoch 2), train_loss = 1.208, time/batch = 2.730 8615/33992 (epoch 2), train_loss = 1.188, time/batch = 2.571 

Συμβολική αναπαράσταση μουσικής — ABC NOTATION T:Milo mou kokkino M:7/8 L:1/8 K:C P:A y("C"C/D/E)E ED DC | ("F"D/E/F)F FE ED | ("C"C/D/E)E ED DC | "G"CB,A, G,4 | ("C"[C/E/][D/F/][EG])[EG] [EG][DF] [DF][CE] | ("F"[D/F/][E/G/][FA])[FA] [FA][EG] [EG][DF] | ("C"[C/E/][D/F/][EG])[EG] [EG][DF] [DF][CE] | [C2E2][DF] ("G"[E/G/][D/F/][C/E/][D/F/] "C"[C2E2]) || P:B |: "C"C3 C2 C2 | E2F G2 GF | "F"A3 A2 A2 |1 "C"PG2F E>F ED :|2 "C"PG2F E4 | |: "C"G3 AG FE | "F"PF2E D2 CD | "G"E3 FE DC |1 "C"E2D CDEF :|2 "C"E2D C4 | 

Συμβολική αναπαράσταση μουσικής — ABC NOTATION T:Milo mou kokkino M:7/8 L:1/8 K:C P:A y("C"C/D/E)E ED DC | ("F"D/E/F)F FE ED | ("C"C/D/E)E ED DC | "G"CB,A, G,4 | ("C"[C/E/][D/F/][EG])[EG] [EG][DF] [DF][CE] | ("F"[D/F/][E/G/][FA])[FA] [FA][EG] [EG][DF] | ("C"[C/E/][D/F/][EG])[EG] [EG][DF] [DF][CE] | [C2E2][DF] ("G"[E/G/][D/F/][C/E/][D/F/] "C"[C2E2]) || P:B |: "C"C3 C2 C2 | E2F G2 GF | "F"A3 A2 A2 |1 "C"PG2F E>F ED :|2 "C"PG2F E4 | |: "C"G3 AG FE | "F"PF2E D2 CD | "G"E3 FE DC |1 "C"E2D CDEF :|2 "C"E2D C4 | 

Folk sampling 1 — Normalized dataset 

Folk sampling 2 — Normalized dataset 

Folk sampling INFINITE LOOP — Normalized dataset 

Folk sampling 3 — Raw dataset 

Jukedeck – Cambridge University spinoff — $3.8M Funding 

Shakespeare sampling 1 him to merquess, see I have night, Whom hast! I do nerless could all found, the head, Bro'llo, to have done of himself and be death I'll come, come up that make him; Thou lives of a mansead oft help! BENVOLIO: Go, knong myself. JOHN OF GAUNT: All, 'twere need of that wime? ISABELLA: O, that met's gabe it of there; Wite that cold from him, eaching a hands; For little dineming to ten accused! CAPULET: Here'st you must stuff'd friends, and mine enember; found! What yields him battle kings; No o 

Shakespeare sampling 2 WARWICK: And reveist is you have like by head's haste. CLIFFORD: Good of you reportion, to do; speak guide; He not to between Wide be sense my bird. Malale. Who York, and dowry all; Your bejurny is two men. KING RICHARD III: Why, what upon me, I store, made to countenarch! Though not England, by yourself? I darrant thou idst them advice. Nurse: Come to dropp-none, Margaret, we'll know. Sir I have brief, my all me, and of this: and heinoul hit you; yes well had in my budian's 

Game of Thrones sampling 1 — Infinite Loop the sound of the castle was a short of the castle and the steps of the castle was a short of the castle and the steps of the castle was a short of the castle and the steps of the castle was a short of the castle 

Game of Thrones sampling 2 It might use no time her and the king was close no sister in your sister, and better than the song of some golden days have a children of the man who ever like the man and for a king king in the armory of the same of Robert Eddard I know I wanted up in wine. The Wall remembered it to get before the red man grown cloak and king to see you to read me some of the horse in the day he began her with the jars were more than she knew for one and you like the gods be done in the starter t 

Game of Thrones sampling 3 winter is very for his hand. A bath was gone to one can part with Jon had seemed to the day the steps Martell Ser Hyle had been for the world like the Seven Greyjoy said of the Great Selmy and her brother of a woman I promise of men and part of Old Tower Tower and the black hand and not some down on his hand and a man king for part of some for Maester Men, children so hard to be a pair and blood and cloak me her brother made him for a woman in one the horse down his dark and warm and made the red han 

Game of Thrones sampling 3 who will sit at the iron throne? I fear with a golden men and the end of your House say no queen has you with me and blood king me in the courage.I want to it to prove Bran Snow.s words and said the steps in the fire. A few grey had been the courage was when when Father said no lord in the part of the steps he could want to be just some so she had a chain red and cloak and the steps and said it was the th 

Further work 

“long-range dependencies such as line lengths, quotes and brackets.” UNDERSTANDING ACTIVATIONS 

No content

“automatically learned to fix its gaze on salient objects while generating the corresponding words.” CNN + RNN = ATTENTION 

No content

No content

References Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2), 157-166. Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.6392. Eck, D., & Schmidhuber, J. (2002). Finding temporal structure in music: Blues improvisation with LSTM recurrent networks. In Neural Networks for Signal Processing, 2002. Proceedings of the 2002 12th IEEE Workshop on (pp. 747- 756). IEEE. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. Sturm Bob L., J. Santos, I. Korshunova. (2015) Folk Music Style Modelling by Recurrent Neural Networks with Long Short Term Memory Units. 16th International Society for Music Information Retrieval Conference.