Lorena Mesa - ¡Escuincla babosa!: Creating a telenovela script in three Python deep learning frameworks

Telenovelas are beloved for their over the top drama and intricate plot twists. In this talk, we’ll review popular telenovelas to synthesize a typical telenovela arc and use it to train a deep learning model.

What would a telenovela script look like as imagined by a neural network? To answer this question, we’ll examine three Python deep learning frameworks - Keras, PyTorch, and TensorFlow - to determine the process of translating a telenovela into a neural network and ultimately determine which one will be best for the task at hand. Be prepared for amor, pasiòn, and y el misterioso!

https://us.pycon.org/2019/schedule/presentation/199/

Transcript

1. ¡Escuincla babosa!: Creating a telenovela script in three Python deep learning frameworks Lorena Mesa @loooorenanicole SLIDES @ http:/ /bit.ly/lmesa-pycon2019
2. HOLA! Soy Lorena Mesa. I am here because I am really interested in deep learning and wanted to start deep diving into the topic more. Also, I love telenovelas. Win win? 2 @loooorenanicole
4. How many of you saw this? Enter the creativity of deep learning. SLIDES @ http:/ /bit.ly/lmesa-pycon2019
7. Why telenovelas? Also … what is a telenovela?
9. “The social phenomenon of televised melodramas, called telenovelas in Spanish … are serial melodramas … [based] on the construction of a relationship of fidelity with the audience” - Jorge Gonzalez, 2003, Understanding Telenovelas as a Cultural Front 9
10. 10 2 billion People watch telenovelas or approximately ⅓ of the global population (Telegraph - 2006)
11. The arc of a telenovela El drama es real
12. “Things have to be cleaned up so the audience has satisfaction. They won’t worry about Maria — did she find true love, her true mother or her true father?” - Dr. Diana Rios, Associate Professor of Communications, University of Connecticut 12
13. The arc of a telenovela Unlike English soap operas telenovelas are defined by: - A fixed melodramatic plot (e.g. love lost, mothers and daughters fighting, long-lost relatives, love found) - A finite beginning and end - A conclusion that ties up loose ends generally with a happy element (e.g. big wedding) 13
14. Examples of Telenovelas Spanish and English SLIDES @ http:/ /bit.ly/lmesa-pycon2019
17. Where does deep learning fit in? Text generation is a deep learning problem. SLIDES @ http:/ /bit.ly/lmesa-pycon2019
18. “A field of study that gives computers the ability to learn without being explicitly programmed” - Arthur Samuel, pioneer in Machine Learning (1959) 18
19. 19 Landscape of Machine Learning Source: https:/ /www.packtpub.com/mapt/book/big_data_and_business_intelligence/9781784399689/1/ch01lvl1sec08/machine-learning
20. “A computer program is said to learn from experience (E) with respect to some task (T) and some performance measure (P), if its performance on T, as measured by P, improves with experience E.” - Tom Mitchell , Machine Learning (1997) 20
21. 21 Source: https:/ /www.embedded-vision.com/industry-analysis/blog/artificial-intelligence-machine-learning-deep-learning-and-computer-visionwha
22. 22 Source: http:/ /datasciencegyan.com/artificial-intelligence-vs-machine-learning-vs-de ep-learning/ Source: https:/ /www.iflscience.com/technology/artificial-intelligence-dreams/
23. Deep Learning: Classification Programming a flashlight to turn on with audible cue “dark” A deep learning implementation can learn: - Phrases with word “dark” - Other cues: “I can’t see” Learns via the “brain”, the activation method 23
24. Deep Learning: Text Generation Sequence processing has applications in several areas Two model approaches: - Character model - Word model Learns from the text corpus provided the flavor and feel of the text to generate its own 24
25. Neural networks vs deep learning: What’s the difference? “The difference between a neural network and deep learning is that deep learning is the act of using a subset of neural networks called deep neural networks” - What is a Neural Network and How are Businesses Using Them, Erika Morphy (May 2018) 25
26. 26 Source: https:/ /hackernoon.com/a-hands-on-introduction-to-neural-networks-6a03afb468b1 What is a neuron?
27. Foundation of a neuron Neuron model: - inputs: Features represented as a weighted number represented as (x1 … xn) - Weights: Strength of the feature represented as (w1j .. wnj) - Outputs: Weighted sum of inputs passed (nj) through the activation function TLDR; a neuron is a mathematical function 27
28. The Activation Function Activation function models the “firing rate” of a biological neuron, converting weighted sum into a new number based on a formula. Training the neuron means: - The neuron takes an initial guess at classifying sample - Update the neuron’s feature weights with the activation function - Repeat 28
29. The many architectural types of Neural Networks Examples of types: - Feedforward - information flows only moves one way, from input layer to output layer - Radial basis function - Includes a distance criterion - Recurrent neural network - propagates data forward and backwards, from later processing stages back to earlier processing stages; it’s a directed graph! 29
30. 30 Source: http:/ /neuralnetworksanddeeplearning.com/chap5.html
31. Making an RNN in Python There’s more than one way to do it! SLIDES @ http:/ /bit.ly/lmesa-pycon2019
32. Python and Deep Learning Frameworks Python is an ideal language to use for scientific programming! Don’t believe me? See Jake Vanderplas “The Unexpected Success of Python in Science” PyCon 2017 keynote! Keras TensorFlow PyTorch 32
33. Which framework to choose? 33 What’s important for your use case? Consider these three guiding questions: 1. How much technical expertise is needed to start using the framework? 2. What are your requirements (e.g. speed, dataset size)? 3. How easy is the framework to start working with?
34. Which framework to choose? 34 How much technical expertise is needed to start using the framework? + Beginner with deep learning + Something that allows us to build a small toy project quickly + Has good developer community and demos
35. Which framework to choose? 35 What are your requirements (e.g. speed, dataset size)? + Working with < 100 megabytes of data (only 3 seasons Queen of the South, 4 seasons of Ugly Betty, 5 seasons Jane the Virgin) + Working with text data + Need something that has debugging functionality
36. Which framework to choose? 36 How easy is the framework to start working with? + Want something that doesn’t require a lot of setup + Has sensical debugging capacity + Has built in bootstrapping tools + Doesn’t require a lot of deep mathematical knowledge
37. Which framework to choose? 37 Keras - Can run on top of TensorFlow - High-level API - Slower vs TensorFlow & PyTorch - Prioritizes readability - Better suited for smaller datasets TensorFlow - Both high and low-level APIs - About same as PyTorch - Can use Keras thereby making architecture simpler - Good fit for larger datasets PyTorch - Low-level API - Works directly on array expressions - Good traction in academia - Same speed as TensorFlow - More complex architecture - Less readable - Good fit for larger datasets
38. Which framework to choose? 38 Keras - Can run on top of TensorFlow - High-level API - Slower vs TensorFlow & PyTorch - Prioritizes readability - Better suited for smaller datasets TensorFlow - Both high and low-level APIs - About same as PyTorch - Can use Keras thereby making architecture simpler - Good fit for larger datasets PyTorch - Low-level API - Works directly on array expressions - Good traction in academia - Same speed as TensorFlow - More complex architecture - Less readable - Good fit for larger datasets
39. Which framework to choose? 39 Keras - Can run on top of TensorFlow - High-level API - Slower vs TensorFlow & PyTorch - Prioritizes readability - Better suited for smaller datasets
40. Making an RNN in Python Framework #1: Keras SLIDES @ http:/ /bit.ly/lmesa-pycon2019
41. 41 Hot Encoding Allows Model to Predict the Probability of Each Character in the Vocabulary Jupyter Notebook here! Steps to creating an RNN Fit model during training epochs Generate character hot encodings Transform data to input sequences and rescale [0,1]
42. 42 Hot Encoding Allows Model to Predict the Probability of Each Character in the Vocabulary Character One-Hot Vector Encoding Corpus: [‘a’, ‘b’, ‘c’, ‘d, ‘e’, ‘f’, g’, ‘h’, ‘i’] — len(Corpus) = 9 Word: [‘bad’] → [‘b’, ‘a’, ‘d’] b → [0, 1, 0, 0, 0, 0, 0, 0, 0], a → [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], d → [0, 0, 0, 1, 0, 0, 0, 0, 0]
43. 43 Building an RNN Long Short Term Memory (LSTM) Model with Keras from keras.models import Sequential from keras.layers import Dense, Dropout, LSTM from keras.callbacks import ModelCheckpoint from keras.utils import np_utils model = Sequential() model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2]))) model.add(Dropout(0.2)) model.add(Dense(y.shape[1], activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='adam') model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)
44. 44 Optimizing Weights with Keras # pick a random seed pattern = dataX[numpy.random.randint(0, len(dataX)-1)] # generate characters for i in range(1000): x = numpy.reshape(pattern, (1, len(pattern), 1)) x = x / float(n_vocab) prediction = model.predict(x, verbose=0) index = numpy.argmax(prediction) result = int_to_char[index] seq_in = [int_to_char[value] for value in pattern] sys.stdout.write(result) pattern.append(index) pattern = pattern[1:len(pattern)]
45. Making an RNN in Python Framework #2: TensorFlow SLIDES @ http:/ /bit.ly/lmesa-pycon2019
46. 46 Building an RNN Long Short Term Memory (LSTM) Model with TensorFlow import tensorflow as tf from tensorflow.contrib import rnn def RNN(x, weights, biases): x = tf.reshape(x, [-1, n_input]) x = tf.split(x,n_input,1) # Generate n_input sequence of inputs rnn_cell = rnn.BasicLSTMCell(n_hidden) # Make prediction outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32) # We only want the last output of the n outputs return tf.matmul(outputs[-1], weights['out']) + biases['out']
47. 47 Optimizing Weights with TensorFlow with tf.Session() as session: session.run([init_op]) coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord) saver = tf.train.Saver() for epoch in range(num_epochs): for step in range(training_input.epoch_size): . . . logic for optimizing weights to use for text generation saver.save(session, data_path + '\\' + model_save_name, global_step=epoch) saver.save(session, data_path + '\\' + model_save_name + '-final') coord.request_stop() # close threads coord.join(threads)
48. Making an RNN in Python Framework #3: PyTorch SLIDES @ http:/ /bit.ly/lmesa-pycon2019
49. 49 Building an RNN Long Short Term Memory (LSTM) Model with PyTorch import torch from torch import nn class CharRNN(nn.Module): def __init__(self, input_size, hidden_size, output_size): self.lstm = nn.LSTM(len(self.chars), n_hidden, n_layers, dropout=drop_prob, batch_first=True) # Set attrs related to hidden, dropout, connected layers, etc. def forward(self, category, input, hidden): r_output, hidden = self.lstm(x, hidden) # Get outputs from hidden output = self.dropout(r_output) # Run through dropout layer out = out.contiguous().view(-1, self.n_hid) # Add LSTM outputs up out = self.fc(out) # Run through connect layer return output, hidden
50. 50 Optimizing Weights with PyTorch net.train() opt = torch.optim.Adam(net.parameters(), lr=lr) criterion = nn.CrossEntropyLoss() for e in range(epochs): h = net.init_hidden(batch_size) # initialize hidden state inputs, targets = torch.from_numpy(x), torch.from_numpy(y)# make torches h = tuple([each.data for each in h]) # New h to prevent backpropagation output, h = net(inputs, h) # Get output # Calculate loss, keep the output with optimal performance
51. Generating a telenovela “script” with an RNN What’s possible? SLIDES @ http:/ /bit.ly/lmesa-pycon2019
52. Sample Text Generated: “do not threaten me. do you understand me? do you understand me? whoa, whoa, all right? hey, look, i” 52
53. Approaches for creating telenovela deep learning scripts Improvements: - Make a LSTM model per character you want in your script - Find similar enough “sources of inspiration” per character - Weave all these models together? - Generate LSTM models for telenovelas of each type of melodrama (e.g. lost love) 53
54. Text generation is hard. Generating a “learned” plot may be possible, but without coherent narratives and emotionally compelling characters we miss the mark. 54
55. How can we do better? Telenovelas may have a format, but text generation is complex. SLIDES @ http:/ /bit.ly/lmesa-pycon2019
56. Varying Data Quality Queen of the South (2016) s01e01 Episode Script 1 (helicopter blades whirring) (upbeat music) TERESA: My name is Teresa Mendoza. I am from México. I was born poor, not that that's bad. But take it from me, I've been poor. And I've been rich. Rich is better. Believe me. 01x01 - Chapter One (Pilot) 01x01 - Chapter One (Pilot) Latin lover narrator: Our story begins 13 and a half years ago, when Jane Gloriana Villanueva was a mere ten years old. It should be noted that at a mere ten years old, Jane's passions include... in no particular order... her family, God, and grilled cheese sandwiches. This is Jane's grandmother, Alba Gloriana Villanueva. Her passions include God and Jane, in that particular order. Woman: Really, Mom? Shh. But this is so lame. 56
57. Let’s generate more data! Want to contribute to my telenovela sets? Please do! Add at http:/ /bit.ly/telenovelas-for-all! 57
58. 58 Source: https:/ /medium.freecodecamp.org/chihuahua-or-muffin-my-search-for-the-best-computer-vision-api-cbda4d6b425d
59. 59 Thwarting deep learning with CVZazzle and One Pixel Attacks.
60. Want to keep learning? Great! SLIDES @ http:/ /bit.ly/lmesa-pycon2019
61. Resources 61 Blogs + Articles - Develop your first NN in Python with Keras Step-by-Step A NN in 11 lines of Python - How to build a 3 layer neural network from scratch - What is the best alternative to Keras? Videos + MOOCs - 10.4 NN: Multilayer Perceptron Part 1: The Nature of Code - Exploring Deep Learning Framework in PyTorch - Stephanie Kim PyCon USA 2018 - Mathematics for Machine Learning: Linear Algebra - Fast.ai
62. ¡Gracias! You can find me at: @loooorenanicole (social) me@lorenamesa.com (email) 62 Also, if you own Maria la del Barrio, please find me!