Long Short-Term Memory (LSTM) models with TensorFlow | Sahil Dua

Long Short-Term Memory (LSTM) is a Recurrent Neural Network (RNN) architecture that looks at a sequence and remembers values over long intervals. LSTMs have been known to have achieved state of the art performance in many sequence classification problems. In this talk, I’ll cover how to write an LSTM using TensorFlow’s Python API for natural language understanding. This is going to be a code-heavy talk where I will implement the LSTM model and explain the math behind it step-by-step.

In short, it will cover:

– Understanding how the math behind LSTM architecture works in case of sequence classification
– Writing an LSTM model using TensorFlow for sentiment classification of variable length English language sentences
– Gotchas of using LSTMs on real data

Transcript

1. Long Short-Term Memory (LSTM) models with TensorFlow Sahil Dua (sahildua2305)
2. ➔ Developer at Booking.com ➔ Open Source Community Leader, DuckDuckGo ➔ Contributor: ◆ Git ◆ Pandas ◆ Go-GitHub ◆ Linguist ◆ etc… whoami
3. Agenda ➔ Deep Learning for NLP ➔ Word Representations ➔ How to handle sequences? ◆ LSTMs ➔ TensorFlow Implementation of LSTM
4. Natural Language Processing (NLP) ?
5. NLP Tasks ➔ Question Answering ➔ Sentiment Analysis ➔ Image to Text mapping ➔ Machine Translation ➔ Speech Recognition
6. Deep Learning for NLP?
7. Inputs to different types of models ➔ Convolutional Neural Networks ➔ Logistic Regression ➔ Reinforcement Learning
8. NLP Input Data Format
9. NLP Input Format Data (desired)
10. Word Embeddings
11. Word Embeddings
12. How to handle sequences?
13. Challenges in modelling sequences ➔ How to preserve order of words? ◆ The food was good, not bad at all. ◆ The food was bad, not good at all.
14. Challenges in modelling sequences ➔ How to deal with different word orders? ◆ On Monday, it was raining. ◆ It was raining on Monday.
15. Challenges in modelling sequences ➔ How to keep track of long term dependencies? ◆ In Berlin, I had a great time and I learnt some of the ________ language.
16. Challenges in modelling sequences ➔ How to deal with variable length of sequences? ◆ Awesome. ◆ Food was good. ◆ I didn’t like that place at all.
17. Feed-Forward Neural Network
18. Detecting activity in a movie scene
19. Ignoring the sequential nature ➔ Low-level patterns ➔ Complex patterns ➔ High-level patterns
22. Remembering information with RNN
23. Recurrent Neural Network (RNN) ➔ Use previous knowledge for prediction ➔ Store knowledge from current prediction
26. Problems with RNNs
27. Problems with RNNs ➔ Unable to keep long memories ➔ Chaotic memory update
28. Longer and more controlled Memories through LSTMs
29. How LSTM works? ➔ Forgetting mechanism ➔ Saving mechanism ➔ Using mechanism
30. How LSTM works?
33. Mathematically,
34. TensorFlow Implementation
35. Thank you! Sahil Dua (@sahildua2305)