Skimming the Surface of Deep Learning

Presented at Zipfian Academy as part of tech talk.

Transcript

1. Skimming the Surface of Deep Learning Melanie Warrick 2/17/14 nyghtowl.io @nyghtowl
2. What is it? • ~ Machine Learning on hyperdrive and AI reboot • Tool of machine learning and AI • Learning algorithms model high level abstraction • Statistical neural networks in multi-layered hierarchical structure • Term officially around since mid 2000’s as Geoff Hinton’s work gained recognition and results • Hinton is highly recognized and followed in DP @nyghtowl
3. History • ‘50s AI & neural nets dreamed about (ANNs) ◦ Rosenblatt built Perceptron (mechanical brain - blind to Xor) • ‘80s Hinton took up neural nets research ◦ Challenges = computers not fast or powerful enough ◦ 20+ years continued work despite lack of interest • ‘90s Support Vector Machines & Linear Classifiers mimic organic interactions • 2004 Hinton started NCAP to pull together top researchers for deep learning • 2006 Hinton proved out successful neural net structure • 2011+ Major organizations started hiring NCAP members ◦ Hinton & Ng at Google ◦ LeCun at Facebook ◦ Sejnowski on US BRAIN Initiative @nyghtowl
4. 2006 Hinton’s Proof Showed many-layered feedforward neural network could effectively pre-train one layer at a time treating each one as an unrestrictive Boltzmann machine and then using supervised backpropagation for fine tuning following slides unpack... Published “A Fast Learning Algorithm for Deep Belief Nets.” @nyghtowl
5. Unpack > Neural Net Process Feedforward • Single-layer • Siloed inputs • Input / Output only • Only 1 direction Backpropagation • Multi-layer • Supervised training • Tweaks all previous weights • Updates neural weights based on output • Requires labeled training data Pre-train • Noises input • Procedure • Uses objective function to nudge weights / defines rules on nodes @nyghtowl
6. Unpack > Neural Nets Overview • Computer system modeled on brain and nervous system • Uses adaptive weights = numerical parameters tuned by learning algorithm and capable of approximating non-linear functions • More than one hidden / latent layer (for deep its many) Benefits • Handles complex questions • Fast and efficient • Scales well across growing # of machines Challenges • Logical inferences, abstractions & stories • Not always appropriate for the data • Overfitting @nyghtowl
7. Types of Deep Learning Neural Nets • Restricted Boltzmann Machine (RBM) ◦ Log-linear Markov Random Field ◦ Learn prob distribution from input ◦ Dimensionality reduction & feature discovery • Denoising Autoencoder ◦ Learn compressed distributed representation ◦ Dimensionality reduction & feature discovery ◦ Narrow hidden layer • Deep Belief Networks ◦ Deep hierarchical representation of data ◦ Trained in greedy manner • Convolutional Network (CNN) ◦ Spatially local correlation used for local connection pattern ◦ Less connections than others & no loss in expressiveness Feedforward / Single Layer Types Backprop / Multi-Layer Types @nyghtowl
8. Popular Uses Natural Language Processing Computer Vision / Video Processing (CNN structures) Automatic Speech Recognition Music / Audio Signal Recognition @nyghtowl
9. Examples MNIST • Yann LeCun’s team built in 1989 • Backpropagation applied to recognize handwritten mail zip codes • 3 days to train system - GPUs improved this time • Solve by binarizing and grayscaling data • Dimensionality reduction - cleans up image as it learns Google Brain • Andrew Ng’s team built • Neural network recognizes higher-level concepts • Finds cats from watching unlabeled images • 2012 Recognizing 1/6th of the objects it was trained @nyghtowl
10. Neural Net = Brain? Originally modeled after brain... • No proof of variable synaptic strengths to encode degrees of certainty • RBM connects all visible units to hidden units vs. synapses shown to reduce when brain learns • No proof neurons fine tuned with supervised learning • Not a winner takes all result (highest hits wins and all else falls away) “Hinton has built a better ladder; but the better ladder doesn’t necessarily get you to the moon.” - Gary Marcus • Many levels of processing • Each level is learning features or representations at increasing level of abstraction Similarities Differences @nyghtowl
11. • Adam Gibson • Thomson Nguyen • Wiki • Deeplearning.net (esp. tutorial section) • “Is Deep Learning a Revolution in artificial Intelligence?” http://www.newyorker. com/online/blogs/newsdesk/2012/11/is-deep-learning-a-revolution-in-artificial-intelligence.html • “Meet the Man Google Hired to Make AI a Reality http://www.wired. com/wiredenterprise/2014/01/geoffrey-hinton-deep-learning • “Deep Learning: Introduction” http://www.hlt.utdallas.edu/~vgogate/seminar/2013f/lectures/intro-deep- learning.pdf • “Machine Perception with Neural Networks” (Strata 2014 Conference) http://cdn.oreillystatic. com/en/assets/1/event/105/Neural%20Networks%20for%20Machine%20Perception%20Presentation. pdf • “Deep Neural Networks” http://www.slideshare.net/hammawan/deep-neural-networks • “Practical Guide to Restricted Boltzmann Machines” Resources @nyghtowl
12. Skimming the Surface of Deep Learning - Melanie Warrick @nyghtowl