Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Skimming the Surface of Deep Learning

Skimming the Surface of Deep Learning

Presented at Zipfian Academy as part of tech talk.


Melanie Warrick

February 17, 2014


  1. Skimming the Surface of Deep Learning Melanie Warrick 2/17/14 nyghtowl.io

  2. What is it? • ~ Machine Learning on hyperdrive and

    AI reboot • Tool of machine learning and AI • Learning algorithms model high level abstraction • Statistical neural networks in multi-layered hierarchical structure • Term officially around since mid 2000’s as Geoff Hinton’s work gained recognition and results • Hinton is highly recognized and followed in DP @nyghtowl
  3. History • ‘50s AI & neural nets dreamed about (ANNs)

    ◦ Rosenblatt built Perceptron (mechanical brain - blind to Xor) • ‘80s Hinton took up neural nets research ◦ Challenges = computers not fast or powerful enough ◦ 20+ years continued work despite lack of interest • ‘90s Support Vector Machines & Linear Classifiers mimic organic interactions • 2004 Hinton started NCAP to pull together top researchers for deep learning • 2006 Hinton proved out successful neural net structure • 2011+ Major organizations started hiring NCAP members ◦ Hinton & Ng at Google ◦ LeCun at Facebook ◦ Sejnowski on US BRAIN Initiative @nyghtowl
  4. 2006 Hinton’s Proof Showed many-layered feedforward neural network could effectively

    pre-train one layer at a time treating each one as an unrestrictive Boltzmann machine and then using supervised backpropagation for fine tuning following slides unpack... Published “A Fast Learning Algorithm for Deep Belief Nets.” @nyghtowl
  5. Unpack > Neural Net Process Feedforward • Single-layer • Siloed

    inputs • Input / Output only • Only 1 direction Backpropagation • Multi-layer • Supervised training • Tweaks all previous weights • Updates neural weights based on output • Requires labeled training data Pre-train • Noises input • Procedure • Uses objective function to nudge weights / defines rules on nodes @nyghtowl
  6. Unpack > Neural Nets Overview • Computer system modeled on

    brain and nervous system • Uses adaptive weights = numerical parameters tuned by learning algorithm and capable of approximating non-linear functions • More than one hidden / latent layer (for deep its many) Benefits • Handles complex questions • Fast and efficient • Scales well across growing # of machines Challenges • Logical inferences, abstractions & stories • Not always appropriate for the data • Overfitting @nyghtowl
  7. Types of Deep Learning Neural Nets • Restricted Boltzmann Machine

    (RBM) ◦ Log-linear Markov Random Field ◦ Learn prob distribution from input ◦ Dimensionality reduction & feature discovery • Denoising Autoencoder ◦ Learn compressed distributed representation ◦ Dimensionality reduction & feature discovery ◦ Narrow hidden layer • Deep Belief Networks ◦ Deep hierarchical representation of data ◦ Trained in greedy manner • Convolutional Network (CNN) ◦ Spatially local correlation used for local connection pattern ◦ Less connections than others & no loss in expressiveness Feedforward / Single Layer Types Backprop / Multi-Layer Types @nyghtowl
  8. Popular Uses Natural Language Processing Computer Vision / Video Processing

    (CNN structures) Automatic Speech Recognition Music / Audio Signal Recognition @nyghtowl
  9. Examples MNIST • Yann LeCun’s team built in 1989 •

    Backpropagation applied to recognize handwritten mail zip codes • 3 days to train system - GPUs improved this time • Solve by binarizing and grayscaling data • Dimensionality reduction - cleans up image as it learns Google Brain • Andrew Ng’s team built • Neural network recognizes higher-level concepts • Finds cats from watching unlabeled images • 2012 Recognizing 1/6th of the objects it was trained @nyghtowl
  10. Neural Net = Brain? Originally modeled after brain... • No

    proof of variable synaptic strengths to encode degrees of certainty • RBM connects all visible units to hidden units vs. synapses shown to reduce when brain learns • No proof neurons fine tuned with supervised learning • Not a winner takes all result (highest hits wins and all else falls away) “Hinton has built a better ladder; but the better ladder doesn’t necessarily get you to the moon.” - Gary Marcus • Many levels of processing • Each level is learning features or representations at increasing level of abstraction Similarities Differences @nyghtowl
  11. • Adam Gibson • Thomson Nguyen • Wiki • Deeplearning.net

    (esp. tutorial section) • “Is Deep Learning a Revolution in artificial Intelligence?” http://www.newyorker. com/online/blogs/newsdesk/2012/11/is-deep-learning-a-revolution-in-artificial-intelligence.html • “Meet the Man Google Hired to Make AI a Reality http://www.wired. com/wiredenterprise/2014/01/geoffrey-hinton-deep-learning • “Deep Learning: Introduction” http://www.hlt.utdallas.edu/~vgogate/seminar/2013f/lectures/intro-deep- learning.pdf • “Machine Perception with Neural Networks” (Strata 2014 Conference) http://cdn.oreillystatic. com/en/assets/1/event/105/Neural%20Networks%20for%20Machine%20Perception%20Presentation. pdf • “Deep Neural Networks” http://www.slideshare.net/hammawan/deep-neural-networks • “Practical Guide to Restricted Boltzmann Machines” Resources @nyghtowl
  12. Skimming the Surface of Deep Learning - Melanie Warrick @nyghtowl