Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Skimming the Surface of Deep Learning

Skimming the Surface of Deep Learning

Presented at Zipfian Academy as part of tech talk.

2168aa4564112d3ba88869ca3cc994b3?s=128

Melanie Warrick

February 17, 2014
Tweet

Transcript

  1. Skimming the Surface of Deep Learning Melanie Warrick 2/17/14 nyghtowl.io

    @nyghtowl
  2. What is it? • ~ Machine Learning on hyperdrive and

    AI reboot • Tool of machine learning and AI • Learning algorithms model high level abstraction • Statistical neural networks in multi-layered hierarchical structure • Term officially around since mid 2000’s as Geoff Hinton’s work gained recognition and results • Hinton is highly recognized and followed in DP @nyghtowl
  3. History • ‘50s AI & neural nets dreamed about (ANNs)

    ◦ Rosenblatt built Perceptron (mechanical brain - blind to Xor) • ‘80s Hinton took up neural nets research ◦ Challenges = computers not fast or powerful enough ◦ 20+ years continued work despite lack of interest • ‘90s Support Vector Machines & Linear Classifiers mimic organic interactions • 2004 Hinton started NCAP to pull together top researchers for deep learning • 2006 Hinton proved out successful neural net structure • 2011+ Major organizations started hiring NCAP members ◦ Hinton & Ng at Google ◦ LeCun at Facebook ◦ Sejnowski on US BRAIN Initiative @nyghtowl
  4. 2006 Hinton’s Proof Showed many-layered feedforward neural network could effectively

    pre-train one layer at a time treating each one as an unrestrictive Boltzmann machine and then using supervised backpropagation for fine tuning following slides unpack... Published “A Fast Learning Algorithm for Deep Belief Nets.” @nyghtowl
  5. Unpack > Neural Net Process Feedforward • Single-layer • Siloed

    inputs • Input / Output only • Only 1 direction Backpropagation • Multi-layer • Supervised training • Tweaks all previous weights • Updates neural weights based on output • Requires labeled training data Pre-train • Noises input • Procedure • Uses objective function to nudge weights / defines rules on nodes @nyghtowl
  6. Unpack > Neural Nets Overview • Computer system modeled on

    brain and nervous system • Uses adaptive weights = numerical parameters tuned by learning algorithm and capable of approximating non-linear functions • More than one hidden / latent layer (for deep its many) Benefits • Handles complex questions • Fast and efficient • Scales well across growing # of machines Challenges • Logical inferences, abstractions & stories • Not always appropriate for the data • Overfitting @nyghtowl
  7. Types of Deep Learning Neural Nets • Restricted Boltzmann Machine

    (RBM) ◦ Log-linear Markov Random Field ◦ Learn prob distribution from input ◦ Dimensionality reduction & feature discovery • Denoising Autoencoder ◦ Learn compressed distributed representation ◦ Dimensionality reduction & feature discovery ◦ Narrow hidden layer • Deep Belief Networks ◦ Deep hierarchical representation of data ◦ Trained in greedy manner • Convolutional Network (CNN) ◦ Spatially local correlation used for local connection pattern ◦ Less connections than others & no loss in expressiveness Feedforward / Single Layer Types Backprop / Multi-Layer Types @nyghtowl
  8. Popular Uses Natural Language Processing Computer Vision / Video Processing

    (CNN structures) Automatic Speech Recognition Music / Audio Signal Recognition @nyghtowl
  9. Examples MNIST • Yann LeCun’s team built in 1989 •

    Backpropagation applied to recognize handwritten mail zip codes • 3 days to train system - GPUs improved this time • Solve by binarizing and grayscaling data • Dimensionality reduction - cleans up image as it learns Google Brain • Andrew Ng’s team built • Neural network recognizes higher-level concepts • Finds cats from watching unlabeled images • 2012 Recognizing 1/6th of the objects it was trained @nyghtowl
  10. Neural Net = Brain? Originally modeled after brain... • No

    proof of variable synaptic strengths to encode degrees of certainty • RBM connects all visible units to hidden units vs. synapses shown to reduce when brain learns • No proof neurons fine tuned with supervised learning • Not a winner takes all result (highest hits wins and all else falls away) “Hinton has built a better ladder; but the better ladder doesn’t necessarily get you to the moon.” - Gary Marcus • Many levels of processing • Each level is learning features or representations at increasing level of abstraction Similarities Differences @nyghtowl
  11. • Adam Gibson • Thomson Nguyen • Wiki • Deeplearning.net

    (esp. tutorial section) • “Is Deep Learning a Revolution in artificial Intelligence?” http://www.newyorker. com/online/blogs/newsdesk/2012/11/is-deep-learning-a-revolution-in-artificial-intelligence.html • “Meet the Man Google Hired to Make AI a Reality http://www.wired. com/wiredenterprise/2014/01/geoffrey-hinton-deep-learning • “Deep Learning: Introduction” http://www.hlt.utdallas.edu/~vgogate/seminar/2013f/lectures/intro-deep- learning.pdf • “Machine Perception with Neural Networks” (Strata 2014 Conference) http://cdn.oreillystatic. com/en/assets/1/event/105/Neural%20Networks%20for%20Machine%20Perception%20Presentation. pdf • “Deep Neural Networks” http://www.slideshare.net/hammawan/deep-neural-networks • “Practical Guide to Restricted Boltzmann Machines” Resources @nyghtowl
  12. Skimming the Surface of Deep Learning - Melanie Warrick @nyghtowl