Neural Nets for Newbies

Neural Nets for Newbies

Neural networks have regained popularity in the last decade, but they get dismissed as being too complicated to understand and implement. This talk breaks down the neural net structure as simply as possible, so you have a framework on which to grow your knowledge in the space. The presentation puts neural nets in the context of real-world applications and shares Python packages and code where you can get started building your own. You won't know everything about neural nets, but you will gain a solid foundation and some resources on where to go next to learn more.

2168aa4564112d3ba88869ca3cc994b3?s=128

Melanie Warrick

April 10, 2015
Tweet

Transcript

  1. Neural Nets for Newbies NN Primer Melanie Warrick nyghtowl.io @nyghtowl

  2. • History • Math & Structure • System Architecture •

    Code • Deep Learning • References Overview @nyghtowl
  3. Artificial Neural Networks • Machine Learning Algorithm • Classification &

    prediction @nyghtowl
  4. Brief Backstory @nyghtowl ‘50s ANNs - “Thinking Machines” ‘90s -

    MNIST ‘06 - Deep Learning Computational Power
  5. Real World ... • Natural Language Processing (Search & Sentiment)

    • Speech Recognition (Siri) • Computer Vision & Facial Recognition (Canon) • Robotics (Google & Tesla Car) • Recommender systems (Netflix) @nyghtowl
  6. Solves Complex Problems accuracy data Neural Nets Other ML Algorithms

    * Andrew Ng’s example @nyghtowl
  7. @nyghtowl Example: Model / Equation y = mx + b

    University GPA High School GPA Linear Regression Model Example coefficients
  8. Model = activation func( sum( input * weight ) )

    > threshold @nyghtowl Basic NN Equation X0 - bias X1 Xn k y j X kj W M Processing Unit f(y) Activation Functions • sigmoid • rectified linear • softmax • binary step
  9. @nyghtowl Example: Attend a Talk? Speaker Time y speaker *

    .3 topic * .5 time *.2 Topic 30% 50% 20% f(y) M threshold 45% Inputs Weights
  10. Basic NN Structure Input = data / features Hidden =

    processors Output = result Parameters are tricky & unique to the problem @nyghtowl
  11. Multi-layer Structure Input Layer 2 Layer 3 Layer 4 Output

    Hidden @nyghtowl
  12. Ada Lovelace? Example: Computer Vision Layers Pixels Edges Object Parts

    Object Models Layer 2 Layer 3 Layer 4 Input @nyghtowl
  13. Cross Validation: Labeled Data Test Data Training Data Rachel Olivia

    @nyghtowl
  14. Goal: minimize error Supervised Loss Function bias input n f(

    ) Loss Functions • mean squared error • negative log likelihood • cross entropy input 1 Error prediction - actual label @nyghtowl Output M Rachel
  15. Optimization Method ~ Back-prop @nyghtowl Input Output Hidden Run until

    error stops improving = converge Loss Function
  16. accuracy? Validation - Evaluate Test Data @nyghtowl Olivia Test Data

  17. Recurrent Nets (RNN) Sample NN Types @nyghtowl Feed Forward Nets

    (FNN) Convolutional Nets (CNN)
  18. Systems Understanding Focus • Graphical Processing Units (GPUs) ◦ Gaming

    ◦ AWS Additional Areas to Understand • General systems architecture • Parallelism • Distributed Systems • High Performance Computing (HPC & MPIs) @nyghtowl
  19. Python Packages Basics & Deep Math GUI & Details Masked

    @nyghtowl • Theano • Lasagne (CNN) • PyLearn2 (CNN) • Kayak (CNN) • Blocks (RNN) • OpenDeep (RNN, CNN) • PyBrain (RNN, CNN) • Keras (RNN, CNN) • MetaMind (RNN, CNN) • Graphlab (CNN) • Sklearn (RBM) • Caffe (CNN) • CXXNet (CNN) • FANN2
  20. Example: MNIST ~ “Hello World” • Classify handwritten digits 0-9

    • Each pixel is an input • Input value ranges 0-255 (white to black) @nyghtowl
  21. 0 0 1 0 0 0 0 1 1 0

    0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 Example: Input @nyghtowl
  22. Example: MNIST Structure Input Output Hidden ...784 Nodes ...1000+ Nodes

    7 layers 10 outputs 4 9 8 7 6 5 4 3 2 1 0 @nyghtowl
  23. @nyghtowl Code: Train Model import graphlab as gl filename =

    'http://s3.amazonaws.com/GraphLab-Datasets/mnist/sframe/train' inputs = gl.SFrame(filename) # Train neural net MNIST model model = gl.neuralnet_classifier.create(inputs, target=’label’)
  24. Example: Layer Features @nyghtowl * Yann LeCunn Input Hidden Output

  25. @nyghtowl # Ratio of number of correct predictions print ‘Accuracy’,

    model.evaluate(test_data) Accuracy: 0.97 Code: Validation
  26. MNIST Repository github.com/nyghtowl/Neural_Net_Newbies @nyghtowl

  27. Deep Learning - Bottom Line Automatic feature detection / Pre-train

    @nyghtowl
  28. • Nature of Code: Neural Networks http://natureofcode.com/book/chapter-10-neural-networks/ • Neural Nets

    and Deep Learning http://neuralnetworksanddeeplearning.com/ • Neural Nets Demystified (Welch) http://www.welchlabs.com/blog/ • Theano Tutorial http://deeplearning.net/software/theano/tutorial/index.html#tutorial • “The State of Deep Learning in 2014” https://speakerdeck.com/ogrisel/the-state-of-deep- learning-in-2014 • “Hacker’s Guide to Neural Nets” https://karpathy.github.io/neuralnets/ • “Automated Image Capturing with ConvNets and Recurrent Nets” (Karpathy) http://cs.stanford. edu/people/karpathy/sfmltalk.pdf • Deeplearning.net • Wikipedia.org Where to go next... @nyghtowl
  29. References: Individuals & Groups Individuals • Adam Gibson (Skymind) •

    Tarin Ziyaee (Apple) • Bryan Catanzaro (Baidu Research) • Fernand Pajot (Notion Labs) • Paco Nathan (Databricks) • Markus Beissinger (OpenDeep) Groups & Research • Deeplearning.net (esp. tutorial section) • Neural Computation & Adaptive Perception Research (research group) http://www.cifar.ca/neural- computation-and-adaptive-perception-research-progress • http://www.jeremyfreeman.net/ • Wikipedia.org @nyghtowl
  30. • Nature of Code: Neural Networks http://natureofcode.com/book/chapter-10-neural-networks/ • Theano Tutorial

    http://deeplearning.net/software/theano/tutorial/index.html#tutorial • Machine Learning (Coursera-Ng) https://class.coursera.org/ml-005/lecture • Hacker’s Guide to Neural Nets (Stanford - Karpathy) https://karpathy.github.io/neuralnets/ • Neural Networks for Machine Learning (Coursera - Hinton) https://class.coursera.org/neuralnets-2012- 001/lecture • Neural Nets and Deep Learning http://neuralnetworksanddeeplearning.com/ • Deep Learning Stanford CS http://deeplearning.stanford.edu/ • Deep Learning Tutorial (NYU - LeCun) http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf • Deep Learning Tutorial (U Montreal - Bengio) http://deeplearning.net/tutorial/deeplearning.pdf • Tutorial on Deep Learning for Vision https://sites.google.com/site/deeplearningcvpr2014/ • ipam Graduate Summer School: Deep Learning, Feature Learning http://www.ipam.ucla. edu/programs/summer-schools/graduate-summer-school-deep-learning-feature-learning/?tab=schedule References: Tutorials @nyghtowl
  31. • “ANN Network Basics” https://en.wikibooks.org/wiki/Artificial_Neural_Networks/Neural_Network_Basics • “The State of Deep

    Learning in 2014” https://speakerdeck.com/ogrisel/the-state-of-deep-learning-in- 2014 • “Automated Image Capturing with ConvNets and Recurrent Nets” http://cs.stanford. edu/people/karpathy/sfmltalk.pdf • “Convolutional Neural Net” http://yann.lecun.com/exdb/publis/psgz/lecun-iscas-10.ps.gz • “Understanding Convolutions” https://colah.github.io/posts/2014-07-Understanding-Convolutions/ • “Hot Yet Little-Known Trend That’ll Supercharge AI” http://www.wired.com/2015/02/hot-yet-little-known- trend-thatll-supercharge-ai/ • “Show and Tell: A Neural Image Caption Generator” http://arxiv.org/pdf/1411.4555.pdf? • “Explain Images with Multimodal Recurrent Neural Networks” http://arxiv.org/pdf/1410.1090v1.pdf References: Articles & Publications @nyghtowl
  32. References: Video/Audio • Neural Nets Demystified (Welch) http://www.welchlabs.com/blog/ • Recent

    Developments in Deep Learning (Hinton) http://youtu.be/VdIURAu1-aU • RSS2014: Deep Learning (Ng) http://youtu.be/W15K9PegQt0 • Deep Learning, Self-Taught Learning and Unsupervised Feature Learning (Ng) http://youtu. be/n1ViNeWhC24 • Deep Learning on GPU Clusters (Catanzaro) http://youtu.be/brui4N2orlI • Automated Image Captioning with ConvNets and Recurrent Nets (Karpathy) http://youtu. be/ZkY7fAoaNcg • Deep Learning for NLP http://techtalks.tv/talks/deep-learning-for-nlp-without-magic-part-1/58414/ • Deep Learning: Machine Perception and Its Applications https://www.youtube.com/watch? v=hykoKDl1AtE • Talking Machines: The History of Machine Learning from Inside Out (12 min in) http://www. thetalkingmachines.com/blog/2015/2/26/the-history-of-machine-learning-from-the-inside-out? lang=en&utm_campaign=SendToFriend&uid=15226420&utm_content=item_link&utm_source=email&ut m_medium=self&china_variant=False • How to make a neural network in your bedroom http://youtu.be/n-YbJi4EPxc • Who is Afraid of Non-Convex Loss Functions http://videolectures.net/eml07_lecun_wia/ @nyghtowl
  33. References: Slides • “Deep Learning Applications & Neural Nets“ http://www.slideshare.net/hammawan/deep-neural-

    networks • “Machine Perception with Neural Networks” (Strata 2014 Conference) http://cdn.oreillystatic. com/en/assets/1/event/105/Neural%20Networks%20for%20Machine%20Perception%20Presentation. pdf • “Theano and LSTM for Sentiment Analysis” https://drive.google. com/file/d/0B8qS82bey7OmbnhaRHdkN0d4bEU/view • “Systems for Deep Learning: A View from the Trenches” https://dl.dropboxusercontent. com/u/21184584/bcatanzaro_fall_2014_sigai.pdf • “Deep Neural Networks” http://www.slideshare.net/hammawan/deep-neural-networks • “Deep Learning: Intro” http://www.hlt.utdallas.edu/~vgogate/seminar/2013f/lectures/intro-deep-learning. pdf • “Recurrent Neural Networks for Text Analysis” https://docs.google. com/presentation/d/1HYfUZLRZRJovQpv5mYxox9bz9erxj7Ak_ZovENMvM90/edit#slide=id.p @nyghtowl
  34. References: Images • http://code-spot.co.za/2009/10/08/15-steps-to-implemented-a-neural-net/ • http://www.texample.net/tikz/examples/neural-network/ • http://onlinestatbook.com/2/regression/intro.html • http://jaoying-google.blogspot.com/2012_12_01_archive.html

    • https://www.kaggle.com/forums/f/15/kaggle-forum/t/10878/feature-representation-in-deep-learning • http://maggieholmes.typepad.com/my_weblog/2011/07/scrapbooking.html • http://www.teds.com.au/photolounge/why-print-photos • http://users.clas.ufl.edu/glue/longman/1/einstein.html • https://en.wikipedia.org/wiki/File:Feed_forward_neural_net.gif • http://www.scienzagiovane.unibo.it/English/artint/8-neural-networks.html • http://www.clarifai.com/technology • http://www.nvidia.com/object/what-is-gpu-computing.html • https://www.classes.cs.uchicago.edu/archive/2013/spring/12300-1/pa/pa1/ • http://yann.lecun.com/exdb/lenet/ • http://people.csail.mit.edu/torralba/research/drawCNN/drawNet.html(CNN Features) • http://vis-www.cs.umass.edu/lfw/ (LFW)
  35. Next for NN Near-term: Feature Engineering Combining Models Natural Language

    Processing Image Analysis Longer-term: Unlabeled data - finding patterns Generative (producing new data with NN) @nyghtowl
  36. Neural Nets for Newbies - Melanie Warrick nyghtowl.io (slides) github.com/nyghtowl/Neural_Net_Newbies

    (code) @nyghtowl
  37. Special Thanks Leon Barovic Lindsay Cade Jeremy Dunck Tim Elser

    Moon Limb Meggie Mahnken Isabel Markl Marissa Marquez Paul Minton Jason Morrison Chris Nicholson Erin O’Connell Tiia Priilaid Kelley Robinson Vijay Ramesh Shawn Scully Megan Speir Jana Thompson Andrew Weiss Python Software Foundation