Neural Nets for Newbies

Neural networks have regained popularity in the last decade, but they get dismissed as being too complicated to understand and implement. This talk breaks down the neural net structure as simply as possible, so you have a framework on which to grow your knowledge in the space. The presentation puts neural nets in the context of real-world applications and shares Python packages and code where you can get started building your own. You won't know everything about neural nets, but you will gain a solid foundation and some resources on where to go next to learn more.

Transcript

1. Neural Nets for Newbies NN Primer Melanie Warrick nyghtowl.io @nyghtowl
2. • History • Math & Structure • System Architecture • Code • Deep Learning • References Overview @nyghtowl
3. Artificial Neural Networks • Machine Learning Algorithm • Classification & prediction @nyghtowl
4. Brief Backstory @nyghtowl ‘50s ANNs - “Thinking Machines” ‘90s - MNIST ‘06 - Deep Learning Computational Power
5. Real World ... • Natural Language Processing (Search & Sentiment) • Speech Recognition (Siri) • Computer Vision & Facial Recognition (Canon) • Robotics (Google & Tesla Car) • Recommender systems (Netflix) @nyghtowl
6. Solves Complex Problems accuracy data Neural Nets Other ML Algorithms * Andrew Ng’s example @nyghtowl
7. @nyghtowl Example: Model / Equation y = mx + b University GPA High School GPA Linear Regression Model Example coefficients
8. Model = activation func( sum( input * weight ) ) > threshold @nyghtowl Basic NN Equation X0 - bias X1 Xn k y j X kj W M Processing Unit f(y) Activation Functions • sigmoid • rectified linear • softmax • binary step
9. @nyghtowl Example: Attend a Talk? Speaker Time y speaker * .3 topic * .5 time *.2 Topic 30% 50% 20% f(y) M threshold 45% Inputs Weights
10. Basic NN Structure Input = data / features Hidden = processors Output = result Parameters are tricky & unique to the problem @nyghtowl
11. Multi-layer Structure Input Layer 2 Layer 3 Layer 4 Output Hidden @nyghtowl
12. Ada Lovelace? Example: Computer Vision Layers Pixels Edges Object Parts Object Models Layer 2 Layer 3 Layer 4 Input @nyghtowl
13. Cross Validation: Labeled Data Test Data Training Data Rachel Olivia @nyghtowl
14. Goal: minimize error Supervised Loss Function bias input n f( ) Loss Functions • mean squared error • negative log likelihood • cross entropy input 1 Error prediction - actual label @nyghtowl Output M Rachel
15. Optimization Method ~ Back-prop @nyghtowl Input Output Hidden Run until error stops improving = converge Loss Function
16. accuracy? Validation - Evaluate Test Data @nyghtowl Olivia Test Data
17. Recurrent Nets (RNN) Sample NN Types @nyghtowl Feed Forward Nets (FNN) Convolutional Nets (CNN)
18. Systems Understanding Focus • Graphical Processing Units (GPUs) ◦ Gaming ◦ AWS Additional Areas to Understand • General systems architecture • Parallelism • Distributed Systems • High Performance Computing (HPC & MPIs) @nyghtowl
19. Python Packages Basics & Deep Math GUI & Details Masked @nyghtowl • Theano • Lasagne (CNN) • PyLearn2 (CNN) • Kayak (CNN) • Blocks (RNN) • OpenDeep (RNN, CNN) • PyBrain (RNN, CNN) • Keras (RNN, CNN) • MetaMind (RNN, CNN) • Graphlab (CNN) • Sklearn (RBM) • Caffe (CNN) • CXXNet (CNN) • FANN2
20. Example: MNIST ~ “Hello World” • Classify handwritten digits 0-9 • Each pixel is an input • Input value ranges 0-255 (white to black) @nyghtowl
21. 0 0 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 Example: Input @nyghtowl
22. Example: MNIST Structure Input Output Hidden ...784 Nodes ...1000+ Nodes 7 layers 10 outputs 4 9 8 7 6 5 4 3 2 1 0 @nyghtowl
23. @nyghtowl Code: Train Model import graphlab as gl filename = 'http://s3.amazonaws.com/GraphLab-Datasets/mnist/sframe/train' inputs = gl.SFrame(filename) # Train neural net MNIST model model = gl.neuralnet_classifier.create(inputs, target=’label’)
24. Example: Layer Features @nyghtowl * Yann LeCunn Input Hidden Output
25. @nyghtowl # Ratio of number of correct predictions print ‘Accuracy’, model.evaluate(test_data) Accuracy: 0.97 Code: Validation
26. MNIST Repository github.com/nyghtowl/Neural_Net_Newbies @nyghtowl
27. Deep Learning - Bottom Line Automatic feature detection / Pre-train @nyghtowl
28. • Nature of Code: Neural Networks http://natureofcode.com/book/chapter-10-neural-networks/ • Neural Nets and Deep Learning http://neuralnetworksanddeeplearning.com/ • Neural Nets Demystified (Welch) http://www.welchlabs.com/blog/ • Theano Tutorial http://deeplearning.net/software/theano/tutorial/index.html#tutorial • “The State of Deep Learning in 2014” https://speakerdeck.com/ogrisel/the-state-of-deep- learning-in-2014 • “Hacker’s Guide to Neural Nets” https://karpathy.github.io/neuralnets/ • “Automated Image Capturing with ConvNets and Recurrent Nets” (Karpathy) http://cs.stanford. edu/people/karpathy/sfmltalk.pdf • Deeplearning.net • Wikipedia.org Where to go next... @nyghtowl
29. References: Individuals & Groups Individuals • Adam Gibson (Skymind) • Tarin Ziyaee (Apple) • Bryan Catanzaro (Baidu Research) • Fernand Pajot (Notion Labs) • Paco Nathan (Databricks) • Markus Beissinger (OpenDeep) Groups & Research • Deeplearning.net (esp. tutorial section) • Neural Computation & Adaptive Perception Research (research group) http://www.cifar.ca/neural- computation-and-adaptive-perception-research-progress • http://www.jeremyfreeman.net/ • Wikipedia.org @nyghtowl
30. • Nature of Code: Neural Networks http://natureofcode.com/book/chapter-10-neural-networks/ • Theano Tutorial http://deeplearning.net/software/theano/tutorial/index.html#tutorial • Machine Learning (Coursera-Ng) https://class.coursera.org/ml-005/lecture • Hacker’s Guide to Neural Nets (Stanford - Karpathy) https://karpathy.github.io/neuralnets/ • Neural Networks for Machine Learning (Coursera - Hinton) https://class.coursera.org/neuralnets-2012- 001/lecture • Neural Nets and Deep Learning http://neuralnetworksanddeeplearning.com/ • Deep Learning Stanford CS http://deeplearning.stanford.edu/ • Deep Learning Tutorial (NYU - LeCun) http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf • Deep Learning Tutorial (U Montreal - Bengio) http://deeplearning.net/tutorial/deeplearning.pdf • Tutorial on Deep Learning for Vision https://sites.google.com/site/deeplearningcvpr2014/ • ipam Graduate Summer School: Deep Learning, Feature Learning http://www.ipam.ucla. edu/programs/summer-schools/graduate-summer-school-deep-learning-feature-learning/?tab=schedule References: Tutorials @nyghtowl
31. • “ANN Network Basics” https://en.wikibooks.org/wiki/Artificial_Neural_Networks/Neural_Network_Basics • “The State of Deep Learning in 2014” https://speakerdeck.com/ogrisel/the-state-of-deep-learning-in- 2014 • “Automated Image Capturing with ConvNets and Recurrent Nets” http://cs.stanford. edu/people/karpathy/sfmltalk.pdf • “Convolutional Neural Net” http://yann.lecun.com/exdb/publis/psgz/lecun-iscas-10.ps.gz • “Understanding Convolutions” https://colah.github.io/posts/2014-07-Understanding-Convolutions/ • “Hot Yet Little-Known Trend That’ll Supercharge AI” http://www.wired.com/2015/02/hot-yet-little-known- trend-thatll-supercharge-ai/ • “Show and Tell: A Neural Image Caption Generator” http://arxiv.org/pdf/1411.4555.pdf? • “Explain Images with Multimodal Recurrent Neural Networks” http://arxiv.org/pdf/1410.1090v1.pdf References: Articles & Publications @nyghtowl
32. References: Video/Audio • Neural Nets Demystified (Welch) http://www.welchlabs.com/blog/ • Recent Developments in Deep Learning (Hinton) http://youtu.be/VdIURAu1-aU • RSS2014: Deep Learning (Ng) http://youtu.be/W15K9PegQt0 • Deep Learning, Self-Taught Learning and Unsupervised Feature Learning (Ng) http://youtu. be/n1ViNeWhC24 • Deep Learning on GPU Clusters (Catanzaro) http://youtu.be/brui4N2orlI • Automated Image Captioning with ConvNets and Recurrent Nets (Karpathy) http://youtu. be/ZkY7fAoaNcg • Deep Learning for NLP http://techtalks.tv/talks/deep-learning-for-nlp-without-magic-part-1/58414/ • Deep Learning: Machine Perception and Its Applications https://www.youtube.com/watch? v=hykoKDl1AtE • Talking Machines: The History of Machine Learning from Inside Out (12 min in) http://www. thetalkingmachines.com/blog/2015/2/26/the-history-of-machine-learning-from-the-inside-out? lang=en&utm_campaign=SendToFriend&uid=15226420&utm_content=item_link&utm_source=email&ut m_medium=self&china_variant=False • How to make a neural network in your bedroom http://youtu.be/n-YbJi4EPxc • Who is Afraid of Non-Convex Loss Functions http://videolectures.net/eml07_lecun_wia/ @nyghtowl
33. References: Slides • “Deep Learning Applications & Neural Nets“ http://www.slideshare.net/hammawan/deep-neural- networks • “Machine Perception with Neural Networks” (Strata 2014 Conference) http://cdn.oreillystatic. com/en/assets/1/event/105/Neural%20Networks%20for%20Machine%20Perception%20Presentation. pdf • “Theano and LSTM for Sentiment Analysis” https://drive.google. com/file/d/0B8qS82bey7OmbnhaRHdkN0d4bEU/view • “Systems for Deep Learning: A View from the Trenches” https://dl.dropboxusercontent. com/u/21184584/bcatanzaro_fall_2014_sigai.pdf • “Deep Neural Networks” http://www.slideshare.net/hammawan/deep-neural-networks • “Deep Learning: Intro” http://www.hlt.utdallas.edu/~vgogate/seminar/2013f/lectures/intro-deep-learning. pdf • “Recurrent Neural Networks for Text Analysis” https://docs.google. com/presentation/d/1HYfUZLRZRJovQpv5mYxox9bz9erxj7Ak_ZovENMvM90/edit#slide=id.p @nyghtowl
34. References: Images • http://code-spot.co.za/2009/10/08/15-steps-to-implemented-a-neural-net/ • http://www.texample.net/tikz/examples/neural-network/ • http://onlinestatbook.com/2/regression/intro.html • http://jaoying-google.blogspot.com/2012_12_01_archive.html • https://www.kaggle.com/forums/f/15/kaggle-forum/t/10878/feature-representation-in-deep-learning • http://maggieholmes.typepad.com/my_weblog/2011/07/scrapbooking.html • http://www.teds.com.au/photolounge/why-print-photos • http://users.clas.ufl.edu/glue/longman/1/einstein.html • https://en.wikipedia.org/wiki/File:Feed_forward_neural_net.gif • http://www.scienzagiovane.unibo.it/English/artint/8-neural-networks.html • http://www.clarifai.com/technology • http://www.nvidia.com/object/what-is-gpu-computing.html • https://www.classes.cs.uchicago.edu/archive/2013/spring/12300-1/pa/pa1/ • http://yann.lecun.com/exdb/lenet/ • http://people.csail.mit.edu/torralba/research/drawCNN/drawNet.html(CNN Features) • http://vis-www.cs.umass.edu/lfw/ (LFW)
35. Next for NN Near-term: Feature Engineering Combining Models Natural Language Processing Image Analysis Longer-term: Unlabeled data - finding patterns Generative (producing new data with NN) @nyghtowl
36. Neural Nets for Newbies - Melanie Warrick nyghtowl.io (slides) github.com/nyghtowl/Neural_Net_Newbies (code) @nyghtowl
37. Special Thanks Leon Barovic Lindsay Cade Jeremy Dunck Tim Elser Moon Limb Meggie Mahnken Isabel Markl Marissa Marquez Paul Minton Jason Morrison Chris Nicholson Erin O’Connell Tiia Priilaid Kelley Robinson Vijay Ramesh Shawn Scully Megan Speir Jana Thompson Andrew Weiss Python Software Foundation