An introduction to Deep Learning

Transcript

1. © 2019, Amazon Web Services, Inc. or its Affiliates. Introduction to Deep Learning Antje Barth Technical Evangelist AI and Machine Learning, AWS @anbarth 6 November 2019
2. © 2019, Amazon Web Services, Inc. or its Affiliates. A little about me… Antje Barth, AWS Technical Evangelist AI/ML • Data Enthusiast • AI / ML / Deep Learning • Machine Learning on Kubernetes • Big Data • Ex: MapR, Cisco #CodeLikeAGirl
3. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. ML/DL is NOT dark magic!
4. © 2019, Amazon Web Services, Inc. or its Affiliates. Machine Learning Artificial Intelligence Deep Learning “ability to make decisions” “ability to learn rules” “ability to learn concepts” Broad Niche Definitions
5. © 2019, Amazon Web Services, Inc. or its Affiliates. John McCarthy 1955 Image of John McCarthy courtesy of Wikipedia One of the ”Founding Fathers" of artificial intelligence
6. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Photo from the 1956 Dartmouth Conference with Marvin Minsky, Ray Solomonoff, Claude Shannon, John McCarthy, Trenchard More, Oliver Selfridge, and Nathaniel Rochester
7. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Frank Rosenblatt, 1957 Perceptron Source: Arvin Calspan Advanced Technology Center; Hecht-Nielsen, R. Neurocomputing (Reading, Mass.: Addison-Wesley, 1990)
8. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. First known deep network Alexey Grigorevich Ivakhnenko, 1965 Image of Prof. Alexey Ivakhnenko courtesy of Wikipedia.
9. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Paul Werbos, 1975 Backpropagation
10. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Yann LeCun, 1989 First application of backpropagation https://www.youtube.com/watch?v=FwFduRA_L6Q
11. © 2019, Amazon Web Services, Inc. or its Affiliates. Explosion of (digital) data GPUs & Acceleration Algorithms The advent of ML: 2000s
12. © 2019, Amazon Web Services, Inc. or its Affiliates. We all want machine learning everywhere
13. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Machine Learning is happening in companies of every size and industry Tens of thousands customers have chosen AWS for their ML workloads. More than twice as many customers using ML than any other cloud providers.
14. © 2019, Amazon Web Services, Inc. or its Affiliates. Supervised learning Run an algorithm on a labeled dataset, i.e., a dataset containing samples and answers. Gradually, the model learns how to correctly predict the right answer. Regression and classification are examples of supervised learning. Unsupervised learning Run an algorithm on an unlabeled dataset, i.e., a dataset containing samples only. Here, the model progressively learns patterns in data and organizes samples accordingly. Clustering and topic modeling are examples of unsupervised learning. Types of Machine Learning
15. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Supervised learning Unsupervised learning SOPHISTICATION OF ML MODELS AMOUNT OF TRAINING DATA REQUIRED Types of Machine Learning
16. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. AMOUNT OF TRAINING DATA REQUIRED Supervised learning Unsupervised learning SOPHISTICATION OF ML MODELS Types of Machine Learning
17. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Reinforcement learning (RL) Supervised learning Unsupervised learning AMOUNT OF TRAINING DATA REQUIRED SOPHISTICATION OF ML MODELS Types of Machine Learning
18. © 2019, Amazon Web Services, Inc. or its Affiliates. Remember when you first learned this?
19. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Let’s start with deep learning
20. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Predicting the price of a house with humans Price City ZIP Code Life Quality Parking Size # Rooms Accessibility Family Friendly
21. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Predicting the price of a house with neural net Price City ZIP Code Life Quality Parking Size # Rooms Accessibility Family Friendly Input Output Discovered by the neural network
22. © 2019, Amazon Web Services, Inc. or its Affiliates. Dataset The MNIST database of handwritten digits, has a training set of 60,000 examples, and a test set of 10,000 examples. Human-in-the-loop Deep Learning Training
23. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. https://ml4a.github.io/ml4a/neural_networks/ Deep Learning Training
24. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. https://ml4a.github.io/ml4a/neural_networks/ Learning Deep Learning Training
25. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Training = Minimizing the loss (error) function Backpropagation
26. © 2019, Amazon Web Services, Inc. or its Affiliates. Imagine you stand on top of a mountain with skis strapped to your feet. You want to get down to the valley as quickly as possible, but there is fog and you can only see your immediate surroundings. How can you get down the mountain as quickly as possible? You look around and identify the steepest path down, go down that path for a bit, again look around and find the new steepest path, go down that path, and repeat — this is exactly what gradient descent does. Tim Dettmers University of Lugano 2015 https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/ The "step size" is called the learning rate z=f(x,y) Stochastic gradient descent (SGD)
27. © 2019, Amazon Web Services, Inc. or its Affiliates. "Do neural networks enter and escape a series of local minima? Do they move at varying speed as they approach and then pass a variety of saddle points? Answering these questions definitively is difficult, but we present evidence strongly suggesting that the answer to all of these questions is no." "Qualitatively characterizing neural network optimization problems," Goodfellow et al, 2015 https://arxiv.org/abs/1412.6544 Local minima and saddle points
28. © 2019, Amazon Web Services, Inc. or its Affiliates. Optimizers
29. © 2019, Amazon Web Services, Inc. or its Affiliates. Validation dataset Trained neural network Validation accuracy Prediction at the end of each epoch * * One epoch consists of one full training cycle on the training set Validation
30. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Training accuracy Loss function Accuracy 100% Epochs Validation accuracy Loss Best epoch OVERFITTING Early Stopping
31. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. It’s able to do better than other ML and humans Deep Learning is a big deal
32. © 2019, Amazon Web Services, Inc. or its Affiliates. … Convolutional neural networks (CNN)
33. © 2019, Amazon Web Services, Inc. or its Affiliates. https://github.com/precedenceguo/mx-rcnn https://github.com/zhreshold/mxnet-yolo CNN: Object Detection
34. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. CNN: Object Detection
35. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. CNN: Object Segmentation
36. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. FDA-approved medical imaging https://www.periscope.tv/AWSstartups/1vAGRgevBXRJl https://www.youtube.com/watch?v=WE81dncwnIc CNN: Object Segmentation
37. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. https://github.com/Bartzi/stn-ocr CNN: Text Detection and Recognition
38. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. https://github.com/tornadomeet/mxnet-face CNN: Face Detection
39. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. https://github.com/dragonfly90/mxnet_Realtime_Multi-Person_Pose_Estimation CNN: Real-time pose estimation
40. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. https://github.com/awslabs/sockeye Long short-term memory (LSTM) networks Oh I remember!
41. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Generating new “celebrity” faces https://github.com/tkarras/progressive_growing_of_gans Generative adversarial networks (GANs) The future at work (already) today
42. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Turning semantic label maps into photo-realistic images or synthesizing portraits from face label maps using conditional generative adversarial networks (conditional GANs) https://github.com/NVIDIA/pix2pixHD Generative adversarial networks (GANs) The future at work (already) today
43. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Models can also generate: • Text from text • Text from images • Text from video • Images from text • Sound from video • 3D models from 2D images • Etc. https://github.com/hanzhanggit/StackGAN Stacked generative adversarial networks Wait! There’s more!
44. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. How do people “build” neural nets?
45. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Performance monitoring & adaptation 8 Data acquisition & storage 1 Model & Framework selection 3 Model training 4 Hyperparameter tuning 5 Model testing and simulation 6 Model deployment (inference) 7 Data labeling 2 ML Workflow
46. © 2019, Amazon Web Services, Inc. or its Affiliates. https://gitlab.com/juliensimon/dlnotebooks/blob/master/keras/03-fashion-mnist- sagemaker/Fashion%20MNIST-SageMaker.ipynb Demo: Image classification with Keras/Tensorflow • train locally without Amazon SageMaker • train locally with Amazon SageMaker (’local mode’) • train on infrastructure managed by Amazon SageMaker
47. © 2019, Amazon Web Services, Inc. or its Affiliates. Where to start?
48. © 2019, Amazon Web Services, Inc. or its Affiliates. FRAMEWORKS INTERFACES INFRASTRUCTURE AI Services Broadest and deepest set of capabilities THE AWS ML STACK VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS ML Services ML Frameworks + Infrastructure P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D & C O M P R E H E N D M E D I C A L L E X F O R E C A S T R E K O G N I T I O N I M A G E R E K O G N I T I O N V I D E O T E X T R A C T P E R S O N A L I Z E Ground Truth Notebooks Algorithms + Marketplace Reinforcement Learning Training Optimization Deployment Hosting Amazon SageMaker F P G A S E C 2 P 3 & P 3 D N E C 2 G 4 E C 2 C 5 I N F E R E N T I A G R E E N G R A S S E L A S T I C I N F E R E N C E D L C O N T A I N E R S & A M I s E L A S T I C K U B E R N E T E S S E R V I C E E L A S T I C C O N T A I N E R S E R V I C E
49. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. aws.com/ml Start building!
50. © 2019, Amazon Web Services, Inc. or its Affiliates. © 2019, Amazon Web Services, Inc. or its Affiliates. Thank you! antje.official antje @anbarth Antje Barth To learn more: AI.aws