Serverless Deep Learning at Edge

Join us in a marvellous journey to the frontiers and possibilities of machine learning on Serverless architectures.

1. www.neosperience.com | blog.neosperience.com | info@neosperience.com May 2018 Neosperience. Empathy in Technology Serverless Machine Learning su dispositivi connessi
2. First of all..
3. aka don’t take me too seriously.. Safe Harbour Statement Our discussion may include predictions, estimates or other information that might be considered forward-looking. While these forward-looking statements represent our current judgment on what the future holds, they are subject to risks and uncertainties that could cause actual results to differ materially. You are cautioned not to place undue reliance on these forward-looking statements, which reflect our opinions only as of the date of this presentation. Please keep in mind that we are not obligating ourselves to revise or publicly release the results of any revision to these forward- looking statements in light of new information or future events. Throughout today’s discussion, we will attempt to present some important factors relating to our business that may affect our predictions.
4. Luca Bianchi Who am I? • Chief Technology Officer @ Neosperience • Working on a lot of bleeding edge technologies • Passionate developer: love writing code, hate meetings Neosperience - Empathy in Technology Understand, engage, and delight customers, using personalization to deliver relevant experiences that drive loyalty and increase value.
5. A cloud platform built on AWS to deliver DCX projects Neosperience Cloud • Deeply understand their customers and be more useful to them by delivering relevant digital experiences. • Delight customers by delivering relevant experiences across mobile, web, in-store. • Maintain their Brand identity and increase value as platforms like Amazon, Google and Facebook drive up disintermediation and make companies unintentional utilities. • Keep pace with the variety of devices and interaction models available to customers to overcome complexity and costs associated with the alignment of apps, web apps, social media and conversational interfaces. Neosperience Cloud is the technology platform that allows creating personalized experiences for your customers that drive loyalty and faster paths 
 to purchase. Unlike existing technologies that rely only on demographics data, we use proprietary models, developed with AI, to personalize your offering to the right segment. A compelling experience for each customer at the right time, place, and situational context.
6. A cloud platform built on AWS to deliver DCX projects Neosperience Cloud • Deeply understand their customers and be more useful to them by delivering relevant digital experiences. • Delight customers by delivering relevant experiences across mobile, web, in-store. • Maintain their Brand identity and increase value as platforms like Amazon, Google and Facebook drive up disintermediation and make companies unintentional utilities. • Keep pace with the variety of devices and interaction models available to customers to overcome complexity and costs associated with the alignment of apps, web apps, social media and conversational interfaces. Neosperience Cloud is the technology platform that allows creating personalized experiences for your customers that drive loyalty and faster paths 
 to purchase. Unlike existing technologies that rely only on demographics data, we use proprietary models, developed with AI, to personalize your offering to the right segment. A compelling experience for each customer at the right time, place, and situational context. …which means fast time to market, machine learning and scalability by design.
7. a few words about our meetup… Serverless Meetup • 598 members and counting.. • Monthly Meetups 
 (https://www.meetup.com/Serverless-Italy/members/) • Serverless OnTheRoad and OnStage
8. a few words about our meetup… Serverless Meetup • 598 members and counting.. • Monthly Meetups 
 (https://www.meetup.com/Serverless-Italy/members/) • Serverless OnTheRoad and OnStage • ServerlessDays (http://serverlessdays.io)
9. Retrocomputing to the future of technology ComPVter • An association of Smart People • Every Thursday in Prado (PV) - via del Commercio 13 • ~120 members • Many projects: IoT, drones, AI, embedded, 3D printing, etc.
10. Retrocomputing Museum ComPVter
11. Projects and Fun stuff ComPVter
12. https://www.eventbrite.it/e/biglietti-blockchain-e-smart-contracts-45798139468 Evento: 14:30 – 15:30 Prof Paolo Giudici – Università di Pavia Tecnologie Finanziarie (Fintech): cosa sono? 15:30 – 17:30 Ing. Diego Ferri - Looptribe Trust me, I’m a Smart Contract 16:30 – 17:30 Avv. Marco Pagani - WizKey Le ICO: le evoluzioni del fenomeno nel 2017 e 2018 16:30 – 17:30 Chiusura lavori e Networking When Thu May, 31st Where Aula 4 - Polo Tecnologico Università di Pavia Via Adolfo Ferrata, 5 5 Via Adolfo Ferrata 27100 Pavia
13. Agenda • Serverless • Deep Learning • Issues of Deep Learning • Serverless Deep Learning • Serverless Deep Learning at edge • Limits of current approaches • A sneak preview on what’s coming next
14. Serverless an over-used buzzword
15. “Serverless architecture replaces long-running virtual machines with ephemeral compute power that comes into existence on request and disappears immediately after use. Use of this architecture can mitigate some security concerns such as security patching and SSH access control, and can make much more efficient use of compute resources. These systems cost very little to operate and can have inbuilt scaling features.” — ThoughtWorks, 2016 What is Serverless?
16. Function as the unit of deployment and scaling Implicitly fault-tolerant Metrics No machines, VMs, or containers Bring Your Own Code Stateless Never pay for idle Scales per request The Serverless Manifesto
17. Why? A lot of noise about machine deep learning… what the hell is it?!
18. Why? A lot of noise about machine deep learning… what the hell is it?!
19. Deep Learning Use Cases (text) •Text classification
20. Deep Learning Use Cases (text) •Text classification •Sentiment analysis
21. Deep Learning Use Cases (text) •Text classification •Sentiment analysis •Text translation
22. Deep Learning Use Cases (text) •Text classification •Sentiment analysis •Text translation •Speech to text
23. Deep Learning Use Cases (text) •Text classification •Sentiment analysis •Text translation •Speech to text •NLP
24. Deep Learning Use Cases (images) •Image classification
25. Deep Learning Use Cases (images) •Image classification •Face / pose detection
26. Deep Learning Use Cases (images) •Image classification •Face / pose detection •Face recognition
27. Neural Networks Revamped Deep Convolutional Neural Networks (CNN) A Neural Network with layers performing convolution operations, to extract features Hidden Layers and Back-propagation Projects error backwards to previous layers, to correct weight estimation Perceptron A set of fully connected layers to perform classification
28. Training Putting Deep Learning to work.. Heavy phase where a data set is processed many times (epochs) through a neural network to estimata its weights and optimize a loss function. Training is a computing intensive task. It must be run on GPU instances which are, really expensive. Inference Lightweight phase where a trained model (can be huge) is used to make inference about an unknown data sample. It should be handled as a DevOps task.
29. Serverless Deep Learning in the cloud
30. Manageable Machine Learning workflow Amazon SageMaker • Provides Jupyter notebooks in the cloud • Manages training instances setup and tear down • Handles multi-GPU training • Handles data load from/to training instances • Handles model persistence to S3 • Handles inference endpoint setup
31. Amazon SageMaker - Datasets and GPUs images are courtesy of Jerry Hargrove - @awsgeek - https://www.awsgeek.com
32. Amazon SageMaker - endpoints images are courtesy of Jerry Hargrove - @awsgeek - https://www.awsgeek.com
33. Amazon SageMaker - capabilities images are courtesy of Jerry Hargrove - @awsgeek - https://www.awsgeek.com
34. Poor network connectivity can affect inference availability Data transfer from client to cloud have an impact (i.e. affects realtime processing) Training Amazon SageMaker - evaluation Inference Data transfer between management instances and training instances can slow down training Data transfer between on premise and S3 requires time Development experience is not at its best Regulation constraints Scalability Managed Workflow
36. …moving to the edge… IDEA! Let’s run ML models on devices
37. Almost every provider support FaaS …but Functions on AWS have run also out of cloud A key element in Serverless: functions
38. 1. Configure the Raspberry Pi (Install IoT Core) 2. Install the MXNet Framework / TensorFlow 3. Create a Model Package 4. Create and Publish a Lambda Function 5. Add the Lambda Function to the Group 6. Add Resources to the Group 7. Add a Subscription to the Group 8. Deploy the Group AWS Greengrass and ML Inference
39. Works even offline Training Inference at Edge - evaluation Inference Data transfer between management instances and training instances can slow down training Data transfer between on premise and S3 requires time Development experience is not at its best Data transfer to cloud is limited only to inference results Regulation constraints Scalability Managed Workflow
41. on premise give up and install everything under your desk!
42. The GPU Rig On premise hardware • A PC with many GPUs • Usually 6-8 NVidia 1080Ti • Uses CUDA version 8.x (9.0?) • Multi-GPU training (40-lane CPU) • Very expensive (8K-10K) • Expensive running costs (requires 1600W)
43. Manual workflow management (store model, upload, create endpoints, etc.) Data is stored locally Limited scalability Data is downloaded once Runs Jupyter notebooks locally Matches data management policies Works even offline Training Inference at Edge - evaluation Inference Data transfer to cloud is limited only to inference results
45. hybrid deep learning? is that even possible??
46. Balance your neural network between on premise and cloud An hybrid deep learning architecture • Built on SageMaker and MXNet framework • Comes with cloud configuration and with a linux core client • Extends MXNet imperative idea to architectures • Every resource has a device descriptor (cores, features, usage_type, etc.) • Balances training on device resources based on capabilities • Splits network layers (slices) and dispatches them to the best available processing resource • Handles data sync between slices • Balances inference execution as well
48. Runs Jupyter notebooks locally but can be deployed on SageMaker Data is downloaded when required Data is stored locally and/or in cloud Managed Workflow Matches data management policies Works even offline Training Inference at Edge - evaluation Inference Data transfer to cloud is limited only to inference results Scalability
50. http://bit.ly/th-2018
51. www.neosperience.com | blog.neosperience.com | info@neosperience.com !42