Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless Deep Learning at Edge

Serverless Deep Learning at Edge

Join us in a marvellous journey to the frontiers and possibilities of machine learning on Serverless architectures.

Aletheia

May 17, 2018
Tweet

More Decks by Aletheia

Other Decks in Technology

Transcript

  1. aka don’t take me too seriously.. Safe Harbour Statement Our

    discussion may include predictions, estimates or other information that might be considered forward-looking. While these forward-looking statements represent our current judgment on what the future holds, they are subject to risks and uncertainties that could cause actual results to differ materially. You are cautioned not to place undue reliance on these forward-looking statements, which reflect our opinions only as of the date of this presentation. Please keep in mind that we are not obligating ourselves to revise or publicly release the results of any revision to these forward- looking statements in light of new information or future events. Throughout today’s discussion, we will attempt to present some important factors relating to our business that may affect our predictions.
  2. Luca Bianchi Who am I? • Chief Technology Officer @

    Neosperience • Working on a lot of bleeding edge technologies • Passionate developer: love writing code, hate meetings Neosperience - Empathy in Technology Understand, engage, and delight customers, using personalization to deliver relevant experiences that drive loyalty and increase value.
  3. A cloud platform built on AWS to deliver DCX projects

    Neosperience Cloud • Deeply understand their customers and be more useful to them by delivering relevant digital experiences. • Delight customers by delivering relevant experiences across mobile, web, in-store. • Maintain their Brand identity and increase value as platforms like Amazon, Google and Facebook drive up disintermediation and make companies unintentional utilities. • Keep pace with the variety of devices and interaction models available to customers to overcome complexity and costs associated with the alignment of apps, web apps, social media and conversational interfaces. Neosperience Cloud is the technology platform that allows creating personalized experiences for your customers that drive loyalty and faster paths 
 to purchase. Unlike existing technologies that rely only on demographics data, we use proprietary models, developed with AI, to personalize your offering to the right segment. A compelling experience for each customer at the right time, place, and situational context.
  4. A cloud platform built on AWS to deliver DCX projects

    Neosperience Cloud • Deeply understand their customers and be more useful to them by delivering relevant digital experiences. • Delight customers by delivering relevant experiences across mobile, web, in-store. • Maintain their Brand identity and increase value as platforms like Amazon, Google and Facebook drive up disintermediation and make companies unintentional utilities. • Keep pace with the variety of devices and interaction models available to customers to overcome complexity and costs associated with the alignment of apps, web apps, social media and conversational interfaces. Neosperience Cloud is the technology platform that allows creating personalized experiences for your customers that drive loyalty and faster paths 
 to purchase. Unlike existing technologies that rely only on demographics data, we use proprietary models, developed with AI, to personalize your offering to the right segment. A compelling experience for each customer at the right time, place, and situational context. …which means fast time to market, machine learning and scalability by design.
  5. a few words about our meetup… Serverless Meetup • 598

    members and counting.. • Monthly Meetups 
 (https://www.meetup.com/Serverless-Italy/members/) • Serverless OnTheRoad and OnStage
  6. a few words about our meetup… Serverless Meetup • 598

    members and counting.. • Monthly Meetups 
 (https://www.meetup.com/Serverless-Italy/members/) • Serverless OnTheRoad and OnStage • ServerlessDays (http://serverlessdays.io)
  7. Retrocomputing to the future of technology ComPVter • An association

    of Smart People • Every Thursday in Prado (PV) - via del Commercio 13 • ~120 members • Many projects: IoT, drones, AI, embedded, 3D printing, etc.
  8. https://www.eventbrite.it/e/biglietti-blockchain-e-smart-contracts-45798139468 Evento: 14:30 – 15:30 Prof Paolo Giudici – Università

    di Pavia Tecnologie Finanziarie (Fintech): cosa sono? 15:30 – 17:30 Ing. Diego Ferri - Looptribe Trust me, I’m a Smart Contract 16:30 – 17:30 Avv. Marco Pagani - WizKey Le ICO: le evoluzioni del fenomeno nel 2017 e 2018 16:30 – 17:30 Chiusura lavori e Networking When Thu May, 31st Where Aula 4 - Polo Tecnologico Università di Pavia Via Adolfo Ferrata, 5 5 Via Adolfo Ferrata 27100 Pavia
  9. Agenda • Serverless • Deep Learning • Issues of Deep

    Learning • Serverless Deep Learning • Serverless Deep Learning at edge • Limits of current approaches • A sneak preview on what’s coming next
  10. “Serverless architecture replaces long-running virtual machines with ephemeral compute power

    that comes into existence on request and disappears immediately after use. Use of this architecture can mitigate some security concerns such as security patching and SSH access control, and can make much more efficient use of compute resources. These systems cost very little to operate and can have inbuilt scaling features.” — ThoughtWorks, 2016 What is Serverless?
  11. Function as the unit of deployment and scaling Implicitly fault-tolerant

    Metrics No machines, VMs, or containers Bring Your Own Code Stateless Never pay for idle Scales per request The Serverless Manifesto
  12. Neural Networks Revamped Deep Convolutional Neural Networks (CNN) A Neural

    Network with layers performing convolution operations, to extract features Hidden Layers and Back-propagation Projects error backwards to previous layers, to correct weight estimation Perceptron A set of fully connected layers to perform classification
  13. Training Putting Deep Learning to work.. Heavy phase where a

    data set is processed many times (epochs) through a neural network to estimata its weights and optimize a loss function. Training is a computing intensive task. It must be run on GPU instances which are, really expensive. Inference Lightweight phase where a trained model (can be huge) is used to make inference about an unknown data sample. It should be handled as a DevOps task.
  14. Manageable Machine Learning workflow Amazon SageMaker • Provides Jupyter notebooks

    in the cloud • Manages training instances setup and tear down • Handles multi-GPU training • Handles data load from/to training instances • Handles model persistence to S3 • Handles inference endpoint setup
  15. Amazon SageMaker - Datasets and GPUs images are courtesy of

    Jerry Hargrove - @awsgeek - https://www.awsgeek.com
  16. Poor network connectivity can affect inference availability Data transfer from

    client to cloud have an impact (i.e. affects realtime processing) Training Amazon SageMaker - evaluation Inference Data transfer between management instances and training instances can slow down training Data transfer between on premise and S3 requires time Development experience is not at its best Regulation constraints Scalability Managed Workflow
  17. Almost every provider support FaaS …but Functions on AWS have

    run also out of cloud A key element in Serverless: functions
  18. 1. Configure the Raspberry Pi (Install IoT Core) 2. Install

    the MXNet Framework / TensorFlow 3. Create a Model Package 4. Create and Publish a Lambda Function 5. Add the Lambda Function to the Group 6. Add Resources to the Group 7. Add a Subscription to the Group 8. Deploy the Group AWS Greengrass and ML Inference
  19. Works even offline Training Inference at Edge - evaluation Inference

    Data transfer between management instances and training instances can slow down training Data transfer between on premise and S3 requires time Development experience is not at its best Data transfer to cloud is limited only to inference results Regulation constraints Scalability Managed Workflow
  20. The GPU Rig On premise hardware • A PC with

    many GPUs • Usually 6-8 NVidia 1080Ti • Uses CUDA version 8.x (9.0?) • Multi-GPU training (40-lane CPU) • Very expensive (8K-10K) • Expensive running costs (requires 1600W)
  21. Manual workflow management (store model, upload, create endpoints, etc.) Data

    is stored locally Limited scalability Data is downloaded once Runs Jupyter notebooks locally Matches data management policies Works even offline Training Inference at Edge - evaluation Inference Data transfer to cloud is limited only to inference results
  22. Balance your neural network between on premise and cloud An

    hybrid deep learning architecture • Built on SageMaker and MXNet framework • Comes with cloud configuration and with a linux core client • Extends MXNet imperative idea to architectures • Every resource has a device descriptor (cores, features, usage_type, etc.) • Balances training on device resources based on capabilities • Splits network layers (slices) and dispatches them to the best available processing resource • Handles data sync between slices • Balances inference execution as well
  23. Runs Jupyter notebooks locally but can be deployed on SageMaker

    Data is downloaded when required Data is stored locally and/or in cloud Managed Workflow Matches data management policies Works even offline Training Inference at Edge - evaluation Inference Data transfer to cloud is limited only to inference results Scalability