Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serving deep learning models through a serverless pipeline

20d0ddc61e80bce04a63680da0160756?s=47 Aletheia
September 22, 2020

Serving deep learning models through a serverless pipeline

A talk about Neosperience Image Memorability architecture, presented at ServerlessDays Zurich on September, 24th 2020

20d0ddc61e80bce04a63680da0160756?s=128

Aletheia

September 22, 2020
Tweet

Transcript

  1. www.neosperience.com | blog.neosperience.com | info@neosperience.com Neosperience Empathy in Technology Serving

    deep learning models through a serverless pipeline September, 24th 2020
  2. Luca Bianchi Who am I? github.com/aletheia https://it.linkedin.com/in/lucabianchipavia https://speakerdeck.com/aletheia Chief Technology

    Officer @ Neosperience Chief Technology Officer @ WizKey Serverless Meetup and ServerlessDays Italy co-organizer www.bianchiluca.com @bianchiluca
  3. what makes every customer unique, them in 1:1 experiences and

    your customer base. Neosperience Cloud Understand Engage Grow
  4. Neosperience Cloud Cloud Understand Engage Grow

  5. Evaluate marketing images to drive customer attention and make your

    campaign memorable An AI-driven image evaluation solution Neosperience Image Memorability is a business driven implementation of memorability deep learning algorithms trained on business specific dataset, to achieve a memorability score and relevant areas analysis. Image memorability service can be tested at image.neosperience.com
  6. An image with high memorability score does not mean it

    is good for marketing How it works? Great authored images are can be misleading when you analyze their memorability map score 9.32 / 10 these areas are the most relevant of the image (red color), so they have the highest stickiness in viewer’s brain this is probably the most interesting part from a brand perspective, but the image does not focus attention on it (cold color)
  7. why is this relevant to us?

  8. Building and deploying a machine learning solution is not an

    easy task Start from service requirements - easy to build - easy to deploy and operate - easy to integrate - implement memorability algorithm (score, heatmap) - serverless (never pay for idle, scalability) - cost-effective - response time within seconds
  9. image processing pipeline

  10. How to handle image processing with serverless? Serverless image pipeline

    Amazon S3 • no need to use AWS SDK on client • upload is performed by client through a POST request to the signed url AWS Lambda • checks user licence plan (business constraints) • synthesizes S3 signed urls to allow image upload Amazon API Gateway • offers authentication • sends request to lambda to build image upload url
  11. Image Memorability 1.0 — pipeline

  12. what about ML algorithms?

  13. evaluate managed services

  14. A fully managed service for computer vision Amazon Rekognition Amazon

    Rekognition • automated AI service with available with pre-trained state-of-the-art deep learning models • ready to analyze images for classification, object detection and labelling • additional feature such as logo and celebrity recognition • support for face detection, analysis and identification How to leverage? • upload your image to S3 and pass the reference to the Rekognition API, or send a base64 encoded image directly from your client to the API.
  15. Provide business specific labels to Rekognition classifier Amazon Rekognition with

    custom labels Amazon Rekognition with custom labels • provide a set of labelled images to fine tune your training • images can be uploaded in batches using Amazon SageMaker manifest format • it solves a classification problem, we need a regression number (score) with something else deeply related to how network performs
  16. Check our requirements Solution evaluation ✓ easy to build ✓

    easy to deploy and operate ✓ easy to integrate ✗ implement memorability algorithm (score, heatmap) ✓ serverless (never pay for idle, scalability) ✓ cost-effective ✓ response time within seconds
  17. build a custom model

  18. A GPU powered EC2 instance Amazon Deep Learning Instances EC2

    instances for deep learning • GPU instances (p2, p3 instance type) • model training and inference on the same cluster • bring your own deep learning framework (PyTorch, Keras) • can implement any ML model on this architecture How to leverage? • needs an image processing pipeline to handle images • needs a scheduling logic to process data • needs S3 bridge to store and retrieve data EC2 p3 instance Deep Learning 
 AMI
  19. EC2 instances for deep learning • GPU instances (p2, p3

    instance type) • model training and inference on the same cluster • bring your own deep learning framework (PyTorch, Keras) • can implement any ML model on this architecture A GPU powered EC2 instance Amazon Deep Learning Instances EC2 instances for deep learning • cost effective GPU instances (any instance type) • model training and inference on different cluster • bring your own deep learning framework (PyTorch, Keras) • can implement any ML model on this architecture How to leverage? • needs an image processing pipeline to handle images • needs a scheduling logic to process data • needs S3 bridge to store and retrieve data EC2 Deep Learning 
 AMI Elastic Inference
  20. Image Memorability 1.0 on prem servers used to reduce cost

    impact
  21. Check our requirements Solution evaluation ✗ easy to build ✗

    easy to deploy and operate - easy to integrate ✓ implement memorability algorithm (score, heatmap) ✗ serverless (never pay for idle, scalability) ✗ cost-effective ✓ response time within seconds EC2 Deep Learning 
 AMI Elastic Inference
  22. build a custom model

  23. build a custom model on a managed service

  24. Fully managed machine learning operation environment Amazon SageMaker Managed Notebook

    Instances, training and inference platform • Jupyter Notebooks managed in the cloud • scales up and down training across multiple instances • support to inference endpoint creation and model hosting • testing, debugging, and fine-tuning through SageMaker Studio • allow building custom ML models such as AMNet How to leverage? • needs an image processing pipeline to handle images • needs S3 bridge to store and retrieve data Amazon SageMaker
  25. Image Memorability 1.0

  26. Image Memorability 2.0 reduced the number of servers, but still

    needed to reduce costs
  27. Achieving near human performances with AMNet in common applications, memorability

    score is usually normalized between [0, 0.68], a score of 1,0 means having an image performance equal to human comparison to state-of-the-art implementation https://arxiv.org/pdf/1804.03115.pdf
  28. Check our requirements Solution evaluation ✓ easy to build ✓

    easy to deploy and operate - easy to integrate ✓ implement memorability algorithm (score, heatmap) ✓ serverless (never pay for idle, scalability) - cost-effective ✓ response time within seconds EC2 Deep Learning 
 AMI Elastic Inference
  29. can we improve this result?

  30. Consider your model intrinsics to improve architecture Same model, two

    results Image inference with a regression model • memorability score is built with a single evaluation of the network • shrinking down the CNN used has a minimum impact on accuracy (from 0.677 to 0.64) but a huge impact on performances • shrinking the network means being able to adopt on a more cost-effective approach Backpropagation to compute image heatmap • is built back propagating network weights and checking activations • is computationally intensive even on GPU (15s on an nVidia Tesla V100)
  31. Image Memorability 2.0

  32. Introducing Amazon EFS is a huge leap forward Considerations •

    source images are shared between Lambdas with no need to upload/download from S3 • S3 interface is maintained to ease image upload • still need Amazon SageMaker to process image heatmap • using Lambda made execution time for score compute rise from 5 seconds to 12 seconds • overall image scoring is improved from an average of 90 seconds to 12 seconds (queue jobs are processed in parallel by Lambdas) • no costs for idle endpoints • costs reduced by an order of magnitude (reduced load on SageMaker endpoints) • could remove on premise servers (SageMaker endpoints + Inference on Lambda is cost effective)
  33. Check our requirements Solution evaluation ✓ easy to build ✓

    easy to deploy and operate ✓ easy to integrate ✓ implement memorability algorithm (score, heatmap) ✓ serverless (never pay for idle image scoring, scalability improved) ✓ cost-effective ✓ response time within seconds
  34. We were able to build a cost effective image processing

    pipeline for a custom ML model Wrap up • custom ML models can be implemented on AWS Lambda (if no GPU is required or the slow down is acceptable) • Amazon SageMaker is a great alternative, but more expensive (you pay for idle at least one instance) • model training is performed on Amazon SageMaker (consider using spot instances for training) • choose the right computing for your machine learning model requirements • Amazon EFS is a game changer in many applications • be aware of AWS Lambda dependency size when using ML libraries
  35. Thank you.

  36. bit.ly/slsdays-20200924

  37. Empathy in Technology

  38. www.neosperience.com | blog.neosperience.com | info@neosperience.com