AI in the Cloud, So seductive, but so overwhelming !

L’IA dans le Cloud constitue l’un des principaux shifts technologiques du moment sur la Data Science. Tous les grands providers de Cloud tiennent maintenant la promesse de démocratiser l'accès à la Data Science et de fournir des outils facilitant le développement et l’industrialisation de modèles de Machine Learning, que l’on soit Data Scientist ou non.

Mais force est de constater que tout n’est pas si simple, et que de nombreux pièges se présentent sur notre chemin pour pouvoir utiliser pleinement le potentiel de l’IA dans le Cloud. Durant ce talk, nous vous présenterons ces principaux pièges et vous proposerons des grandes leçons à en tirer pour faire de vos projets d’IA dans la Cloud un succès.

Transcript

1. AI in the cloud So seductive, but so overwhelming ! 1
2. Yoann Benoit @dicaormu @YoannBENOIT Diana Ortega 2
3. Context 3
4. Real World ML System https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf 4
5. Real World ML System https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf 5
6. Cloud Providers offer a wide range of perspectives But it’s also full of traps ! 6
7. Do machine learning like the great engineer you are, not like the great machine learning expert you aren’t. Martin Zinkevich (Research Scientist at Google) 7
8. 1. Before starting your project 2. While implementing your project 3. While running in production Agenda 8
9. 1. Before starting your project 9
10. Lesson 1 We are in the cloud! 10
11. 11 We are in the cloud! • Easily create ML workflows • Easy to scale up and down depending on service demand • Service availability and cutting out the high cost of hardware • Access to the last technos, machines and updates • Access to managed services that you can use easily • Pay for what you use
12. 12 We are in the cloud!
13. Pipeline of a data science project Identify the problem Collect & Explore data Train & test models Deploy & use models 13
14. 1. Do I need cloud all the time ? • Is your use case strategic for the business ? • Is it technically feasible ? Do I need to prototype first ? LESSON 1 Don’t be afraid to launch or test your product locally 14
15. Lesson 2 I want a chatbot with a Natural Language Processing (NLP) engine 15
16. System integration 16
17. The team we think we need 17 Expert NLP Expert ML
18. 2. What are my options? • Is there a managed service doing what I want to do ? LESSON 2 Explore existing managed services. It is difficult to create new algorithms. It is easier to use existing ones. 18
19. What are my options? Based on https://mattturck.com/data2019 19 Computer Vision General ML Speech & NLP
20. 2. What are my options? • Am I a provider of the service I’m creating ? • Do I want to be cloud-agnostic ? ◦ If so, what do I have to do to implement the same existing thing ? LESSON 2 Explore existing managed services. It is difficult to create new algorithms. It is easier to use existing ones. 20
21. 2. While implementing your project 21
22. Lesson 3 I think I forgot something 22
23. We’ve got the power ! 23
24. $165 24 With great power comes great responsibility
25. 25 GPU Type Virtual GPU Mem (G) Price (hour) p3.2xlarge 8 61 $3.06 p3.8xlarge 32 244 $12.24 p3.16xlarge 64 488 $24.48 p2.xlarge 4 61 $0.90 p2.8xlarge 32 488 $7.20 p2.16xlarge 64 768 $14.40 Examples of AWS machines
26. 26 GPU Type Mem/GPU GPUs Price/GPU (hour) V100 16 1,8 $2.48 P100 16 1,2,4 $1.46 K80 12 1,2,4,8 $0.45 Examples of GCP machines with GPU
27. 27 Examples of docker images in Docker hub
28. 28 A new paradigm for choosing your machines Paradigm #1 : Storage VS compute Dissociating storage from computing capabilities Computing capabilities become ephemeral
29. 29 A new paradigm for choosing your machines Paradigm #1 : Storage VS compute Dissociating storage from computing capabilities Computing capabilities become ephemeral Paradigm #2 : Notebook VS training VS inference Dissociating the machine on which runs the notebook instance from the machines needed for training models and inference
30. 3. Save energy... and money • Are there machines or docker images ready to use ? • Do I have to use all services ? • Do I really need to use the last trendy available GPU ? • Do I even need a GPU ? LESSON 3 Reuse existing images. Embrace new paradigms for machines sizing. 30
31. Lesson 4 It’s the jungle out there 31
32. 32 Endpoints list in AWS SageMaker
33. 33 Endpoint configuration in AWS SageMaker
34. 34 Repository configuration for versioning in AWS SageMaker
35. • Do you have to let all your notebooks open ? ◦ You can version your notebooks in Github with SageMaker • Are you going to save all your models ? ◦ You need to iterate to create your models. Storage has a cost. • Stop everything you don't use ! LESSON 4 Versioning is not just about the code. Manage your notebooks and models like the great craftsman you are. 4. Think about versioning 35
36. Lesson 5 Build or Buy ? 36
37. Managed VS In-House VS Managed In house 37 AI Platform
38. • Pre-tested algorithms/models • Optimisation • External infrastructure • Auto-scalable Managed VS In-House • Can be installed / migrated on any provider • Total control of the generated models 38 Managed In house • Vendor lock-in • No control of the training dataset • You need to think about infrastructure • More time in the model and infrastructure optimisation
39. Choose your customisation needs 39 Custom implementations on dedicated machines Custom Algorithms within a platform Dedicated algorithms within a platform Fully Managed Services Flexibility Ease of Use
40. 5. Benefit from Cloud Provider in-house implementations • On being vendor lock-in : Choose your fights ! • Specify your customisation needs and select the best according strategy LESSON 5 There is a tradeoff between delegation and implementation. Focus on your strategic advantage. Don’t reinvent the wheel for the rest. 40
41. 3. While running in production 41
42. Lesson 6 You have to see 42
43. You just finished your project and it is in production 43
44. Some days later... 44 I can’t access my endpoint Why is it so slow? How much is it going to cost me?
45. 45 EC2 CloudWatch Lambda
46. 6. Think about observability • Look for your endpoint status • Failed jobs • How many invocations do you have? • Latency • Memory, Disk and Cpu • Logs LESSON 6 You should see all the time what is going on with your AI model, with your pipeline and with the resources you are using. 46
47. Take Aways 47
48. Take Away Before starting your project 48 Do I need cloud all the time ? Don’t be afraid to launch or test your product locally.
49. Take Away Before starting your project 49 What are my options ? Explore existing managed services. It is difficult to create new algorithms. It is easier to use existing ones.
50. Take Away While implementing your project 50 Save energy … and money Reuse existing images. Embrace new paradigms for machines sizing
51. Take Away While implementing your project 51 Think about versionning Versioning is not just about the code. Manage your notebooks and models like the great craftsman you are.
52. Take Away While implementing your project 52 Choose your fights There is a tradeoff between delegation and implementation. Focus on your strategic advantage. Don’t reinvent the wheel for the rest.
53. Take Away While running in production 53 Think about observability You should see all the time what is going on with your AI model, with your pipeline and with the resources you are using.
54. AI in the Cloud is very seductive and can unlock many problems. ... But AI in the Cloud, without Software Engineering best practices, is just a buzz word. Diana Ortega & Yoann Benoit (XebiCon 2019) 54
55. Thank you! Questions? @Xebiconfr - #Xebicon19 55