PyCon & PyData. Applying deployment oriented mindset for building Machine Learning models

Developing a complicated ensemble model with hundreds of features fetched from a bunch of different sources? Give me two! Showing great metrics to the stakeholders and already discussing how it will hit a home run in production? Why not! And then getting stuck for months trying to deploy the model and fighting with data inconsistency and bugs? Sounds familiar? This talk will focus on providing guidelines on how to build your model development process keeping in mind the deployment phase to come later on.

Transcript

1. APPLYING DEPLOYMENT ORIENTED MINDSET FOR BUILDING MACHINE LEARNING MODELS MARIANNA DIACHUK
2. AGENDA ➔ Why am I telling you this? ➔ What I mean by deployment? ➔ Deployment problems ➔ Model development process ➔ What deployment oriented mindset gives you 2
3. ABOUT ME ✍ I’m a data scientist from Kyiv (Ukraine) ✍ I developed and deployed multiple scoring and antifraud ensemble models ✍ I leaded a small but proud team of 2 data scientists and 1 data engineer ✍ I’m a mother of 3 dragons ducks 3
4. 4 WHY AM TELLING YOU THIS?
5. WHAT 5 WHY AM TELLING YOU THIS?
6. WHAT WHY 6 WHY AM TELLING YOU THIS?
7. WHAT WHY 7 WHY AM TELLING YOU THIS?
8. WHAT I MEAN BY DEPLOYMENT? 8
9. WHAT I MEAN BY DEPLOYMENT? 9
10. I HAD MY SHARE OF FAILS TOO… 10
11. DEPLOYMENT PROBLEMS (SOME OF THEM) ➢ Model response inconsistency 11 research environment development environment
12. ➢ Model response inconsistency 12 research environment development environment dataset DEPLOYMENT PROBLEMS (SOME OF THEM)
13. ➢ Features inconsistency 13 research environment development environment dataset DEPLOYMENT PROBLEMS (SOME OF THEM)
14. ➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency 14 DEPLOYMENT PROBLEMS (SOME OF THEM)
15. ➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency ➢ Model is not scalable and so on and so on… 15 DEPLOYMENT PROBLEMS (SOME OF THEM)
16. ➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency ➢ Model is not scalable and so on and so on… 16 DEPLOYMENT PROBLEMS (SOME OF THEM)
17. MODEL DEVELOPMENT PROCESS LOOKS LIKE... 17
18. OR LIKE THIS... Research process Agile process 18
19. MODEL DEVELOPMENT PROCESS 19 Remember about deployment.
20. How fast should our model respond? Are there any lim itations for deploym ent? Can we fetch the data from DB quickly? BUSINESS UNDERSTANDING STAGE Can developers help us with deploym ent? Should we worry about features calculation time? 20
21. your model the system BUSINESS UNDERSTANDING STAGE 21
22. your model the system model object BUSINESS UNDERSTANDING STAGE 22
23. your model the system model object features calculation fetching raw data BUSINESS UNDERSTANDING STAGE 23
24. BUSINESS UNDERSTANDING STAGE Pay attention to: ! Model response time ! Feature calculation time ! Database response ! Human resources availability 24
25. DATA UNDERSTANDING STAGE You can: ! Limit data sources 25
26. DATA UNDERSTANDING STAGE 1 request per 1 min. 1 application per <1 min. 26
27. You can: ! Limit data sources ! Work closely with colleagues 27 DATA UNDERSTANDING STAGE
28. WORK CLOSELY WITH YOUR COLLEAGUES. 28
29. We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 29
30. Ouch… We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 30
31. Ouch… It was one of the top features :( We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 31
32. MODELLING You can: ! Refactor your code 32
33. REFACTOR YOUR CODE 1. Write your feature calculation 33
34. REFACTOR YOUR CODE 1. Write your feature calculation 2. Test the feature 34
35. REFACTOR YOUR CODE 1. Write your feature calculation 2. Test the feature 3. Improve your code readability 35
36. REFACTOR YOUR CODE 1. Write your feature calculation 4. Improve your code efficiency 2. Test the feature 3. Improve your code readability 36
37. REFACTOR YOUR CODE 1. Write your feature calculation 4. Improve your code efficiency 2. Test the feature 3. Improve your code readability 37
38. 38 REFACTOR YOUR CODE
39. MODELLING You can: ! Refactor your code ! Test model response 39
40. TEST MODEL RESPONSE TIME ExecuteTime from jupyter_contrib_nbextensions time module %%time 40
41. Keep track of versions. Always! but not like this… JUST TO KEEP IN MIND… 41
42. KEEP TRACK OF VERSIONS. 42
43. Document everything! So your future self and your teammates would be happy. JUST TO KEEP IN MIND… 43
44. DOCUMENT EVERYTHING ➔ keep clean Jupyter notebooks ➔ keep all changes in code 44
45. WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. 45
46. WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. ✍ Less risks in the late phases -> less postponed deadlines 46
47. WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. ✍ Easier to estimate tasks ✍ Less risks in the late phases -> less postponed deadlines 47
48. WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. ✍ Easier to estimate tasks ✍ Less risks in the late phases -> less postponed deadlines ✍ Better to understand responsibilities distribution 48
49. THANKS FOR YOUR ATTENTION AND HAPPY DATA SCIENCING!
50. CONTACTS marianna-diachuk @dark_matter88_ @mariaannadiachuk DarkMatter88
51. Q & A SESSION