Applying deployment oriented mindset for building Machine Learning models at Data Science fwdays '19

Developing a complicated ensemble model with hundreds of features fetched from a bunch of different sources? Give me two! Showing great metrics to the stakeholders and already discussing how it will hit a home run in production? Why not! And then getting stuck for months trying to deploy the model and fighting with data inconsistency and bugs? Sounds familiar?

This talk will focus on providing guidelines on how to build your model development process keeping in mind the deployment phase to come later on.

Transcript

1. APPLYING DEPLOYMENT ORIENTED MINDSET FOR BUILDING MACHINE LEARNING MODELS Marianna Diachuk WOMEN WHO CODE KYIV
2. AGENDA ➔ Why am I telling you this? ➔ What I mean by deployment? ➔ Deployment problems ➔ Model development process ➔ What deployment oriented mindset gives you 2
3. WHY AM TELLING YOU THIS? - I developed and deployed to production 7 scoring and antifraud ensemble models - I leaded a small but proud team of 2 data scientists and 1 data engineer - I’m a mother of 3 dragons ducks 3
4. WHY AM TELLING YOU THIS? WHAT WHY 4
5. WHAT I MEAN BY DEPLOYMENT? 5
6. I HAD MY SHARE OF FAILS TOO… 6
7. DEPLOYMENT PROBLEMS (SOME OF THEM) ➢ Model response inconsistency 7 research environment development environment
8. DEPLOYMENT PROBLEMS (SOME OF THEM) ➢ Model response inconsistency 8 research environment development environment dataset
9. DEPLOYMENT PROBLEMS (SOME OF THEM) ➢ Features inconsistency 9 research environment development environment dataset
10. DEPLOYMENT PROBLEMS (SOME OF THEM) ➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency ➢ Model is not scalable and so on and so on… 10
11. MODEL DEVELOPMENT PROCESS LOOKS LIKE... 11
12. OR LIKE THIS... Research process Agile process 12
13. MODEL DEVELOPMENT PROCESS Remember about deployment. 13
14. How fast should our model respond? Are there any lim itations for deploym ent? Can we fetch the data from DB quickly? BUSINESS UNDERSTANDING STAGE Can developers help us with deploym ent? Should we worry about features calculation time? 14
15. your model the system model object features calculation fetching raw data BUSINESS UNDERSTANDING STAGE 15
16. BUSINESS UNDERSTANDING STAGE Pay attention to: ! Model response time ! Feature calculation time ! Database response ! Human resources availability 16
17. DATA UNDERSTANDING STAGE You can: ! Limit data sources 17
18. DATA UNDERSTANDING STAGE 1 request per 1 min. 1 application per <1 min. 18
19. DATA UNDERSTANDING STAGE You can: ! Limit data sources ! Work closely with colleagues 19
20. Ouch… It was one of the top features :( We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 20
21. MODELLING You can: ! Refactor your code 21
22. REFACTOR YOUR CODE 1. Write your feature calculation 4. Improve your code efficiency 2. Test the feature 3. Improve your code readability 22
23. REFACTOR YOUR CODE 23
24. MODELLING You can: ! Refactor your code ! Test model response 24
25. TEST MODEL RESPONSE TIME ExecuteTime from jupyter_contrib_nbextensions time module %%time 25
26. Keep track of versions. Always! but not like this JUST TO KEEP IN MIND… 26
27. KEEP TRACK OF VERSIONS. 27
28. Document everything! So your future self and your teammates would be happy. JUST TO KEEP IN MIND… 28
29. DOCUMENT EVERYTHING ➔ keep clean Jupyter notebooks ➔ keep all changes in code 29
30. WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. - Easier to estimate tasks - Less risks in the late phases -> less postponed deadlines - Better to understand responsibilities distribution 30
31. THANKS FOR YOUR ATTENTION AND HAPPY DATA SCIENCING!
32. CONTACTS marianna-diachuk @dark_matter88_ @mariaannadiachuk DarkMatter88
33. Q & A SESSION