PyCon & PyData. Applying deployment oriented mindset for building Machine Learning models

Slide 1

Slide 1 text

APPLYING DEPLOYMENT ORIENTED MINDSET FOR BUILDING MACHINE LEARNING MODELS MARIANNA DIACHUK

Slide 2

Slide 2 text

AGENDA ➔ Why am I telling you this? ➔ What I mean by deployment? ➔ Deployment problems ➔ Model development process ➔ What deployment oriented mindset gives you 2

Slide 3

Slide 3 text

ABOUT ME ✍ I’m a data scientist from Kyiv (Ukraine) ✍ I developed and deployed multiple scoring and antifraud ensemble models ✍ I leaded a small but proud team of 2 data scientists and 1 data engineer ✍ I’m a mother of 3 dragons ducks 3

Slide 4

Slide 4 text

4 WHY AM TELLING YOU THIS?

Slide 5

Slide 5 text

WHAT 5 WHY AM TELLING YOU THIS?

Slide 6

Slide 6 text

WHAT WHY 6 WHY AM TELLING YOU THIS?

Slide 7

Slide 7 text

WHAT WHY 7 WHY AM TELLING YOU THIS?

Slide 8

Slide 8 text

WHAT I MEAN BY DEPLOYMENT? 8

Slide 9

Slide 9 text

WHAT I MEAN BY DEPLOYMENT? 9

Slide 10

Slide 10 text

I HAD MY SHARE OF FAILS TOO… 10

Slide 11

Slide 11 text

DEPLOYMENT PROBLEMS (SOME OF THEM) ➢ Model response inconsistency 11 research environment development environment

Slide 12

Slide 12 text

➢ Model response inconsistency 12 research environment development environment dataset DEPLOYMENT PROBLEMS (SOME OF THEM)

Slide 13

Slide 13 text

➢ Features inconsistency 13 research environment development environment dataset DEPLOYMENT PROBLEMS (SOME OF THEM)

Slide 14

Slide 14 text

➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency 14 DEPLOYMENT PROBLEMS (SOME OF THEM)

Slide 15

Slide 15 text

➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency ➢ Model is not scalable and so on and so on… 15 DEPLOYMENT PROBLEMS (SOME OF THEM)

Slide 16

Slide 16 text

➢ Model response inconsistency ➢ Impossibility to implement features calculations ➢ Features inconsistency ➢ Model is not scalable and so on and so on… 16 DEPLOYMENT PROBLEMS (SOME OF THEM)

Slide 17

Slide 17 text

MODEL DEVELOPMENT PROCESS LOOKS LIKE... 17

Slide 18

Slide 18 text

OR LIKE THIS... Research process Agile process 18

Slide 19

Slide 19 text

MODEL DEVELOPMENT PROCESS 19 Remember about deployment.

Slide 20

Slide 20 text

How fast should our model respond? Are there any lim itations for deploym ent? Can we fetch the data from DB quickly? BUSINESS UNDERSTANDING STAGE Can developers help us with deploym ent? Should we worry about features calculation time? 20

Slide 21

Slide 21 text

your model the system BUSINESS UNDERSTANDING STAGE 21

Slide 22

Slide 22 text

your model the system model object BUSINESS UNDERSTANDING STAGE 22

Slide 23

Slide 23 text

your model the system model object features calculation fetching raw data BUSINESS UNDERSTANDING STAGE 23

Slide 24

Slide 24 text

BUSINESS UNDERSTANDING STAGE Pay attention to: ! Model response time ! Feature calculation time ! Database response ! Human resources availability 24

Slide 25

Slide 25 text

DATA UNDERSTANDING STAGE You can: ! Limit data sources 25

Slide 26

Slide 26 text

DATA UNDERSTANDING STAGE 1 request per 1 min. 1 application per <1 min. 26

Slide 27

Slide 27 text

You can: ! Limit data sources ! Work closely with colleagues 27 DATA UNDERSTANDING STAGE

Slide 28

Slide 28 text

WORK CLOSELY WITH YOUR COLLEAGUES. 28

Slide 29

Slide 29 text

We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 29

Slide 30

Slide 30 text

Ouch… We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 30

Slide 31

Slide 31 text

Ouch… It was one of the top features :( We removed one field from the application form. WORK CLOSELY WITH YOUR COLLEAGUES. 31

Slide 32

Slide 32 text

MODELLING You can: ! Refactor your code 32

Slide 33

Slide 33 text

REFACTOR YOUR CODE 1. Write your feature calculation 33

Slide 34

Slide 34 text

REFACTOR YOUR CODE 1. Write your feature calculation 2. Test the feature 34

Slide 35

Slide 35 text

REFACTOR YOUR CODE 1. Write your feature calculation 2. Test the feature 3. Improve your code readability 35

Slide 36

Slide 36 text

REFACTOR YOUR CODE 1. Write your feature calculation 4. Improve your code efficiency 2. Test the feature 3. Improve your code readability 36

Slide 37

Slide 37 text

REFACTOR YOUR CODE 1. Write your feature calculation 4. Improve your code efficiency 2. Test the feature 3. Improve your code readability 37

Slide 38

Slide 38 text

38 REFACTOR YOUR CODE

Slide 39

Slide 39 text

MODELLING You can: ! Refactor your code ! Test model response 39

Slide 40

Slide 40 text

TEST MODEL RESPONSE TIME ExecuteTime from jupyter_contrib_nbextensions time module %%time 40

Slide 41

Slide 41 text

Keep track of versions. Always! but not like this… JUST TO KEEP IN MIND… 41

Slide 42

Slide 42 text

KEEP TRACK OF VERSIONS. 42

Slide 43

Slide 43 text

Document everything! So your future self and your teammates would be happy. JUST TO KEEP IN MIND… 43

Slide 44

Slide 44 text

DOCUMENT EVERYTHING ➔ keep clean Jupyter notebooks ➔ keep all changes in code 44

Slide 45

Slide 45 text

WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. 45

Slide 46

Slide 46 text

WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. ✍ Less risks in the late phases -> less postponed deadlines 46

Slide 47

Slide 47 text

WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. ✍ Easier to estimate tasks ✍ Less risks in the late phases -> less postponed deadlines 47

Slide 48

Slide 48 text

WHAT DEPLOYMENT ORIENTED MINDSET GIVES YOU. ✍ Easier to estimate tasks ✍ Less risks in the late phases -> less postponed deadlines ✍ Better to understand responsibilities distribution 48