Deploying models to production with TensorFlow Model Server

Slide 1

Slide 1 text

Event link: https://www.meetup.com/GDG-Ahmedabad/events/270477738/ Rishit Dagli 10-grade student, past TED-X and Ted-Ed speaker Deploying models to production with TensorFlow model server

Slide 2

Slide 2 text

Rishit Dagli RESEARCH RESEARCH RESEARCH RESEARCH RESEARCH

Slide 3

Slide 3 text

Rishit Dagli RESEARCH RESEARCH RESEARCH RESEARCH RESEARCH

Slide 4

Slide 4 text

Event link: https://www.meetup.com/GDG-Ahmedabad/events/270477738/ Rishit Dagli 10-grade student, past TED-X and Ted-Ed speaker Deploying models to production with TensorFlow model server

Slide 5

Slide 5 text

Ideal Audience ● Devs who have worked on Deep Learning Models (Keras) ● Devs looking for ways to put their model into production ready manner

Slide 6

Slide 6 text

01 Motivation behind a process for deployment What things to take care of? What is TF Model server? 02 03 04 05 06 What can it do? ● Versioning ● Iaas ● CI/ CD Auto Scaling QnA

Slide 7

Slide 7 text

Motivation behind a process for deployment Source: memegenerator.net

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

What things to take care of?

Slide 10

Slide 10 text

What things to take care of? ● Package the model

Slide 11

Slide 11 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server

Slide 12

Slide 12 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server

Slide 13

Slide 13 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server ○ Auto-scale

Slide 14

Slide 14 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server ○ Auto-scale

Slide 15

Slide 15 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server ○ Auto-scale ○ Global availability

Slide 16

Slide 16 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server ○ Auto-scale ○ Global availability ○ And many more ...

Slide 17

Slide 17 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server ○ Auto-scale ○ Global availability ○ And many more ... ● API

Slide 18

Slide 18 text

What things to take care of? ● Package the model ● Post the model on Cloud Hosted Server ● Maintain the server ○ Auto-scale ○ Global availability ○ And many more ... ● API ● Model Versioning

Slide 19

Slide 19 text

What is TF Model Server?

Slide 20

Slide 20 text

Serving

Slide 21

Slide 21 text

Serving

Slide 22

Slide 22 text

Credits: @lmoroney

Slide 23

Slide 23 text

Credits: @lmoroney

Slide 24

Slide 24 text

Credits: @lmoroney

Slide 25

Slide 25 text

What can it do?

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Serving

Slide 34

Slide 34 text

Installing https://www.tensorflow.org/tfx/serving/setup

Slide 35

Slide 35 text

The Process

Slide 36

Slide 36 text

Converting the model tf.saved_model.simple_save( keras.backend.get_session(), directory_path, inputs = {'input_image': model.input}, outputs = {i.name: i for i in model.outputs} )

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Starting the Model Server os.environ["MODEL_DIR"] = MODEL_DIR

Slide 39

Slide 39 text

Starting the model server os.environ["MODEL_DIR"] = MODEL_DIR %%bash --bg nohup tensorflow_model_server \ --rest_api_port = 8501 \ --model_name = test \ --model_base_path="${MODEL_DIR}" >server.log 2>&1

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Doing Inference!

Slide 45

Slide 45 text

Keep In Mind ● No data as lists but as lists of lists

Slide 46

Slide 46 text

Data as lists of lists xs = np.array([[case_1], [case_2] ... [case_n]])

Slide 47

Slide 47 text

Making calls xs = np.array([[case_1], [case_2] ... [case_n]]) data = json.dumps({"signature_name": " ", "instances": xs.tolist()})

Slide 48

Slide 48 text

Doing Inference xs = np.array([[case_1], [case_2] ... [case_n]]) data = json.dumps({"signature_name": " ", "instances": xs.tolist()}) json_response = requests.post( 'http://localhost:8501/v1/models/test:predict', data = data, headers = headers)

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

● High availability

Slide 53

Slide 53 text

● High availability ● No downtime

Slide 54

Slide 54 text

● High availability ● No downtime ● Focus on real code

Slide 55

Slide 55 text

● High availability ● No downtime ● Focus on real code ● Build better apps

Slide 56

Slide 56 text

Serve the model on Cloud

Slide 57

Slide 57 text

Why Care?

Slide 58

Slide 58 text

Why Care?

Slide 59

Slide 59 text

Creating a Kubernetes cluster gcloud container clusters create resnet-serving-cluster --num-nodes 5

Slide 60

Slide 60 text

Pushing the docker image docker tag $USER/resnet_serving gcr.io/tensorflow-serving/resnet docker push gcr.io/tensorflow-serving/resnet

Slide 61

Slide 61 text

Pushing the docker image kubectl create -f [yaml]

Slide 62

Slide 62 text

● Use the external IP Inference

Slide 63

Slide 63 text

gdg-ahm.rishit.tech Code Repo

Slide 64

Slide 64 text

Demos

Slide 65

Slide 65 text

Key Takeaways ● Why a process for deployment ● What it takes to deploy models ● Serving a model with TF Model server ● Why TF Model server? ● What can TF Model server do? ● Deploying on Cloud

Slide 66

Slide 66 text

Rishit Dagli Rishit-dagli rishit_dagli rishit.tech [email protected] @rishit.dagli About Me