Tensorflow serving Introduction

Transcript

1. PRODUCTION ML WITH TENSORFLOW SERVING David Kenney
2. WHO AM I? ➤ Software engineer on the Machine Learning team at Cookpad ➤ Mainly working with image related stuff ➤ Studied Mechanical Engineering at University with some machine learning
3. OUTLINE ➤ What is Tensorflow serving ➤ Is it for me? ➤ How can I get it working? ➤ Simplifying with Docker ➤ Model Export ➤ Client requests ➤ Exporting from Keras
4. WHAT IS TENSORFLOW SERVING ➤ If you have models created in Tensorflow, you can serve them over the internet with tensorflow serving ➤ Tensorflow serving acts as a web-host capable of inference Client (mobile App, internal service, etc.) Tensorflow Serving User photo (JPEG) Binary Protobuf File Requested Model params Ex. model version, output layer Binary Protobuf File Send request to inference server Model Params Photo Tensorflow Model Result Result passed back to client
5. WHAT ARE PROTOBUFS ➤ Protocol Buffer format ➤ If you’ve worked with saving tensorflow models (as .pb files) then you’re already familiar with them ➤ A binary format capable of storing structured information, which is described in a matching .proto file ➤ For example, Tensorflow has a standard .proto for saving models ➤ For tensorflow serving, can contain our inference request, encapsulating our image and request for classification
6. IS IT FOR ME? ➤ Tensorflow Serving is written in C++ with performance in mind (Can use GPUs as well) ➤ Communicates with gRPC (Remote Procedure Call) ➤ Supports batching to allow inference over a range of images ➤ Alternatives include writing your own inference web host ➤ You can use software like Flask and Bottle for hosting a webserver in python that can also run a model for inference ➤ No need to use protobuf format, can write endpoints to accept images in any form (binary jpeg, base64) and convert on backend
7. HOW CAN I USE IT (BASIC STEPS) ➤ Clone tensorflow serving github ➤ Use Bazel to configure and build tensorflow serving from source (currently TensorFlow serving only supports linux) ➤ You can test serving by generating it’s sample ImageNet inception model, and then classifying an image
 
 Next steps ➤ Export your own tensorflow models in the SavedModel format ➤ Deploy! TensorFlow Serving example export code: https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/inception_saved_model.py
8. MAKE YOUR LIFE SIMPLER WITH DOCKER ➤ Bazel can be tricky to work with, and environment specific problems can easily popup ➤ Especially going from development to deployment! ➤ Included with Tensorflow serving is a Dockerfile which enables you to build tensorflow in an environment including dependencies, that’s been tested Official Docker Guide: https://tensorflow.github.io/serving/docker.html
9. WHAT’S IN THE DOCKERFILE ➤ The Dockerfile tells Docker how to create an image which will hold all of the dependencies that Tensorflow serving needs to work with ➤ It WON’T contain a working instance of tensorflow serving itself ➤ You still need to configure and build it with Bazel
10. FROM ubuntu:16.04 MAINTAINER Craig Citro <craigcitro@google.com> RUN apt-get update && apt-get install -y --no- install-recommends \ build-essential \ curl \ git \ libcurl3-dev \ libfreetype6-dev \ libpng12-dev \ libzmq3-dev \ pkg-config \ python-dev \ rsync \ software-properties-common \ unzip \ zip \ zlib1g-dev \ openjdk-8-jdk \ openjdk-8-jre-headless \ && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* RUN curl -fSsL -O https://bootstrap.pypa.io/ get-pip.py && \ python get-pip.py && \ rm get-pip.py RUN pip --no-cache-dir install \ ipykernel \ jupyter \ matplotlib \ numpy \ scipy \ sklearn \ pandas \ && \ python -m ipykernel.kernelspec # Set up our notebook config. COPY jupyter_notebook_config.py / root/.jupyter/ # Jupyter has issues with being run directly: # https://github.com/ipython/ipython/issues/ 7062 # We just add a little wrapper script. COPY run_jupyter.sh / # Set up Bazel. # Running bazel inside a `docker build` command causes trouble, cf: # https://github.com/bazelbuild/bazel/issues/ 134 # The easiest solution is to set up a bazelrc file forcing --batch. RUN echo "startup --batch" >>/etc/bazel.bazelrc # Similarly, we need to workaround sandboxing issues: # https://github.com/bazelbuild/bazel/issues/ 418 RUN echo "build --spawn_strategy=standalone -- genrule_strategy=standalone" \ >>/etc/bazel.bazelrc # Install the most recent bazel release. ENV BAZEL_VERSION 0.5.0 WORKDIR / RUN mkdir /bazel && \ cd /bazel && \ curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -O https://github.com/bazelbuild/bazel/releases/ download/$BAZEL_VERSION/bazel-$BAZEL_VERSION- installer-linux-x86_64.sh && \ curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -o / bazel/LICENSE.txt https:// raw.githubusercontent.com/bazelbuild/bazel/ master/LICENSE && \ chmod +x bazel-*.sh && \ ./bazel-$BAZEL_VERSION-installer-linux- x86_64.sh && \ cd / && \ rm -f /bazel/bazel-$BAZEL_VERSION-installer- linux-x86_64.sh # Download and build TensorFlow. RUN git clone https://github.com/ tensorflow/tensorflow.git && \ cd tensorflow && \ git checkout r1.2 WORKDIR /tensorflow # TODO(craigcitro): Don't install the pi package, since it makes it # more difficult to experiment with loca changes. Instead, just add # the built directory to the path. ENV CI_BUILD_PYTHON python RUN tensorflow/tools/ci_build/builds/ configured CPU \ bazel build -c opt --cxxopt="- D_GLIBCXX_USE_CXX11_ABI=0" \ tensorflow/tools/ pip_package:build_pip_package && \ bazel-bin/tensorflow/tools/ pip_package/build_pip_package /tmp/pip & \ pip --no-cache-dir install --upgrade tmp/pip/tensorflow-*.whl && \ rm -rf /tmp/pip && \ rm -rf /root/.cache # Clean up pip wheel and Bazel cache whe done. # TensorBoard EXPOSE 6006 # IPython EXPOSE 8888 WORKDIR /root CMD ["/bin/bash"]
11. MORE DOCKER INFORMATION ➤ Building a Dockerfile creates an image, a static environment with all the instructions that were defined in the Dockerfile finished 
 [$docker build] ➤ Dependencies installed, files copied to the image, environment variables defined ➤ Running the dockerfile produces a container [$ docker run] ➤ A container is a running instance of a image. ➤ When running the image you can additionally define mappings for local directories to map inside the container, or ports that will be opened up to the host ➤ Docker provides no access to environment specific hardware like a GPU ➤ Must use Nvidia-docker to access a system GPU
12. EXPORTING YOUR MODEL FOR TENSORFLOW SERVING ➤ Load model in Tensorflow ➤ Define the signature definition ➤ i.e. Which model layer maps to the default input, what the default classification layer is ➤ Call the SavedModelBuilder and pass it your tensorflow session, and the default signature definition map
13. SERVING SIGNATURES ➤ When exporting your model, take note of the signatures used such as the name of the output layer ➤ You can define defaults using the default signature constants that tensorflow defines ➤ Can also add your own custom signatures ➤ When your client makes an inference request it uses these same signatures in the request classification_signature = ( tf.saved_model.signature_def_utils.build_signature_def( inputs={ tf.saved_model.signature_constants.CLASSIFY_INPUTS: classify_inputs_tensor_info }, outputs={ tf.saved_model.signature_constants.CLASSIFY_OUTPUT_CLASSES: classes_output_tensor_info, tf.saved_model.signature_constants.CLASSIFY_OUTPUT_SCORES: scores_output_tensor_info }, method_name=tf.saved_model.signature_constants.CLASSIFY_METHOD_NAME)) Serving signature reference: https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/signature_defs.md
14. VERSIONING AND FOLDER INFORMATION ➤ Tensorflow serving supports model versioning, where it can ‘load' different versions of a model at the same time ➤ Clients can request a specific version of a model (ex. For A/B testing, or if input type changes) ➤ Also helps when upgrading or rolling back models because this can be done in place, without reloading the server ➤ Tensorflow requires the folder structure with:
 {model_version}/{saved_model_file}
 {model_version}/variables/{variable_files}
15. SERVING MULTIPLE MODELS FROM THE SAME SERVER ➤ You can configure Tensorflow serving to use multiple models if you set up a model_config_file ➤ Drawbacks include needing to restart the server when adding new models (to load the new config file) ➤ Call server with flag:
 - -config_file model_config.conf model_config_list: { config: { name: "my_model_1", base_path: "/tmp/model_1", model_platform: "tensorflow" }, config: { name: "my_model_2", base_path: "/tmp/model_2", model_platform: "tensorflow" } } model_config.conf
16. TENSORFLOW DERIVATIVE MODELS ➤ If you work primarily with Keras models, you can serve these with tensorflow as well, but it requires more work ➤ Models can be exported with the same SavedModel builder, after manually defining signatures and layer mappings. import keras.backend as K from keras import models from tensorflow.python.saved_model import builder as saved_model_builder from tensorflow.python.saved_model import utils, signature_constants from tensorflow.python.saved_model.signature_def_utils_impl import build_signature_def model_version = 20 model = models.load_model('keras_model.h5py') path = os.path.join('model_export', str(model_version)) builder = saved_model_builder.SavedModelBuilder(path) info_input = utils.build_tensor_info(new_model.input) info_output = utils.build_tensor_info(new_model.output) signature = build_signature_def( inputs={'images': info_input}, outputs={signature_constants.CLASSIFY_OUTPUT_SCORES: info_output}, method_name=signature_constants.CLASSIFY_METHOD_NAME) with K.get_session() as sess: builder.add_meta_graph_and_variables( sess=sess, tags=[tag_constants.SERVING], signature_def_map={ signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature, }) builder.save() This code not tested for inference, only for loading keras models in TF Serving. 
 More work might be needed..
17. SOME ISSUES ➤ Many guides and tutorials on the internet were written before the current ‘SavedModel’ builder was implemented in Tensorflow, and generate models that are incompatible with the current versions of Tensorflow serving ➤ While you can generate the protobuf request in environments that you control (like mobile applications, on your server) things are more difficult if you’re dealing with a webpage running in the browser where traditional file uploads (ex. XMLHttpRequest) make more sense ➤ To solve this, you can use Tensorflow serving as your own inference server, and run your client requests through a gateway server that handles formatting, etc.
18. CONCLUSION ➤ Tensorflow serving gives you a out-of-the box option for hosting Tensorflow models in a standard way ➤ Can setup easily with Docker (and extend to remote hosting, ex. AWS, Kubernetes) ➤ Extends to handle model versioning and multiple models ➤ For smaller applications bespoke python web server may make more sense ➤ If you want to run a server for your Tensorflow models, give it a shot!
19. THANK YOU AND ACKNOWLEDGEMENTS ➤ Any questions? Thanks to: - Google for making Tensorflow and Tensorflow serving - All the people on Stack Overflow and Github issues with solutions to the many issues/ misunderstandings I had while working with Tensorflow Serving, grpc and protobufs