Machines Can Learn - a Practical Take on Machine Intelligence Using Spring Cloud Data Flow and TensorFlow By Christian Tzolov @christzolov 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Disclaimer 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 3 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Does It Matter? 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Rise of the machines • Machine learning brings unprecedented abilities to the Software Engineering field • Provides different way to reason about problems • Solves “un-programmable” tasks • But there is a technological gap (BLAS/CPU, CUDA/GPU, C++/Python) • And cultural difference J • How Machine Learning can enable us deliver richer business solutions? 5 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Real-Time Predictions with TensorFlow and Spring Cloud Data Flow (SCDF) • Supervised, Deep Learning (DL) • TensorFlow • Model Inference (e.g. Evaluation, Prediction) • (Near) Real-Time - Streaming • Java and Spring 6 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Can Machine Learn? 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Classical Programming 8 Classical Programming Rules Data Answers 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Machine Learning Paradigm 9 Machine Learning Rules’ Data* Answers* * Historical, annotated Datasets 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ ML Lifecycle 10 Model Inference Output Predictions MODEL (e.g. Rules) New Data Model Training Annotated answers Historical Data Offline Online (Production) Software Engineering & Ops Data Science 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Remote Server Model Serving & Embedding 11 Data Source HTTP/REST, RPC Processor(s) Data Sink Data Source Processor(s) Model Data Sink Runtime Model Remote Serving In-Pipeline Embedded 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deep Learning Inference with Java • Deep Learning Java Runtime • Portable Models 12 DL Java Runtime Input Data Output Predictions Pre-trained Model (e.g. Rules) 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deep Learning Frameworks 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deep Learning Frameworks • Tensorflow (Google) • PyTorch • CNTK (Microsoft) • Caffe2 (Facebook) • MXNet (Amazon) • DeepLearning4J • Keras 14 Deep Learning Java Runtime 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Real-Time Model Evaluation with Spring Cloud Data Flow 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Transport Middleware Stream A Spring Cloud Data Flow (SCDF) Toolkit for building data integration, real-time, and batch data processing pipelines • Multi-platform support (PCF, Kubernetes) • Comprehensive app lifecycle (Skipper) • Pipes & Filters like DSL: Source | Processor … | Sink Source Processor Processor Sink 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Localizing and identifying multiple objects in a single image • TF Object Detection API: http://bit.ly/tensorflow-object-detection-api • Pre-trained models: http://bit.ly/object-detection-api-model-zoo • Object Detection Processor Doc: http://bit.ly/scdf-object-detection-processor • Blog: https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream Object Detection Processor file --directory=‘/tmp/scdf/od/input’ | object-detection --mode=header --model='file:/model.pb’ --labels='file:/label_map.pbtxt’ | image-viewer 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ TensorFlow SCDF Processor 21 TensorFlowService Message payload headers Output Converter Input Converter Message payload headers Map Load from file:// , http://, classpath:// TensorFlow Build/Train/Export TensorFlow Runtime TF Model Offline Real Time 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Object Detection - RT Demo 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Available TensorFlow Processors • Object Detection – TF Object Detection API: [12] [13] • Instance Segmentation, TF Object Detection API: [11] • Semantic Segmentation, TF API: [8][9][10] • Image Recognition (Inception V3), TensorFlow API: [7] • Pose Estimation (OpenPose), TensorFlow API [5], [6] • Twitter Sentiment Analysis, TensorFlow API: [4] • Face Detection (MTCNN), ND4J, ND4J-Tensorflow, JavaCV: [1], [2] • Face Recognition (FaceNet), TensorFlow API: [3] 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Quick Starters • TensorFlow Spring Cloud Stream App Starters: https://github.com/spring-cloud-stream-app-starters/tensorflow • Computer Vision Projects (experimental): https://github.com/tzolov/computer-vision • SCDF TensorFlow Demos (Docker): https://github.com/tzolov/scdf-tensorflow-demos • Real-time Object-Detection With Spring Cloud Stream https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream • Real-time Object-Detection With Spring Cloud Stream https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream • Real-Time Multi-Person, Human Pose Estimation with Spring Cloud Data Flow and TensorFlow http://bit.ly/scdf-pose-estimationn-blog 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Conclusions 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Convinced? • Data Science and Deep Learning mature at a rapid pace • Pivotal is committed to make them easily accessible for Spring/Java developers. • Way to start: Check the TensorFlow processors for Spring Cloud Data Flow • You feedback is appreciated! 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Next 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unofficial agenda • Top level TensorFlow (or DL) project – move the utilities and existing implementations (such as Image Recognition, Object Detection, Sentiment Analysis) out of SCSt into a top level, generic projects. • Use with Spring Cloud Stream, Spring Cloud Function, Spring Cloud Batch • Research “Transfer Learning” possibilities • Evaluate Deeplearning4J, ND4J as Runtime alternative • ONNX Runtime to use models from other DL frameworks • REST, Spring Boot as remote DL serving application 39 

Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ References [1] Face Detection Lib: https://github.com/tzolov/mtcnn-java [2] Face Detection Processor: https://github.com/tzolov/computer-vision/blob/master/spring-cloud-starter-stream-processor-face-detection-mtcnn [3] Face Recognition: https://github.com/tzolov/computer-vision/tree/master/spring-cloud-starter-stream-processor-face-recognition [4] Twitter Sentiment Analysis: https://github.com/spring-cloud-stream-app-starters/tensorflow/blob/master/spring-cloud-starter-stream-processor-twitter-sentiment [5] Pose Estimation Processor: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-pose-estimation [6] Pose Estimation Blog: https://content.pivotal.io/blog/making-data-science-accessible-to-developers-real-time-multi-person-human-pose-estimation-with-spring-cloud- data-flow-and-tensorflow [7] Image Recognition: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-image-recognition [8] Semantic Segmentation Blog: https://goo.gl/dtwAfo [9] Semantic Segmentation - Java2D: https://github.com/tzolov/computer-vision/tree/master/spring-cloud-starter-stream-processor-semantic-segmentation [10] Semantic Segmentation - JavaCV: https://github.com/tzolov/computer-vision/tree/master/spring-cloud-starter-stream-processor-semantic-segmentation-cv [11] Instance Segmentation - Object Detection Processor with Masks: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter- stream-processor-object-detection [12] Object Detection Processor: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-object-detection [13] Object Detection Blog: https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream 40 

> Stay Connected.