ML KIT & TensorFlow Lite

Let's discover the potentialities introduced by ML Kit for Firebase in interacting with the outside world. We will use the APIs provided by Google integrating them into our code with a few lines of code for recognizing realtime face contours, the "sentiment" of a person, famous monuments through photos and more. We will also show how to generate personal models by exporting them in the TensorflowLite format. We will use models of the YOLO family (you only look once) for the training and recognition of custom objects, thus specializing the recognition only for objects of interest and / or increasing the accuracy of those recognized by Google's basic API.

Transcript

1. ML KIT & TensorFlow Lite Rendiamo i nostri phone realmente smart
2. HELLO! I am Enrico Sallusti Senior Mobile Engineer @Kalpa Srl enrico.sallusti@kalpa.it 2
3. “ Machine learning is the scientific study of algorithms and statistical models that Computer Systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. 3 CAN MACHINES THINKS?
4. 4 Is a mobile SDK that makes easy to add Machine Learning functionality in a mobile app (both iOS and Android) ▸ Suitable for experts and for dummies ▸ Provides ready to use APIs for common use cases ▸ Offers on device processing (Free) and cloud processing (Pay-Per-Use) ▸ Integrated with TensorFlow
5. 5 Available Features
6. TEXT RECOGNITION 6 ▸ Uses OCR to recognize text from images ▸ Two different APIs ▹ General purpose (Street sign, images) text recognition ▹ On-Device: Recognize Latin chars ▹ Cloud-Based: Recognize also Arabic, Chinese, Cyrillic, etc. ▹ Uses Text Blocks ▹ Optimized for Document text recognition ▹ Only Cloud-Based ▹ Recognizes Words, Paragraphs, Blocks and Symbols
7. 7 BARCODE SCANNING ▸ Based on the old Google Mobile Vision APIs to recognize Barcodes and QR Codes ▸ Recognizes most standard formats (linear and 2D) ▸ No need to specify the barcode format to the scanner ▸ SSID, SMS, URL, ISBN value types automatically classified
8. 8 LANGUAGE IDENTIFICATION ▸ Recognize the language of a string of text ▹ 103 different languages ▸ In case of difficult texts the API gives a list of possible languages with their confidence scores ▸ The language codes are returned in the BCP-47 code standard https://firebase.google.com/docs/ml-kit/langid- support
9. 9 LANDMARK RECOGNITION ▸ Uses Cloud Vision API to recognize well-known landmarks ▸ Only available on the cloud (no On-device support) ▸ The APIs returns the Common name of the Landmark and the Knowledge Graph entity ID ▹ The ID can be used with the Google Knowledge Graph Search APIs in order to get more information on the landmark ▹ https://developers.google.com/knowledge- graph/
10. 10 FACE DETECTION ▸ Detect one or more human faces in images and videos and locates the coordinates and rotations of different facial features ▸ Can be used to track the contours of the facial features ▸ Can be used to get the face “sentiment” based on the probabilities of facial features like: ▹ Smile ▹ Left Eye opened ▹ Right Eye opened ▸ Two settings ▹ Fast Mode: to be used in Real-time detection and video frame detection ▹ Accurate Mode: to be used in Image detection
11. 11 IMAGE LABELING ▸ Return a list of entities recognized in the image with their confidence scores ▸ Fast but unreliable on device (400+ labels) ▸ Slow but accurate on the cloud (10.000+ labels) ▸ Return a Knowledge Graph Entity ID if it is present for a certain label exactly like the landmark recognition ▸ Can be extended using Custom Models
12. 12 Ciao! I am Danilo Sia Software Architect @Kalpa Srl danilo.sia@kalpa.it
13. 13 CUSTOM MODEL Bring it on!
14. 14 CUSTOM MODEL Hurt me plenty. ▸ Select a model as custom model ▸ Build and train a custom model using TensorFlow or, re- train an existing model that solves a problem similar to what you want to achieve. ▸ Convert the model to TensorFlow Lite ▸ Use the TensorFlow Lite model for inference
15. 15 CUSTOM MODEL Hurt me plenty. ▸ Object detection
16. 16 CUSTOM MODEL Hurt me plenty. The difference between image classification and object detection
17. 17 CUSTOM MODEL Hurt me plenty. We can thus think of image classification as: • One image in • And one class label out When performing object detection, given an input image, we wish to obtain: • A list of bounding boxes, or the (x, y)-coordinates for each object in an image • The class label associated with each bounding box • The probability/confidence score associated with each bounding box and class label Therefore, object detection allows us to: • Present one image to the network • And obtain multiple bounding boxes and class labels out
18. 18 CUSTOM MODEL Hurt me plenty. Yolo Network Architecture
19. 19 CUSTOM MODEL Hurt me plenty.
20. 20 CUSTOM MODEL Hurt me plenty.
21. 21 CUSTOM MODEL Hurt me plenty. Toco/tflite_ converter: bazel build tensorflow/contrib/lite/toco:toco ▸ Annotate the photos (labeling) ▸ Prepare dataset metadata ▸ Training ▸ Convert the model (Darkflow). Convert .weights to a frozen graph ▸ Convert protobuf to tflite
22. 22 CUSTOM MODEL Demo time
23. 23 CUSTOM MODEL A bit of Ultra-Violence. Select TensorFlow custom operators to use in TensorFlow Lite SELECT_TF_OPS - Converts models using TensorFlow ops. The exact subset of supported ops can be found in the whitelist at lite/toco/tflite/whitelisted_flex_ops.cc bazel run --define=with_select_tf_ops=true tensorflow/lite/python:tflite_convert -- --graph_def_file=yolov2-tiny-kpoke.pb --output_file=yolov2-tiny-kpoke.tflite - -input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --input_arrays=input -- input_shapes=1,416,416,3 --output_arrays=output --enable_select_tf_ops -- target_ops=TFLITE_BUILTINS,SELECT_TF_OPS bazel build --cxxopt='--std=c++11' -c opt --config=android_arm -- config=monolithic //tensorflow/lite/java:tensorflow-lite-with-select-tf-ops will generate an AAR file in bazel-genfiles/tensorflow/lite/java/
24. REFERENCES 24 ▸ Firebase Console https://console.firebase.google.com/u/0/ ▸ MLKit Docs https://firebase.google.com/docs/ml-kit/ ▸ TensorFlow Lite https://www.tensorflow.org/lite ▸ Darknet https://pjreddie.com/darknet/yolo/ ▸ Darkflow https://github.com/thtrieu/darkflow ▸ Yolo https://hackernoon.com/understanding-yolo-f5a74bbc7967
25. THANK YOU! DOMANDEEEEEEEE? 25