Making Android Apps with Intelligence Margaret Maynard-Reid, July 29 2016 @margaretmz +MargaretMaynardReid @margaretmz

Machine learning everywhere! 2

What is Machine Learning? ● A subset of Artificial Intelligence ● Study of algorithms ● Learn from examples and experience (instead of hard-coded programming rules) Source: ML Recipes #1 by Google Developers - https://www.youtube.com/watch?v=cKxRvEZd3Mw 3 Collect training data Train a model Use model to predict

Topics covered today 4 Complexity Other topics ● Amazon Alexa ● Pepper the Robot ● Design considerations ML Services (REST APIs) Vision Speech Language Text… (Google, MS, IBM, HP…) ML Services (build or train your ML models) Google Cloud ML Platform TensorFlow Amazon ML Google Play services Awareness API Mobile Vision API

Google Play services 5

Google Awareness API ● Unifies 7 signals into one API - time, location, place, activity, beacons, headphones & weather ● Two options - ○ Snapshot API: current state ○ Fence API: state change I/O YouTube recording: https://youtu.be/37ia7S4Lsv4 https://developers.google.com/awareness/ 6

Getting started ● Add this to app build.gradle file ● Get an Awareness API key from Google dev console ● Add the API key in manifest ● (Optional) add required permissions in manifest ● (Optional) enable additional APIs for Places and Beacon states Demo: SampleAwareness 7 com.google.android.gms:play-services-contextmanager:9.2.1

Google Mobile Vision API ● Part of the Google Play services ● Very easy to get started ● Can detect: ○ Face (FaceDetector) | codelab ○ Barcode (BarcodeDetector) | codelab ○ Text (TextRecognizer) | codelab https://developers.google.com/vision/ 8

Face API ● Facial detection not recognition ● Detection from image or live video ● Face can be detected from different angles ● Detects activities: ○ winking, blinking and smiling ● Detect landmarks on face 9

Face detector Add this to app build.gradle file Create a detector Make sure detector is operational first 10 com.google.android.gms:play-services-vision:9.2.1 FaceDetector faceDetector = new FaceDetector.Builder(getApplicationContext()) .setTrackingEnabled(false) .build(); if(!faceDetector.isOperational()){ // let user know face detector isn’t working // log error etc... }

Face detector Detect the face Draw Rectangles on the Faces Refer to code lab for details... Demo: Multi tracker and OCR reader 11 Frame frame = new Frame.Builder().setBitmap(myBitmap).build(); SparseArray faces = faceDetector.detect(frame);

Machine Learning Services - via REST APIs 12

Machine learning services Pre-trained models No need for much ML knowledge Just REST API calls “Try the API” directly online from the websites ● Google Cloud ML APIs ● Microsoft Cognitive Services ● IBM Watson ● HPE Haven OnDemand 13

Getting started Free trial based on time or # of transactions A typical process: ● Get API key ● Include key in Android app ● Download SDK or include dependency in build.gradle (if sdk available) ● Make your REST API calls 14

REST API calls 15 ML REST APIs Vision Speech Language Text Translate Search…. HTTP response images text HTTP call audio video

Microsoft Vision APIs Vision APIs ● Computer Vision, Emotion & Face Can detect ● Gender ● Age ● Emotions etc. Demo: SampleVision - image caption 16

Demo - Mimicker Alarm Mimicker Alarm uses 3 ML APIs: ● Computer Vision ● Emotion ● Speech https://github.com/Microsoft/ProjectOxford-Apps-MimickerAlarm 17

Microsoft LUIS ● Language Understanding Intelligence Service https://www.luis.ai/ ● Train a pre-built language understanding model ● Deploy your model to http endpoint ● Activate models on mobile device 18

Machine Learning Services - build your own ML models 19

ML Services Build your own ML models, or train your own models. Examples: ● Google Cloud ML Platform ● Amazon ML 20

Google TensorFlow ● Open source machine learning library https://www.tensorflow.org/ ● A Neural Network Playground - TensorFlow Demo: Android sample app with TensorFlow *** note this app doesn’t need network for it work Code lab: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/index.html 21

Beyond ML Services 22

Amazon Alexa Voice as the user interface! ● Train Alexa by building new skills ● Connect what you build with Android ● Create your own echo - DIY echo with Alexa + Raspberry Pi 23

Pepper the Robot + Android 1. Download Android plugin & SDK 2. Create an Android project 3. Enable robot project structure Now you are ready to program robots! https://android.aldebaran.com/doc/introduction.html 24

Closing 25

Design considerations ● Does your app really need the intelligence? ● Choose a API/service ○ ease of use, support, pricing ● Get user permissions ● Inform user about data privacy ● Be mindful with user's data 26

What is next? The future is already here! ● Build an app with intelligence ● Make your own echo ● Program a robot ● Study machine learning... 27

Thank you! Reminder to fill out session feedback Keep in touch @margaretmz +MargaretMaynardReid @margaretmz 28

Appendix ML Sessions at I/O - ● Google’s Vision on Machine Learning ● Machine Learning for Art ● Machine learning is not the future ● Breakthroughs in Machine Learning ● How to build a smart RasPi Bot with Cloud Vision and Speech API 29

Appendix Getting started with ML - Udacity ● Intro to Machine Learning ● Deep Learning (TensorFlow) Stanford ● Andrew Ng’s ML course on Coursera Google Developers ● Machine Learning Recipes Reading ● Python Machine Learning by Sabastian Raschka 30