Making Android Apps with Intelligence

This talk was presented at the 360|AnDev conference on 7/29/2016.

Would you like to make your own Echo with a Raspberry Pi and Alexa, or an Android app that recognizes your emotions? Interested in making Android apps with intelligence, such as speech recognition, face detection, natural language processing, prediction and search? Yes you absolutely can! There are open source machine learning services and platforms now available to app developers. In this session I will give an overview of how you can integrate these machine learning services with your Android apps, without any deep expertise in ML. I will walk through several examples to help you get started with making your Android apps with intelligence.

Transcript

1. Making Android Apps with Intelligence Margaret Maynard-Reid, July 29 2016 @margaretmz +MargaretMaynardReid @margaretmz
2. Machine learning everywhere! 2
3. What is Machine Learning? • A subset of Artificial Intelligence • Study of algorithms • Learn from examples and experience (instead of hard-coded programming rules) Source: ML Recipes #1 by Google Developers - https://www.youtube.com/watch?v=cKxRvEZd3Mw 3 Collect training data Train a model Use model to predict
4. Topics covered today 4 Complexity Other topics • Amazon Alexa • Pepper the Robot • Design considerations ML Services (REST APIs) Vision Speech Language Text… (Google, MS, IBM, HP…) ML Services (build or train your ML models) Google Cloud ML Platform TensorFlow Amazon ML Google Play services Awareness API Mobile Vision API
5. Google Play services 5
6. Google Awareness API • Unifies 7 signals into one API - time, location, place, activity, beacons, headphones & weather • Two options - ◦ Snapshot API: current state ◦ Fence API: state change I/O YouTube recording: https://youtu.be/37ia7S4Lsv4 https://developers.google.com/awareness/ 6
7. Getting started • Add this to app build.gradle file • Get an Awareness API key from Google dev console • Add the API key in manifest • (Optional) add required permissions in manifest • (Optional) enable additional APIs for Places and Beacon states Demo: SampleAwareness 7 com.google.android.gms:play-services-contextmanager:9.2.1
8. Google Mobile Vision API • Part of the Google Play services • Very easy to get started • Can detect: ◦ Face (FaceDetector) | codelab ◦ Barcode (BarcodeDetector) | codelab ◦ Text (TextRecognizer) | codelab https://developers.google.com/vision/ 8
9. Face API • Facial detection not recognition • Detection from image or live video • Face can be detected from different angles • Detects activities: ◦ winking, blinking and smiling • Detect landmarks on face 9
10. Face detector Add this to app build.gradle file Create a detector Make sure detector is operational first 10 com.google.android.gms:play-services-vision:9.2.1 FaceDetector faceDetector = new FaceDetector.Builder(getApplicationContext()) .setTrackingEnabled(false) .build(); if(!faceDetector.isOperational()){ // let user know face detector isn’t working // log error etc... }
11. Face detector Detect the face Draw Rectangles on the Faces Refer to code lab for details... Demo: Multi tracker and OCR reader 11 Frame frame = new Frame.Builder().setBitmap(myBitmap).build(); SparseArray<Face> faces = faceDetector.detect(frame);
12. Machine Learning Services - via REST APIs 12
13. Machine learning services Pre-trained models No need for much ML knowledge Just REST API calls “Try the API” directly online from the websites • Google Cloud ML APIs • Microsoft Cognitive Services • IBM Watson • HPE Haven OnDemand 13
14. Getting started Free trial based on time or # of transactions A typical process: • Get API key • Include key in Android app • Download SDK or include dependency in build.gradle (if sdk available) • Make your REST API calls 14
15. REST API calls 15 ML REST APIs Vision Speech Language Text Translate Search…. HTTP response images text HTTP call audio video
16. Microsoft Vision APIs Vision APIs • Computer Vision, Emotion & Face Can detect • Gender • Age • Emotions etc. Demo: SampleVision - image caption 16
17. Demo - Mimicker Alarm Mimicker Alarm uses 3 ML APIs: • Computer Vision • Emotion • Speech https://github.com/Microsoft/ProjectOxford-Apps-MimickerAlarm 17
18. Microsoft LUIS • Language Understanding Intelligence Service https://www.luis.ai/ • Train a pre-built language understanding model • Deploy your model to http endpoint • Activate models on mobile device 18
19. Machine Learning Services - build your own ML models 19
20. ML Services Build your own ML models, or train your own models. Examples: • Google Cloud ML Platform • Amazon ML 20
21. Google TensorFlow • Open source machine learning library https://www.tensorflow.org/ • A Neural Network Playground - TensorFlow Demo: Android sample app with TensorFlow *** note this app doesn’t need network for it work Code lab: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/index.html 21
22. Beyond ML Services 22
23. Amazon Alexa Voice as the user interface! • Train Alexa by building new skills • Connect what you build with Android • Create your own echo - DIY echo with Alexa + Raspberry Pi 23
24. Pepper the Robot + Android 1. Download Android plugin & SDK 2. Create an Android project 3. Enable robot project structure Now you are ready to program robots! https://android.aldebaran.com/doc/introduction.html 24
25. Closing 25
26. Design considerations • Does your app really need the intelligence? • Choose a API/service ◦ ease of use, support, pricing • Get user permissions • Inform user about data privacy • Be mindful with user's data 26
27. What is next? The future is already here! • Build an app with intelligence • Make your own echo • Program a robot • Study machine learning... 27
28. Thank you! Reminder to fill out session feedback Keep in touch @margaretmz +MargaretMaynardReid @margaretmz 28
29. Appendix ML Sessions at I/O - • Google’s Vision on Machine Learning • Machine Learning for Art • Machine learning is not the future • Breakthroughs in Machine Learning • How to build a smart RasPi Bot with Cloud Vision and Speech API 29
30. Appendix Getting started with ML - Udacity • Intro to Machine Learning • Deep Learning (TensorFlow) Stanford • Andrew Ng’s ML course on Coursera Google Developers • Machine Learning Recipes Reading • Python Machine Learning by Sabastian Raschka 30