Making Android Apps with Intelligence - Big Android BBQ 2016

This talk was presented at the Big Android BBQ 2016.

Machine learning is everywhere: from personal assistants to self driving cars to robots. Now there are various machine learning services and platforms available to app developers, some are even open source e.g. TensorFlow. Would you like to make an Android app that recognizes human emotions? Interested in making Android apps with intelligence, such as speech recognition, face detection, natural language processing, prediction and search? In this talk I will give an overview of how you can incorporate the machine learning services with your Android apps, without any deep expertise in ML. I will walk through several examples to help you get started with making Android apps with intelligence.

Transcript

1. Making Android Apps with Intelligence Big Android BBQ, 10/22/2016 Margaret Maynard-Reid @margaretmz
2. About me 2 • Android developer @MSR • GDG Seattle organizer • Women Techmakers lead • Taught Android @UW • Love the Android community!
3. @margaretmz #BABBQ16 #androiddevs #machinelearning Goals of this talk ✅ Get started making android apps with intelligence with no knowledge in ML ✅ ML Resources available to us ✅ Importance of ML for us developers ❌ Disclaimer: not a crash course on ML 3
4. @margaretmz #BABBQ16 #androiddevs #machinelearning Machine learning everywhere! 4
5. @margaretmz #BABBQ16 #androiddevs #machinelearning Machine learning everywhere! • Google TensorFlow makes music • IBM Watson creates movie trailer • Uber uses Microsoft Cognitive services to selfie check drivers 5
6. @margaretmz #BABBQ16 #androiddevs #machinelearning What is Machine Learning? • A subset of Artificial Intelligence • Study of algorithms • Learn from examples and experience (instead of hard-coded programming rules) Source: ML Recipes #1 by Google Developers - https://www.youtube.com/watch?v=cKxRvEZd3Mw Use model to predict Train a model Collect training data 6
7. @margaretmz #BABBQ16 #androiddevs #machinelearning Make Apps with Intelligence Complexity ML Services (via REST APIs) Vision Speech Language Text… (Google, MS, IBM, HP…) ML Services ( train or build your ML models) Google Cloud ML Platform TensorFlow Amazon ML Google Play Services Mobile Vision API 7
8. @margaretmz #BABBQ16 #androiddevs #machinelearning 8 Mobile Vision API via Google Play Services
9. @margaretmz #BABBQ16 #androiddevs #machinelearning Google Mobile Vision API • Part of the Google Play services • Very easy to get started • Can detect: ◦ Face (FaceDetector) | codelab ◦ Barcode (BarcodeDetector) | codelab ◦ Text (TextRecognizer) | codelab https://developers.google.com/vision/ 9
10. @margaretmz #BABBQ16 #androiddevs #machinelearning Face API • Facial detection not recognition • Detection from image or live video • Face can be detected from different angles • Detects activities: ◦ winking, blinking and smiling • Detect landmarks on face 10
11. @margaretmz #BABBQ16 #androiddevs #machinelearning Face detector Add this to app build.gradle file Create a detector Make sure detector is operational first com.google.android.gms:play-services-vision:9.2.1 FaceDetector faceDetector = new FaceDetector.Builder(getApplicationContext()) .setTrackingEnabled(false) .build(); if(!faceDetector.isOperational()){ // let user know face detector isn’t working // log error etc... } 11
12. @margaretmz #BABBQ16 #androiddevs #machinelearning Face detector Detect the face Draw Rectangles on the Faces Refer to Face Detection with Vision API code lab for details... Frame frame = new Frame.Builder().setBitmap(myBitmap).build(); SparseArray<Face> faces = faceDetector.detect(frame); 12
13. Googly Eyes Android App Slide & video credit: Google 1. Create a face detector for facial landmarks (e.g., eyes) 3. For each face, draw the eyes FaceDetector detector = new FaceDetector.Builder() .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build(); SparseArray<Face> faces = detector.detect(image); for (int i = 0; i < faces.size(); ++i) { Face face = faces.valueAt(i); for (Landmark landmark : face.getLandmarks()) { // Draw eyes 2. Detect faces in the image
14. @margaretmz #BABBQ16 #androiddevs #machinelearning 14 Machine Learning Services via REST APIs
15. @margaretmz #BABBQ16 #androiddevs #machinelearning Machine learning services Pre-trained models No need for much ML knowledge Just REST API calls “Try the API” directly online from the websites • Google Cloud ML APIs • Microsoft Cognitive Services • IBM Watson • HPE Haven OnDemand 15
16. @margaretmz #BABBQ16 #androiddevs #machinelearning Online demo • Google cloud vision API • Microsoft • HP Haven on demand 16
17. @margaretmz #BABBQ16 #androiddevs #machinelearning Getting started Free trial based on time or # of transactions A typical process: • Get API key • Include key in Android app • Download SDK or include dependency in build.gradle (if sdk available) • Make your REST API calls 17
18. @margaretmz #BABBQ16 #androiddevs #machinelearning REST API calls ML REST APIs Vision Speech Language Text Translate Search…. HTTP response images text HTTP call audio video 18
19. @margaretmz #BABBQ16 #androiddevs #machinelearning Microsoft Vision APIs Vision APIs • Computer Vision, Emotion & Face Can detect • Gender • Age • Emotions etc. Demo: SampleVision - image caption 19
20. @margaretmz #BABBQ16 #androiddevs #machinelearning Demo - Mimicker Alarm This Android app uses 3 ML APIs: • Computer Vision • Emotion • Speech https://github.com/Microsoft/ProjectOxford-Apps-MimickerAlarm 20
21. @margaretmz #BABBQ16 #androiddevs #machinelearning 21 Machine Learning Services train or build ML models
22. @margaretmz #BABBQ16 #androiddevs #machinelearning ML Services Train or build your own ML models. Examples: • Microsoft LUIS • Google TensorFlow • Google Cloud ML Platform • Amazon ML 22
23. @margaretmz #BABBQ16 #androiddevs #machinelearning Microsoft LUIS • Language Understanding Intelligence Service https://www.luis.ai/ • Train a pre-built language understanding model ◦ Create an Intent - CreateCalendarEvent ◦ Add parameters - when, what, who • Deploy model to an http endpoint - interprets language • Use the model on mobile app - JSON response 23
24. @margaretmz #BABBQ16 #androiddevs #machinelearning TensorFlow • Open source Machine Learning library ◦ Most popular repo on GitHub! • Especially useful for Deep Learning • For research and production • Apache 2.0 license 24
25. 25 Source: https://youtu.be/oZikw5k_2FM
26. @margaretmz #BABBQ16 #androiddevs #machinelearning TensorFlow code lab • Install and run the TensorFlow Docker image • Retrieve the images • Retrain the final layer of Inception 3 • Use the retrained model to classify an image 26
27. @margaretmz #BABBQ16 #androiddevs #machinelearning TensorFlow on mobile • Android • iOS • Demo: Android sample app with TensorFlow 27
28. @margaretmz #BABBQ16 #androiddevs #machinelearning Beyond ML Services 28
29. @margaretmz #BABBQ16 #androiddevs #machinelearning Amazon Alexa Voice as the user interface! • Train Alexa by building new skills • Connect what you build with Android • Create your own echo - DIY echo with Alexa + Raspberry Pi 29
30. @margaretmz #BABBQ16 #androiddevs #machinelearning Pepper the Robot + Android 1. Download Android plugin & SDK 2. Create an Android project 3. Enable robot project structure Now you are ready to program robots! https://android.aldebaran.com/doc/introduction.html 30
31. @margaretmz #BABBQ16 #androiddevs #machinelearning Closing 31
32. @margaretmz #BABBQ16 #androiddevs #machinelearning Design considerations • Does your app really need the intelligence? • Choose a API/service ◦ ease of use, support, pricing • Get user permissions • Inform user about data privacy • Be mindful with user's data 32
33. @margaretmz #BABBQ16 #androiddevs #machinelearning What is next? • Try out machine learning APIs • Try out TensorFlow tutorials and code labs • Study machine learning basics • Build an app with intelligence 33
34. @margaretmz #BABBQ16 #androiddevs #machinelearning Thank you! @margaretmz +MargaretMaynardReid @margaretmz 34
35. @margaretmz #BABBQ16 #androiddevs #machinelearning Appendix ML Sessions at I/O - • Google’s Vision on Machine Learning • Machine Learning for Art • Machine learning is not the future • Breakthroughs in Machine Learning • How to build a smart RasPi Bot with Cloud Vision and Speech API 35
36. @margaretmz #BABBQ16 #androiddevs #machinelearning Appendix ML learning resources Udacity • Intro to Machine Learning • Deep Learning (TensorFlow) Stanford • Andrew Ng’s ML course on Coursera Google Developers • Machine Learning Recipes Reading • Python Machine Learning by Sabastian Raschka 36