Making Android Apps with Intelligence - AnDevCon SF 2016

Machine learning is everywhere: from personal assistants to self-driving cars to robots. Now there are various machine learning services available to app developers, some are even open source e.g. TensorFlow. Would you like to make an Android app that recognizes human emotions? Interested in making Android apps with intelligence, such as speech recognition, face detection, natural language processing, prediction and search? In this class, you will get an overview of how you to incorporate machine learning services into your Android apps, without any deep expertise in ML. We will walk through several examples to help you get started with making Android apps with intelligence.

Transcript

1. Making Android Apps with Intelligence AnDevCon San Francisco, 12/01/2016 Margaret Maynard-Reid @margaretmz
2. @margaretmz #androiddev #machinelearning About me 2 • Android developer @MSR • GDG Seattle organizer • Women Techmakers lead • Taught Android @UW • Love the Android community!
3. @margaretmz #androiddev #machinelearning Goals of this talk ✅ Get started making android apps with intelligence with no knowledge in ML ✅ ML Resources available to us ✅ Importance of ML for us developers ❌ Disclaimer: not a crash course on ML 3
4. @margaretmz #androiddev #machinelearning Machine learning everywhere! 4
5. @margaretmz #androiddev #machinelearning Machine learning everywhere! • Google TensorFlow makes music • IBM Watson creates movie trailer • Uber uses Microsoft Cognitive services to selfie check drivers 5
6. @margaretmz #androiddev #machinelearning What is Machine Learning? • A subset of Artificial Intelligence • Study of algorithms • Learn from examples and experience (instead of hard-coded programming rules) Source: ML Recipes #1 by Google Developers - https://www.youtube.com/watch?v=cKxRvEZd3Mw Use model to predict Train a model Collect training data 6
7. @margaretmz #androiddev #machinelearning Make Apps with Intelligence Complexity ML Services (via REST APIs) Vision Speech Language Text… (Google, Microsoft, IBM, HP…) ML Services ( train or build your ML models) Microsoft LUIS Google Cloud ML Platform TensorFlow Amazon ML Google Play Services Mobile Vision API 7
8. @margaretmz #androiddev #machinelearning 8 Mobile Vision API - - via Google Play Services
9. @margaretmz #androiddev #machinelearning Google Mobile Vision API • Part of the Google Play services • Very easy to get started • Can detect: ◦ Face (FaceDetector) | codelab ◦ Barcode (BarcodeDetector) | codelab ◦ Text (TextRecognizer) | codelab https://developers.google.com/vision/ 9
10. @margaretmz #androiddev #machinelearning Mobile Vision API - Face Detection • Detect one or multiple faces • Facial detection not recognition • Detection from image or live video • Face can be detected from different angles • Detects activities: ◦ winking, blinking and smiling • Detect landmarks on face 10
11. @margaretmz #androiddev #machinelearning Face detector Add this to app build.gradle file Create a detector Make sure detector is operational first com.google.android.gms:play-services-vision:9.2.1 FaceDetector faceDetector = new FaceDetector.Builder(getApplicationContext()) .setTrackingEnabled(false) .build(); if(!faceDetector.isOperational()){ // let user know face detector isn’t working // log error etc... } 11
12. @margaretmz #androiddev #machinelearning Face detector Detect the face Draw Rectangles on the Faces Refer to Face Detection with Vision API code lab for details... Frame frame = new Frame.Builder().setBitmap(myBitmap).build(); SparseArray<Face> faces = faceDetector.detect(frame); 12
13. Googly Eyes Android App Slide & video credit: Google 1. Create a face detector for facial landmarks (e.g., eyes) 3. For each face, draw the eyes FaceDetector detector = new FaceDetector.Builder() .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build(); SparseArray<Face> faces = detector.detect(image); for (int i = 0; i < faces.size(); ++i) { Face face = faces.valueAt(i); for (Landmark landmark : face.getLandmarks()) { // Draw eyes 2. Detect faces in the image 13
14. @margaretmz #androiddev #machinelearning Mobile Vision API - Barcode Detection • Detects barcodes in real-time, on device, in any orientation. • Detects multiple barcodes at once • Detects and parses barcodes and Qr codes of various formats 14
15. @margaretmz #androiddev #machinelearning Mobile Vision API - Text Recognition • Detects text in real-time, on device. • Detects text in Latin based languages (French, German, English, etc.) • Recognizes text in images and video streams 15
16. @margaretmz #androiddev #machinelearning 16 Machine Learning Services - - via REST APIs
17. @margaretmz #androiddev #machinelearning Machine learning services Pre-trained models No need for much ML knowledge Just REST API calls “Try out the APIs” directly online from the websites • Google Cloud ML APIs • Microsoft Cognitive Services • IBM Watson • HPE Haven OnDemand 17
18. @margaretmz #androiddev #machinelearning Online demo • Google cloud vision API • Microsoft emotion API • HP Haven on demand 18
19. @margaretmz #androiddev #machinelearning Getting started Free trial based on time or # of transactions A typical process: • Get API key • Include key in Android app • Download SDK or include dependency in build.gradle (if sdk available) • Make your REST API calls 19
20. @margaretmz #androiddev #machinelearning REST API calls ML REST APIs Vision Speech Language Text Translate Search…. HTTP response images text HTTP call audio video 20
21. @margaretmz #androiddev #machinelearning Microsoft Vision APIs Vision APIs • Computer Vision, Emotion & Face Can detect • Gender • Age • Emotions etc. Demo: SampleVision - image caption 21
22. @margaretmz #androiddev #machinelearning Demo - Mimicker Alarm This Android app uses 3 ML APIs: • Computer Vision • Emotion • Speech https://github.com/Microsoft/ProjectOxford-Apps-MimickerAlarm 22
23. @margaretmz #androiddev #machinelearning 23 Machine Learning Services - - train or build ML models
24. @margaretmz #androiddev #machinelearning ML Services Train or build your own ML models. Examples: • Microsoft LUIS • Google TensorFlow • Google Cloud ML Platform • Amazon ML 24
25. @margaretmz #androiddev #machinelearning Microsoft LUIS Language Understanding Intelligence Service - • User friendly web interface • Train a pre-built language understanding model • Deploy model to an http endpoint - interprets human language • Use the model on mobile app - JSON response 25
26. @margaretmz #androiddev #machinelearning Demo of LUIS - a weather app • Go to https://www.luis.ai/ • Create an Application - “Weather app AnDevCon demo” • Add two Pre-built Entities: geography & datetime • Add Intent ◦ Name = “GetWeather” ◦ Example utterance = “What is the weather like in Seattle today?” ◦ Two parameters “where” (geography) & “when” (datetime) • Train with a few New Utterances (at least 5) What is the weather in San Francisco this Sunday? What is the weather in New York tomorrow? • Publish the http endpoint 26
27. @margaretmz #androiddev #machinelearning Using LUIS in your app • Download LUIS Android SDK • Add the keys to your strings.xml - ◦ Cognitive Services subscription key for LUIS ◦ LUIS Application Id (Application Settings on luis.ai) • (Optional) combine Speech to Text with LUIS 27
28. @margaretmz #androiddev #machinelearning TensorFlow • Open source Machine Learning library ◦ Most popular repo on GitHub! • Especially useful for Deep Learning • For research and production • Apache 2.0 license 28
29. 29 Source: https://youtu.be/oZikw5k_2FM
30. @margaretmz #androiddev #machinelearning Code lab: TensorFlow for Poets • Install and run the TensorFlow Docker image • Retrieve the images (flowers: daisy, dandelion, roses, sunflowers, & tulips) • Retrain the final layer of Inception 3 • Use the retrained model to classify an image https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#0 30
31. @margaretmz #androiddev #machinelearning TensorFlow on mobile • Android • iOS • Demo: Android sample app with TensorFlow 31
32. @margaretmz #androiddev #machinelearning Machine Learning - - some other examples 32
33. @margaretmz #androiddev #machinelearning Amazon Alexa Voice Service Voice as the user interface! Train Alexa by building new skills • Create your own echo with Alexa Voice Service on Raspberry Pi, Linux, Mac & Windows. • Access Alexa Voice Service from Android 33
34. @margaretmz #androiddev #machinelearning Pepper the Robot + Android Pepper’s Intelligence is powered by IBM Watson 1. Download Android plugin & SDK 2. Create an Android project 3. Enable robot project structure Now you are ready to program robots! https://android.aldebaran.com/doc/introduction.html 34
35. @margaretmz #androiddev #machinelearning Google AI Experiments https://aiexperiments.withgoogle.com/ 35
36. @margaretmz #androiddev #machinelearning Closing 36
37. @margaretmz #androiddev #machinelearning Design considerations • Does your app really need the intelligence? • Choose a API/service ◦ ease of use, support, pricing • Get user permissions • Inform user about data privacy • Be mindful with user's data 37
38. @margaretmz #androiddev #machinelearning What is next? • Try out machine learning APIs • Try out TensorFlow tutorials and code labs • Check out the AI experiments • Study machine learning basics • Build an app with intelligence 38
39. @margaretmz #androiddev #machinelearning Thank you! @margaretmz +MargaretMaynardReid @margaretmz 39
40. @margaretmz #androiddev #machinelearning Appendix ML Sessions at Google I/O - • Google’s Vision on Machine Learning • Machine Learning for Art • Machine learning is not the future • Breakthroughs in Machine Learning • How to build a smart RasPi Bot with Cloud Vision and Speech API Google AI Experiments • https://aiexperiments.withgoogle.com/ 40
41. @margaretmz #androiddev #machinelearning Appendix - AI & ML learning resources Udacity • Intro to Machine Learning • AI Engineer Nanodegree • ML Engineer Nanodegree • Self-driving Car Engineer Nanodegree • Deep Learning (TensorFlow) Stanford • Andrew Ng’s ML course on Coursera Google Developers • Machine Learning Recipes (short Youtube videos) Reading • Python Machine Learning by Sabastian Raschka 41