Making Android Apps With Intelligience - GDG Bellevue

This talk was presented at the Seattle Eastside Android (GDG Bellevue) meetup on 9/28/2016.

Machine learning is everywhere: from personal assistants to self driving cars to robots. Machine learning is not new, but the exciting news is that the various machine learning services are now available to app developers, some are even open source e.g. TensorFlow. In this talk I will give an overview of how you can incorporate the machine learning services with your Android apps without any deep expertise in ML.

Transcript

1. Making Android Apps with Intelligence Eastside Android meetup @Offerup, 9/28/2016 By Margaret Maynard-Reid @margaretmz +MargaretMaynardReid @margaretmz @margaretmz +MargaretMaynardReid @margaretmz
2. Machine learning everywhere! 2
3. Machine learning everywhere! News stories - • Google TensorFlow makes music • IBM Watson creates movie trailer • Uber uses Microsoft Cognitive services to selfie check drivers 3
4. What is Machine Learning? • A subset of Artificial Intelligence • Study of algorithms • Learn from examples and experience (instead of hard-coded programming rules) Source: ML Recipes #1 by Google Developers - https://www.youtube.com/watch?v=cKxRvEZd3Mw Collect training data Train a model Use model to predict 4
5. Make Apps with Intelligence Complexity ML Services (via REST APIs) Vision Speech Language Text… (Google, MS, IBM, HP…) ML Services (build or train your ML models) Google Cloud ML Platform TensorFlow Amazon ML Google Play Services Mobile Vision API 5
6. Google Mobile Vision API 6
7. Google Mobile Vision API • Part of the Google Play services • Very easy to get started • Can detect: ◦ Face (FaceDetector) | codelab ◦ Barcode (BarcodeDetector) | codelab ◦ Text (TextRecognizer) | codelab https://developers.google.com/vision/ 7
8. Face API • Facial detection not recognition • Detection from image or live video • Face can be detected from different angles • Detects activities: ◦ winking, blinking and smiling • Detect landmarks on face 8
9. Face detector Add this to app build.gradle file Create a detector Make sure detector is operational first com.google.android.gms:play-services-vision:9.2.1 FaceDetector faceDetector = new FaceDetector.Builder(getApplicationContext()) .setTrackingEnabled(false) .build(); if(!faceDetector.isOperational()){ // let user know face detector isn’t working // log error etc... } 9
10. Face detector Detect the face Draw Rectangles on the Faces Refer to Face Detection with Vision API code lab for details... Frame frame = new Frame.Builder().setBitmap(myBitmap).build(); SparseArray<Face> faces = faceDetector.detect(frame); 10
11. Googly Eyes Android App Slide & video credit: Google 1. Create a face detector for facial landmarks (e.g., eyes) 3. For each face, draw the eyes FaceDetector detector = new FaceDetector.Builder() .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build(); SparseArray<Face> faces = detector.detect(image); for (int i = 0; i < faces.size(); ++i) { Face face = faces.valueAt(i); for (Landmark landmark : face.getLandmarks()) { // Draw eyes 2. Detect faces in the image
12. Machine Learning Services - via REST APIs 12
13. Machine learning services Pre-trained models No need for much ML knowledge Just REST API calls “Try the API” directly online from the websites • Google Cloud ML APIs • Microsoft Cognitive Services • IBM Watson • HPE Haven OnDemand 13
14. Getting started Free trial based on time or # of transactions A typical process: • Get API key • Include key in Android app • Download SDK or include dependency in build.gradle (if sdk available) • Make your REST API calls 14
15. REST API calls ML REST APIs Vision Speech Language Text Translate Search…. HTTP response images text HTTP call audio video 15
16. Microsoft Vision APIs Vision APIs • Computer Vision, Emotion & Face Can detect • Gender • Age • Emotions etc. Demo: SampleVision - image caption 16
17. Demo - Mimicker Alarm Mimicker Alarm uses 3 ML APIs: • Computer Vision • Emotion • Speech https://github.com/Microsoft/ProjectOxford-Apps-MimickerAlarm 17
18. Microsoft LUIS • Language Understanding Intelligence Service https://www.luis.ai/ • Train a pre-built language understanding model ◦ Create an Intent - CreateCalendarEvent ◦ Add parameters - when, what, who • Deploy model to an http endpoint - interprets language • Use the model on mobile app - JSON response 18
19. Machine Learning Services - build your own ML models 19
20. ML Services Build your own ML models, or train your own models. Examples: • Google Cloud ML Platform • Amazon ML 20
21. Google TensorFlow • Open source machine learning library https://www.tensorflow.org/ • A Neural Network Playground - TensorFlow • Code lab: TensorFlow for Poets Demo: Android sample app with TensorFlow (works offline) 21
22. Beyond ML Services 22
23. Amazon Alexa Voice as the user interface! • Train Alexa by building new skills • Connect what you build with Android • Create your own echo - DIY echo with Alexa + Raspberry Pi 23
24. Pepper the Robot + Android 1. Download Android plugin & SDK 2. Create an Android project 3. Enable robot project structure Now you are ready to program robots! https://android.aldebaran.com/doc/introduction.html 24
25. Closing 25
26. Design considerations • Does your app really need the intelligence? • Choose a API/service ◦ ease of use, support, pricing • Get user permissions • Inform user about data privacy • Be mindful with user's data 26
27. What is next? The future is already here! • Build an app with intelligence • Make your own echo • Program a robot • Study machine learning basics 27
28. Thank you! @margaretmz +MargaretMaynardReid @margaretmz 28
29. Appendix ML Sessions at I/O - • Google’s Vision on Machine Learning • Machine Learning for Art • Machine learning is not the future • Breakthroughs in Machine Learning • How to build a smart RasPi Bot with Cloud Vision and Speech API 29
30. Appendix ML learning resources Udacity • Intro to Machine Learning • Deep Learning (TensorFlow) Stanford • Andrew Ng’s ML course on Coursera Google Developers • Machine Learning Recipes Reading • Python Machine Learning by Sabastian Raschka 30