Making Android Apps with Intelligence Devoxx US 2017

Artificial Intelligence is everywhere: from personal assistants to self driving cars to robots. Machine learning (subset of AI) is not new, but the exciting news is that the various machine learning services are now available to app developers, some are even open source e.g. TensorFlow. Would you like to make an Android app that recognizes human emotions? Interested in making Android apps with intelligence, such as speech recognition, face detection, natural language processing, prediction and search? In this talk I will give an overview of how you can incorporate the machine learning services with your Android apps, without any deep expertise in ML. I will walk through several examples to help you get started with making Android apps with intelligence.

Transcript

1. #DevoxxUS Making Android Apps with Intelligence 3/22/2017 Margaret Maynard-Reid @margaretmz
2. #DevoxxUS @margaretmz About me 2 • Android developer @MS AI & Research • Student of Udacity AI Nanodegree • GDG Seattle organizer • Women Techmakers lead • Taught Android @UW
3. #DevoxxUS @margaretmz Artificial Intelligence everywhere! 3
4. #DevoxxUS @margaretmz Machine learning everywhere! • Google TensorFlow makes music • IBM Watson creates movie trailer • Uber uses Microsoft Cognitive services to selfie check drivers 4
5. #DevoxxUS @margaretmz What is Machine Learning? • A subset of Artificial Intelligence • Study of algorithms • Learn from examples and experience (instead of hard-coded programming rules) Source: ML Recipes #1 by Google Developers - https://www.youtube.com/watch?v=cKxRvEZd3Mw Use model to predict Train a model Collect training data 5
6. #DevoxxUS @margaretmz Goals of this talk • Help you get started with making apps with intelligence • Share with you ML resources • Understand the importance of ML for app developers • Disclaimer: not a crash course on ML 6
7. #DevoxxUS @margaretmz Make Apps with Intelligence Complexity ML Services (via REST APIs) Vision Speech Language Text… (Google, Microsoft, IBM, HP…) ML Services ( train or build your ML models) TensorFlow Microsoft LUIS Google Cloud ML Platform Amazon ML Google Play Services Mobile Vision API 7
8. #DevoxxUS @margaretmz 8 Mobile Vision API - via Google Play Services
9. #DevoxxUS @margaretmz Google Mobile Vision API • Part of the Google Play services • Very easy to get started • Can detect: ◦ Face (FaceDetector) | codelab ◦ Barcode (BarcodeDetector) | codelab ◦ Text (TextRecognizer) | codelab Detailed documentation on Mobile Vision API - https://developers.google.com/vision/ 9
10. #DevoxxUS @margaretmz Mobile Vision API - Face Detection • Detect one or multiple faces • Facial detection not recognition • Detection from image or live video • Face can be detected from different angles • Detects activities: ◦ winking, blinking and smiling • Detect landmarks on face 10
11. #DevoxxUS @margaretmz Face detector Add this to app build.gradle file Create a detector Make sure detector is operational first com.google.android.gms:play-services-vision:9.2.1 FaceDetector faceDetector = new FaceDetector.Builder(getApplicationContext()) .setTrackingEnabled(false) .build(); if(!faceDetector.isOperational()){ // let user know face detector isn’t working // log error etc... } 11
12. #DevoxxUS @margaretmz Face detector Detect the face(s) Draw Rectangles on the Faces Refer to Face Detection with Vision API code lab for details... Frame frame = new Frame.Builder().setBitmap(myBitmap).build(); SparseArray<Face> faces = faceDetector.detect(frame); 12
13. #DevoxxUS @margaretmz Slide & video credit: Google 1. Create a face detector for facial landmarks (e.g., eyes) 3. For each face, draw the eyes FaceDetector detector = new FaceDetector.Builder() .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build(); SparseArray<Face> faces = detector.detect(image); for (int i = 0; i < faces.size(); ++i) { Face face = faces.valueAt(i); for (Landmark landmark : face.getLandmarks()) { // Draw eyes 2. Detect faces in the image 13 Googly Eyes Android App
14. #DevoxxUS @margaretmz Mobile Vision API - Barcode Detection • Detects barcodes in real-time, on device, in any orientation. • Detects multiple barcodes at once • Detects and parses barcodes and Qr codes of various formats 14
15. #DevoxxUS @margaretmz Mobile Vision API - Text Recognition • Detects text in real-time, on device. • Detects text in Latin based languages (French, German, English, etc.) • Recognizes text in images and video streams 15
16. #DevoxxUS @margaretmz 16 Machine Learning Services - via REST APIs
17. #DevoxxUS @margaretmz REST API calls ML REST APIs Vision Speech Language Text Translate Search…. HTTP response images text HTTP call audio video 17
18. #DevoxxUS @margaretmz Getting started Free trial based on time or # of transactions A typical process: • Get API key • Include key in Android app • Download SDK or include dependency in build.gradle (if sdk available) • Make your REST API calls 18
19. #DevoxxUS @margaretmz Machine learning services Pre-trained models No need for much ML knowledge Just REST API calls “Try out the APIs” directly online from the websites • Google Cloud ML APIs • Microsoft Cognitive Services • IBM Watson • HPE Haven OnDemand 19
20. #DevoxxUS @margaretmz Cloud ML services demo • Google Cloud vision API - drag and drop an image for analysis • Microsoft emotion API - recognize emotions • HP Haven on demand 20
21. #DevoxxUS @margaretmz Microsoft Vision APIs Vision APIs • Computer Vision, Emotion & Face Can detect • Gender • Age • Emotions etc. Demo: SampleVision - image caption 21
22. #DevoxxUS @margaretmz Demo - Mimicker Alarm This Android app uses 3 ML APIs: • Computer Vision - find a color • Emotion - facial expression • Speech - tongue twister Source code - https://github.com/Microsoft/ProjectOxford-Apps-MimickerAlarm 22
23. #DevoxxUS @margaretmz 23 Machine Learning Services - train pre-trained models
24. #DevoxxUS @margaretmz ML Services Train a pre-trained model, or build your own ML models - • Google TensorFlow • Microsoft LUIS • Google Cloud ML Platform • Amazon ML 24
25. #DevoxxUS @margaretmz TensorFlow • Open source Machine Learning library o Most popular repo on GitHub! • Especially useful for Deep Learning • For research and production • Apache 2.0 license • 1.0.1 released recently 25
26. #DevoxxUS @margaretmz 26 Source: https://youtu.be/oZikw5k_2FM
27. #DevoxxUS @margaretmz Code lab: TensorFlow for Poets • Install and run the TensorFlow Docker image • Retrieve the images (flowers: daisy, dandelion, roses, sunflowers, & tulips) • Retrain the final layer of Inception 3 • Use the retrained model to classify an image https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#0 27
28. #DevoxxUS @margaretmz TensorFlow on mobile • Android and iOS • Android sample app with TensorFlow ◦ TF Classify (demo) ◦ TF Detect ◦ TF Stylize (demo) • Checkout all TensorFlow Dev Summit videos ◦ Pete Warden’s talk Mobile and Embedded TensorFlow 28
29. #DevoxxUS @margaretmz Microsoft LUIS Language Understanding Intelligence Service - • User friendly web interface • Train a pre-built language understanding model • Deploy model to an http endpoint - interprets human language • Use the model on mobile app - JSON response 29
30. #DevoxxUS @margaretmz Demo of LUIS - a weather app • Go to https://www.luis.ai/ • Create an Application - “Weather app demo” • Add two Pre-built Entities: geography & datetime • Add Intent ◦ Name = “GetWeather” ◦ Example utterance = “What is the weather like in Seattle today?” ◦ Two parameters “where” (geography) & “when” (datetime) • Train with a few New Utterances (at least 5) What is the weather in San Francisco this Sunday? What is the weather in New York tomorrow? • Publish the http endpoint 30
31. #DevoxxUS @margaretmz Using LUIS in your app • Download LUIS Android SDK • Add the keys to your strings.xml - ◦ Cognitive Services subscription key for LUIS ◦ LUIS Application Id (Application Settings on luis.ai) • (Optional) combine Speech to Text with LUIS 31
32. #DevoxxUS @margaretmz Machine Learning - some other examples 32
33. #DevoxxUS @margaretmz Amazon Alexa Voice Service Voice as the user interface! Train Alexa by building new skills • Create your own echo with Alexa Voice Service on Raspberry Pi, Linux, Mac & Windows. • Access Alexa Voice Service from Android 33
34. #DevoxxUS @margaretmz Pepper the Robot + Android Pepper’s Intelligence is powered by IBM Watson 1. Download Android plugin & SDK 2. Create an Android project 3. Enable robot project structure Now you are ready to program robots! https://android.aldebaran.com/doc/introduction.html 34
35. #DevoxxUS @margaretmz Google AI Experiments Fun experiments with images, music, game & code https://aiexperiments.withgoogle.com/ 35
36. #DevoxxUS @margaretmz Closing 36
37. #DevoxxUS @margaretmz Design considerations • Does your app really need the intelligence? • Choose a API/service ◦ ease of use, support, pricing • Get user permissions • Inform user about data privacy • Be mindful with user's data 37
38. #DevoxxUS @margaretmz What is next? •Try out machine learning APIs • Try out TensorFlow tutorials and code labs • Check out the AI experiments • Study machine learning basics • Build an app with intelligence 38
39. #DevoxxUS @margaretmz Thank you! @margaretmz +MargaretMaynardReid @margaretmz 39
40. #DevoxxUS @margaretmz Appendix ML Sessions at Google I/O 2016 - • Google’s Vision on Machine Learning • Machine Learning for Art • Machine learning is not the future • Breakthroughs in Machine Learning • How to build a smart RasPi Bot with Cloud Vision and Speech API Google AI Experiments • https://aiexperiments.withgoogle.com/ Udacity blog • The What, How, and Why of Artificial Intelligence, Machine Learning, and Self-Driving Cars 40
41. #DevoxxUS @margaretmz Appendix - AI & ML learning resources Udacity • Intro to Machine Learning • AI Engineer Nanodegree • ML Engineer Nanodegree • Self-driving Car Engineer Nanodegree • Deep Learning (TensorFlow) Stanford • Andrew Ng’s ML course on Coursera Google Developers • Machine Learning Recipes (short Youtube videos) Reading • Python Machine Learning by Sabastian Raschka 41