CoreML Hands On

An introduction to what is possible when adding machine learning models to an iOS app via CoreML. Includes example CoreML models available today, some fanciful ideas for tomorrow, and our exploration of teaching an iPhone to play pinball in the real world.

Transcript

1. CoreML Hands On smallplanet.machinelearning Quinn McHenry, Technical Director @qmchenry
2. CoreML Hands On What will our robot overlords be really good at? ⇢
   or
   ⇠ What is machine learning?
3. More Specifically + =¯\_(ツ)_/¯ CoreML
4. Nutshell Information Information Model
5. CoreML developer.apple.com/machine-learning/ + =
6. CoreML developer.apple.com/videos/play/wwdc2017/703/
7. CoreML developer.apple.com/documentation/coreml • Command line tools • coremltools • coremlc • Capabilities • Dynamic model download • Vision framework integration • Metal performance shaders integration
8. Awesome CoreML github.com/likedan/Awesome-CoreML-Models Collection of CoreML models ready for use • References • Demos • Model zoo
9. Sensory Information Sight Sound % Smell Taste Touch Temperature ) Kinesthetic Movement Pain ⏲ Time Echolocation ⚡ Electric Magnetic Infrared Location
10. Sensory Information Sight Sound % Smell Taste Touch Temperature ) Kinesthetic Movement Pain ⏲ Time Echolocation ⚡ Electric Magnetic Infrared Location 2Human
11. Sensory Information Sight Sound % Smell Taste Touch Temperature ) Kinesthetic Movement Pain ⏲ Time Echolocation ⚡ Electric Magnetic Infrared Location iPhone
12. Sight
13. Image Classification • Classify >1000 things • Can locate items in scene • Requires lots of tagged images • Supervised learning github.com/hanleyweng/CoreML-in-ARKit + Inception v3
14. Image Classification • CarRecognition dataset • 163 makes • 1,716 models • 136,726 images mmlab.ie.cuhk.edu.hk/datasets/comp_cars
16. Image Classification • Age • Gender • Emotions github.com/cocoa-ai/FacesVisionDemo
17. Face/Feature Detection sighthound.com, github.com/cocoa-ai/FacesVisionDemo
18. Face-Tracking apple.com/iphone-x/#truedepth-camera
19. arxiv.org/pdf/1610.07629v1.pdf Style Transfer
20. Style Transfer arxiv.org/pdf/1610.07629v1.pdf
21. Style Transfer arxiv.org/pdf/1610.07629v1.pdf
22. Stories from Images medium.com/@samim/generating-stories-about-images-d163ba41e4ed
23. Stories from Images medium.com/@samim/generating-stories-about-images-d163ba41e4ed
24. Live Translation translate.google.com/intl/en/about/
25. Sound
26. Sound Classification • air conditioner • car horn • children playing • dog bark • drilling • engine idling • gunshot • jackhammer • siren • street music serv.cusp.nyu.edu/projects/urbansounddataset/ Urban Sounds
27. Bird Song Classification ncbi.nlm.nih.gov/pmc/articles/PMC4106198/
28. Lung Sound Classification ncbi.nlm.nih.gov/pubmed/27286184
29. Speech Recognition
30. Movement
31. Characterizing Parkinson’s ncbi.nlm.nih.gov/pubmed/21096671
32. Demographic Classification ncbi.nlm.nih.gov/pmc/articles/PMC4721824/ One Small Step for a Man: Estimation of Gender, Age and Height from Recordings of One Step by a Single Inertial Sensor
33. Gesture Recognition https://github.com/akimach/GestureAI-CoreML-iOS
34. Cognitive
35. Kill Beat Humans deepmind.com/research/publications/mastering-game-go-deep-neural-networks-tree-search/ Professional Go player Lee Sedol reviewing his match with AlphaGo after defeat.
36. Self-driving Car
37. instacart Shopping tech.instacart.com/deep-learning-with-emojis-not-math-660ba1ad6cdc
38. Reinforcement Learning deepmind.com/research/publications/human-level-control-through-deep-reinforcement-learning/ • 2014 Nature Article • Human-level or better • 49 Atari 2600 games • Single model • Using only pixel data & score
39. Reinforcement Learning gym.openai.com/envs/ With OpenAI Gym:
40. Reinforcement Learning deepmind.com/research/publications/human-level-control-through-deep-reinforcement-learning/
41. Decide Which… • Word you might type next • Ad you’ll most likely click • Tweets you’ll ❤ • Netflix series you’ll binge • Shoes best go with those pants • Diagnosis is correct • Cryptocurrency will • Driving path is fastest • Elevator to summon
42. Decide Which… • Word you might type next • Ad you’ll most likely click • Tweets you’ll ❤ • Netflix series you’ll binge • Shoes best go with those pants • Diagnosis is correct • Cryptocurrency will • Driving path is fastest • Elevator to summon • Flipper to flip
43. smallplanet.pinball
44. smallplanet.pinball github.com/SmallPlanetiOS/smallplanet_Pinball • Interface with hardware • Build structures to hold devices • Read scores from machine • Solder transistors • Manage lighting/photography • Master physical debugging • Network multiple devices
45. Hardware Interface • Onion Omega2 • Relay expansions • Python server
46. Score OCR
48. smallplanet.pinball gym.openai.com/envs/
50. smallplanet.machinelearning Questions, humans?
51. smallplanet.machinelearning Let’s go see some pinball!