Bootstrapping the Machine Learning Training Process

AppBuilders (https://appbuilders.ch)
April 2018
Lugano, Switzerland

When is it valuable to consider using machine learning to help solve your iOS app’s problems? We will go through criteria to evaluate that, practical use cases of incorporating ML models on device, and how to bootstrap the training process for creating machine learning models that will run on device (iOS). Using techniques such as transfer learning and taking inspiration from the plentiful resources from industry and academia, you can get started doing this now. In this talk, we will go through some examples across different domains, and, yes, we will move beyond the dog / cat classifier.

Transcript

1. Bootstrapping the Machine Learning Training Process Meghan Kane, @meghafon AppBuilders April 2018
2. ! Hey, I'm Meghan! @meghafon iOS Engineer @ Novoda Berlin
5. is it all the hype?
6. machine learning is a powerful tool but, it is still just another tool
7. how can we think about ML as ! developers?
8. Can this be solved without ML? if so, choose that
9. ML vs not ML basic unit of solving problem = function ("model") ML: enabling a machine to learn function on its own classify sign language alphabet images not ML: explicitly defining function determining if a number is even/odd
10. If you decide to use ML still go with the simplest solution
11. Why do ML (predictions) on mobile? 1. low latency user experience 2. user privacy
12. What's available from Apple? image classification of 1000 common categories → trees, animals, food, vehicles, people → SqueezeNet (5 MB), MobileNet (17 MB), Inception V3 (95 MB), ResNet50 (103 MB), VGG16 (554 MB) scene classification of 205 categories → airport terminal, bedroom, forest, coast → Places205-GoogLeNet (25 MB)
13. If not, train custom ML model 1. use framework for training TensorFlow, keras, Turi Create , Caffe, etc ⚠ warning, there are a lot of them 1. convert to .mlmodel format (OSS) →  coremltools github.com/apple/coremltools → tf-coreml github.com/tf-coreml
15. It has been quite a year
18. beyond the cat/dog classifier (TM)
19. Some questions you may have is it practical for my app? ! barriers to entry? data, compute power, phd, time? which framework should i use for training? ! how do i navigate all of the material out there?
20. Is it practical for my app? image classification audio classification speech recognition text classification object localization gesture recognition optical character recognition (OCR) translation
21. embrace idea generation & experimentation
22. Mobile specific concerns 1. size of model 2. time it takes to run predictions 3. supported layers
25. End-to-end Process as a developer? 1.Define problem 2. Gather data 3. Train ML model 4.Convert to coreml 5. Import into Xcode project 6. Run predictions using Core ML framework
26. examples!
27. 1.Define problem American Sign Language (ASL) alphabet classifier
28. 2.Gather data !
29. barriers to entry? 1. A large dataset 2. Access to high end compute power 3. PhD in machine learning 4. All the time in the world ...nope!
30. enter.. transfer learning ! use knowledge learned from source task (MobileNet) --> to train target task (ASL classifier)
31. Quick Review: Deep Learning neural network model with many layers deep = many layers -> deep neural network https://heartbeat.fritz.ai/mobile-machine- learning-101-glossary-7a4ee36e0a1a
32. sometime way back in B.C. people used to train deep neural network from scratch
33. still some (more recent) time in B.C. people stand on the shoulders of giants' work utilizing transfer learning
34. don't reinvent the wheel
36. Transfer Learning
38. Why Transfer Learning works neural networks are universal approximators in theory, they can approximate any function
39. 2. Gather data
40. how much data do i need? depends on problem just 100s images per category
41. where can i get it? google kaggle record video + use ffmpeg
43. what if i don't have enough? data augmentation! deeplearning.ai video lesson https://www.youtube.com/watch?v=JI8saFjK84o
45. 3. Train ML model
47. confession: this is in python
53. Performance so how did our training go? ~20 min to run on my MacBook 95.3% accuracy on the test data
56. audio classification 1.Define problem 2. Gather data 3. Train ML model
57. 1.Define problem Audio classifier of urban sounds air conditioner, car horn, children playing, drilling, siren, etc
58. 2.Gather data UrbanSound 8K open dataset https://serv.cusp.nyu.edu/projects/ urbansounddataset/
59. should we use raw audio (.wav)? no, it's too computationally expensive
60. convert wav to spectrogram represent audio as image (3 dimensions) 1st dimension: time (x-axis) 2nd dimension: frequency (y-axis) 3rd dimension: sound intensity (color)
66. Performance so how did our training go? ~1 hour to run on my MacBook 77.1% accuracy on the test data
70. what to focus on
71. Where to find inspiration look at open datasets read research papers! follow heartbeat blog, openAI
72. Reproduce results research papers often include this make sure to do the same if you publish check licensing + attribute proper credit
73. Looking forward to the future ML interpretability swift for TensorFlow
74. Review is it practical for my app? ! barriers to entry? data, compute power, phd, time? which framework should i use for training? ! how do i navigate all of the material out there?
75. Resources (abridged) developer.apple.com/machine-learning codelabs.developers.google.com/codelabs/ tensorflow-for-poets/ heartbeat.fritz.ai/ serv.cusp.nyu.edu/projects/urbansounddataset/ github.com/tf-coreml/
76. Further Learning oreilly.com/data/free/files/building-mobile- applications-with-tensorflow.pdf neuralnetworksanddeeplearning.com/ cs231n.stanford.edu/ playground.tensorflow.org youtube.com/watch?v=JI8saFjK84o
77. Attributions & Mentions researchgate.net/publication/ 316748306TransferLearningofTemporalInformationfor DriverAction_Classification devonzuegel.com/post/strategically-ignorant research.googleblog.com/2018/03/the-building- blocks-of-interpretability.html
78. Grazie