Core ML 🏃 iOS Engineer

Transcript

1. 1.

   Core ML ! iOS Engineer ɹ 2017/12/16 - Tokyo iOS

   meetup @shingt (Shinichi Goto)
2. 2.

   https://developer.apple.com/videos/play/wwdc2017/710/ 2

3. 3.

   shingt (Shinichi Goto) GitHub: @shingt Twitter: @_shingt iOS Engineer at

   Mercari, Inc. 3
4. 4.

   Core ML 4

5. 5.

   ML (Machine Learning) 5

6. 6.

   ML (Machine Learning) tasks • Regression • Classification • Clustering

   • Dimension reduction • etc. 6
7. 7.

   Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning

   with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 7
8. 8.

   8

9. 9.

   9

10. 10.

    10

11. 11.

    Core ML 11

12. 12.

    Core ML ɹ A new foundational machine learning framework used

    across Apple products, including Siri, Camera, and QuickType. https://developer.apple.com/machine-learning/ 12
13. 13.

    Core ML • Framework to integrate pre-trained models to apps

    • Inference only • Introduced Core ML model format (**.mlmodel) • Xcode automatically generates Swift interface for model • coremltools 13
14. 14.

    ɹ let animalModel = AnimalModel() if let prediction = try?

    animalModel.prediction(animalImage: image) { return prediction.animalType } 14
15. 15.

    15

16. 16.

    Model conversion • coremltools (provided by Apple) • Enables converting

    models trained by major tools to mlmodel format • tools: Keras, Caffe, scikit-learn, etc. • TensorFlow model converter • MXNet Model converter 16
17. 17.

    Lets' make an app! 17

18. 18.

    BeerClassifier • Classify beer (bottles) in real time • B*dweiser

    / Co*ona / Hei*eken • Input: image • Output: beer name • https://github.com/shingt/BeerClassifier 18
19. 19.

    Where is my "model"? 19

20. 20.

    https://developer.apple.com/machine-learning/ 20

21. 21.

    21

22. 22.

    Step1: Collect images • Drink beer (and take photos) •

    Crawl beer (bottle) images • Bing Image Search API (v7) • Query: "B*dweiser bottle" • Need to exclude wrong images by myself • ~= 1,600 images (not big) • scraper.py 22
23. 23.

    • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 23
24. 24.

    Step 2: Pre-process images • Process images for algorithms input

    • In my case • Resize (to 32x32) • Center-crop • preprocess.py 24
25. 25.

    • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 25
26. 26.

    Step 3: Training • Using Keras • Construct the same

    network as cifar10_cnn.py • Available in Keras example networks • cifar10 (dataset): 80 million images, 10 classes • create_data_csv.py, train.py • (Skipped data augmentation) 26
27. 27.

    model = Sequential() model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:])) model.add(Activation('relu')) model.add(Conv2D(32,

    (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Conv2D(64, (3, 3), padding='same')) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(512)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(num_classes)) model.add(Activation('softmax')) opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6) model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) 27
28. 28.

    • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 28
29. 29.

    Step 4: Convert model to mlmodel format # convert.py coreml_model

    = coremltools.converters.keras.convert('./models/beer_model.h5', input_names = 'image', image_input_names = 'image', class_labels = './texts/beer_labels.txt' ) coreml_model.license = 'MIT' coreml_model.short_description = 'Beer Classifier' coreml_model.author = 'shingt' coreml_model.save('./models/BeerClassifier.mlmodel') 29
30. 30.

    Demo 30

31. 31.

    Demo 2 (dynamic model loading) 31

32. 32.

    To replace model • Update model and release app again

    ! OR • Download your model and compile it on device ! • MLModel#compileModel(at:) • Ref: Downloading and Compiling a Model on the User's Device | Apple Developer Documentation 32
33. 33.

    Download / compile model on device // On background thread

    // Assume in `do` block let url = URL(string: "https://your_model_url.mlmodel")! let data = try Data(contentsOf: url) try data.write(to: filePathUrl) let compiledUrl = try MLModel.compileModel(at: filePathUrl) let model = try MLModel(contentsOf: compiledUrl) // And use model // ... 33
34. 34.

    Concerns / Thoughts 34

35. 35.

    Learning cost • Great if you work with your team

    ! • And if Core ML satisfies your business requirements • if you're an individual developer? • Easy to copy-and-paste, but how to improve? • Understainding of statistics, learning algorithms, evaluation, ... • life is limited (it's great if you want to learn ML of course) 35
36. 36.

    "You do not need to be a machine learning expert

    to make use of these models. You continue to focus on the use case you're trying to enable and let Core ML handle those low-level details." https://developer.apple.com/videos/play/wwdc2017/703/ 36
37. 37.

    37

38. 38.

    apple/turicreate • Application-oriented toolkit • Developers can focus on tasks

    instead of algorithms • Supported tasks • Image classification, Image similarity, Object detection • Recommender systems, Text classifier, Activity classification • Apple acquires Turi, a machine learning company | TechCrunch (posted Aug 5, 2016) 38
39. 39.

    # https://apple.github.io/turicreate/docs/userguide/image_classifier/introduction.html import turicreate as tc data = tc.SFrame('cats-dogs.sframe') train_data,

    test_data = data.random_split(0.8) # Automatically picks the right model based on your data. # Based on Transfer Learning model = tc.image_classifier.create(train_data, target='label') predictions = model.predict(test_data) metrics = model.evaluate(test_data) model.save('mymodel.model') model.export_coreml('MyCustomImageClassifier.mlmodel') 39
40. 40.

    Model size • BeerClassifier...5MB • Provided models by apple •

    ResNet50...102.6MB, VGG16...553.5 MB ! • Integrating / replacing is not realistic • No model compression technique in Core ML (currently) • Depends on your task / application • Reducing the Size of Your Core ML App | Apple Developer Documentation 40
41. 41.

    Other concerns • Unsupported machine learning components • You cannot

    convert your model if it include components Core ML doesn't support • Model data protection • Parameters are not encrypted / can be seen • etc. • Why Core ML will not work for your app (most likely) 41
42. 42.

    Recap • Core ML basics • BeerClassifier • Concerns •

    learning cost, model size, etc. • apple/turicreate? 42
43. 43.

    Reference • Introducing Core ML • Core ML in depth

    • Vision Framework: Building on Core ML • iOS 11: Machine Learning for everyone • Why Core ML will not work for your app (most likely) • CoreMLͰΞΠυϧإೝࣝΞϓϦΛ࡞Ζ͏ 43
44. 44.

    Thanks! 44