Core ML 🏃 iOS Engineer

Transcript

1. Core ML ! iOS Engineer ɹ 2017/12/16 - Tokyo iOS

   meetup @shingt (Shinichi Goto)

2. https://developer.apple.com/videos/play/wwdc2017/710/ 2

3. shingt (Shinichi Goto) GitHub: @shingt Twitter: @_shingt iOS Engineer at

   Mercari, Inc. 3

4. Core ML 4

5. ML (Machine Learning) 5

6. ML (Machine Learning) tasks • Regression • Classification • Clustering

   • Dimension reduction • etc. 6

7. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning

   with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 7

8. 8

9. 9

10. 10

11. Core ML 11

12. Core ML ɹ A new foundational machine learning framework used

    across Apple products, including Siri, Camera, and QuickType. https://developer.apple.com/machine-learning/ 12

13. Core ML • Framework to integrate pre-trained models to apps

    • Inference only • Introduced Core ML model format (**.mlmodel) • Xcode automatically generates Swift interface for model • coremltools 13

14. ɹ let animalModel = AnimalModel() if let prediction = try?

    animalModel.prediction(animalImage: image) { return prediction.animalType } 14

15. 15

16. Model conversion • coremltools (provided by Apple) • Enables converting

    models trained by major tools to mlmodel format • tools: Keras, Caffe, scikit-learn, etc. • TensorFlow model converter • MXNet Model converter 16

17. Lets' make an app! 17

18. BeerClassifier • Classify beer (bottles) in real time • B*dweiser

    / Co*ona / Hei*eken • Input: image • Output: beer name • https://github.com/shingt/BeerClassifier 18

19. Where is my "model"? 19

20. https://developer.apple.com/machine-learning/ 20

21. 21

22. Step1: Collect images • Drink beer (and take photos) •

    Crawl beer (bottle) images • Bing Image Search API (v7) • Query: "B*dweiser bottle" • Need to exclude wrong images by myself • ~= 1,600 images (not big) • scraper.py 22

23. • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 23

24. Step 2: Pre-process images • Process images for algorithms input

    • In my case • Resize (to 32x32) • Center-crop • preprocess.py 24

25. • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 25

26. Step 3: Training • Using Keras • Construct the same

    network as cifar10_cnn.py • Available in Keras example networks • cifar10 (dataset): 80 million images, 10 classes • create_data_csv.py, train.py • (Skipped data augmentation) 26

27. model = Sequential() model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:])) model.add(Activation('relu')) model.add(Conv2D(32,

    (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Conv2D(64, (3, 3), padding='same')) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(512)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(num_classes)) model.add(Activation('softmax')) opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6) model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) 27

28. • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 28

29. Step 4: Convert model to mlmodel format # convert.py coreml_model

    = coremltools.converters.keras.convert('./models/beer_model.h5', input_names = 'image', image_input_names = 'image', class_labels = './texts/beer_labels.txt' ) coreml_model.license = 'MIT' coreml_model.short_description = 'Beer Classifier' coreml_model.author = 'shingt' coreml_model.save('./models/BeerClassifier.mlmodel') 29

30. Demo 30

31. Demo 2 (dynamic model loading) 31

32. To replace model • Update model and release app again

    ! OR • Download your model and compile it on device ! • MLModel#compileModel(at:) • Ref: Downloading and Compiling a Model on the User's Device | Apple Developer Documentation 32

33. Download / compile model on device // On background thread

    // Assume in `do` block let url = URL(string: "https://your_model_url.mlmodel")! let data = try Data(contentsOf: url) try data.write(to: filePathUrl) let compiledUrl = try MLModel.compileModel(at: filePathUrl) let model = try MLModel(contentsOf: compiledUrl) // And use model // ... 33

34. Concerns / Thoughts 34

35. Learning cost • Great if you work with your team

    ! • And if Core ML satisfies your business requirements • if you're an individual developer? • Easy to copy-and-paste, but how to improve? • Understainding of statistics, learning algorithms, evaluation, ... • life is limited (it's great if you want to learn ML of course) 35

36. "You do not need to be a machine learning expert

    to make use of these models. You continue to focus on the use case you're trying to enable and let Core ML handle those low-level details." https://developer.apple.com/videos/play/wwdc2017/703/ 36

37. 37

38. apple/turicreate • Application-oriented toolkit • Developers can focus on tasks

    instead of algorithms • Supported tasks • Image classification, Image similarity, Object detection • Recommender systems, Text classifier, Activity classification • Apple acquires Turi, a machine learning company | TechCrunch (posted Aug 5, 2016) 38

39. # https://apple.github.io/turicreate/docs/userguide/image_classifier/introduction.html import turicreate as tc data = tc.SFrame('cats-dogs.sframe') train_data,

    test_data = data.random_split(0.8) # Automatically picks the right model based on your data. # Based on Transfer Learning model = tc.image_classifier.create(train_data, target='label') predictions = model.predict(test_data) metrics = model.evaluate(test_data) model.save('mymodel.model') model.export_coreml('MyCustomImageClassifier.mlmodel') 39

40. Model size • BeerClassifier...5MB • Provided models by apple •

    ResNet50...102.6MB, VGG16...553.5 MB ! • Integrating / replacing is not realistic • No model compression technique in Core ML (currently) • Depends on your task / application • Reducing the Size of Your Core ML App | Apple Developer Documentation 40

41. Other concerns • Unsupported machine learning components • You cannot

    convert your model if it include components Core ML doesn't support • Model data protection • Parameters are not encrypted / can be seen • etc. • Why Core ML will not work for your app (most likely) 41

42. Recap • Core ML basics • BeerClassifier • Concerns •

    learning cost, model size, etc. • apple/turicreate? 42

43. Reference • Introducing Core ML • Core ML in depth

    • Vision Framework: Building on Core ML • iOS 11: Machine Learning for everyone • Why Core ML will not work for your app (most likely) • CoreMLͰΞΠυϧإೝࣝΞϓϦΛ࡞Ζ͏ 43

44. Thanks! 44