Core ML 🏃 iOS Engineer

at #tokyoiosmeetup
codes: https://github.com/shingt/BeerClassifier

Transcript

1. Core ML ! iOS Engineer
   ɹ
   2017/12/16 - Tokyo iOS meetup
   @shingt (Shinichi Goto)
   

   View Slide

2. https://developer.apple.com/videos/play/wwdc2017/710/
   2
   

   View Slide

3. shingt (Shinichi Goto)
   GitHub: @shingt
   Twitter: @_shingt
   iOS Engineer at Mercari, Inc.
   3
   

   View Slide

4. Core ML
   4
   

   View Slide

5. ML (Machine Learning)
   5
   

   View Slide

6. ML (Machine Learning) tasks
   • Regression
   • Classification
   • Clustering
   • Dimension reduction
   • etc.
   6
   

   View Slide

7. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
   convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
   7
   

   View Slide

8. View Slide

9. View Slide

10. View Slide

11. Core ML
    11
    

    View Slide

12. Core ML
    ɹ
    A new foundational machine learning framework used
    across Apple products, including Siri, Camera, and
    QuickType.
    https://developer.apple.com/machine-learning/
    12
    

    View Slide

13. Core ML
    • Framework to integrate pre-trained models to apps
    • Inference only
    • Introduced Core ML model format (**.mlmodel)
    • Xcode automatically generates Swift interface for model
    • coremltools
    13
    

    View Slide

14. ɹ
    let animalModel = AnimalModel()
    if let prediction = try? animalModel.prediction(animalImage: image) {
    return prediction.animalType
    }
    14
    

    View Slide

15. View Slide

16. Model conversion
    • coremltools (provided by Apple)
    • Enables converting models trained by major tools to mlmodel
    format
    • tools: Keras, Caffe, scikit-learn, etc.
    • TensorFlow model converter
    • MXNet Model converter
    16
    

    View Slide

17. Lets' make an app!
    17
    

    View Slide

18. BeerClassifier
    • Classify beer (bottles) in real time
    • B*dweiser / Co*ona / Hei*eken
    • Input: image
    • Output: beer name
    • https://github.com/shingt/BeerClassifier
    18
    

    View Slide

19. Where is my "model"?
    19
    

    View Slide

20. https://developer.apple.com/machine-learning/
    20
    

    View Slide

21. View Slide

22. Step1: Collect images
    • Drink beer (and take photos)
    • Crawl beer (bottle) images
    • Bing Image Search API (v7)
    • Query: "B*dweiser bottle"
    • Need to exclude wrong images by myself
    • ~= 1,600 images (not big)
    • scraper.py
    22
    

    View Slide

23. • Step 1: Collect images
    • Step 2: Pre-process images
    • Step 3: Training
    • Step 4: Convert model to mlmodel format
    23
    

    View Slide

24. Step 2: Pre-process
    images
    • Process images for algorithms input
    • In my case
    • Resize (to 32x32)
    • Center-crop
    • preprocess.py
    24
    

    View Slide

25. • Step 1: Collect images
    • Step 2: Pre-process images
    • Step 3: Training
    • Step 4: Convert model to mlmodel format
    25
    

    View Slide

26. Step 3: Training
    • Using Keras
    • Construct the same network as cifar10_cnn.py
    • Available in Keras example networks
    • cifar10 (dataset): 80 million images, 10 classes
    • create_data_csv.py, train.py
    • (Skipped data augmentation)
    26
    

    View Slide

27. model = Sequential()
    model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))
    model.add(Activation('relu'))
    model.add(Conv2D(32, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(Activation('relu'))
    model.add(Conv2D(64, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes))
    model.add(Activation('softmax'))
    opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)
    model.compile(loss='categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'])
    27
    

    View Slide

28. • Step 1: Collect images
    • Step 2: Pre-process images
    • Step 3: Training
    • Step 4: Convert model to mlmodel format
    28
    

    View Slide

29. Step 4: Convert model to mlmodel format
    # convert.py
    coreml_model = coremltools.converters.keras.convert('./models/beer_model.h5',
    input_names = 'image',
    image_input_names = 'image',
    class_labels = './texts/beer_labels.txt'
    )
    coreml_model.license = 'MIT'
    coreml_model.short_description = 'Beer Classifier'
    coreml_model.author = 'shingt'
    coreml_model.save('./models/BeerClassifier.mlmodel')
    29
    

    View Slide

30. Demo
    30
    

    View Slide

31. Demo 2 (dynamic model loading)
    31
    

    View Slide

32. To replace model
    • Update model and release app again !
    OR
    • Download your model and compile it on device !
    • MLModel#compileModel(at:)
    • Ref: Downloading and Compiling a Model on the User's Device |
    Apple Developer Documentation
    32
    

    View Slide

33. Download / compile model on device
    // On background thread
    // Assume in `do` block
    let url = URL(string: "https://your_model_url.mlmodel")!
    let data = try Data(contentsOf: url)
    try data.write(to: filePathUrl)
    let compiledUrl = try MLModel.compileModel(at: filePathUrl)
    let model = try MLModel(contentsOf: compiledUrl)
    // And use model
    // ...
    33
    

    View Slide

34. Concerns / Thoughts
    34
    

    View Slide

35. Learning cost
    • Great if you work with your team !
    • And if Core ML satisfies your business requirements
    • if you're an individual developer?
    • Easy to copy-and-paste, but how to improve?
    • Understainding of statistics, learning algorithms, evaluation, ...
    • life is limited (it's great if you want to learn ML of course)
    35
    

    View Slide

36. "You do not need to be a machine learning expert to make use of
    these models. You continue to focus on the use case you're trying
    to enable and let Core ML handle those low-level details."
    https://developer.apple.com/videos/play/wwdc2017/703/
    36
    

    View Slide

37. View Slide

38. apple/turicreate
    • Application-oriented toolkit
    • Developers can focus on tasks instead of algorithms
    • Supported tasks
    • Image classification, Image similarity, Object detection
    • Recommender systems, Text classifier, Activity classification
    • Apple acquires Turi, a machine learning company | TechCrunch
    (posted Aug 5, 2016)
    38
    

    View Slide

39. # https://apple.github.io/turicreate/docs/userguide/image_classifier/introduction.html
    import turicreate as tc
    data = tc.SFrame('cats-dogs.sframe')
    train_data, test_data = data.random_split(0.8)
    # Automatically picks the right model based on your data.
    # Based on Transfer Learning
    model = tc.image_classifier.create(train_data, target='label')
    predictions = model.predict(test_data)
    metrics = model.evaluate(test_data)
    model.save('mymodel.model')
    model.export_coreml('MyCustomImageClassifier.mlmodel')
    39
    

    View Slide

40. Model size
    • BeerClassifier...5MB
    • Provided models by apple
    • ResNet50...102.6MB, VGG16...553.5 MB !
    • Integrating / replacing is not realistic
    • No model compression technique in Core ML (currently)
    • Depends on your task / application
    • Reducing the Size of Your Core ML App | Apple Developer Documentation
    40
    

    View Slide

41. Other concerns
    • Unsupported machine learning components
    • You cannot convert your model if it include components Core ML
    doesn't support
    • Model data protection
    • Parameters are not encrypted / can be seen
    • etc.
    • Why Core ML will not work for your app (most likely)
    41
    

    View Slide

42. Recap
    • Core ML basics
    • BeerClassifier
    • Concerns
    • learning cost, model size, etc.
    • apple/turicreate?
    42
    

    View Slide

43. Reference
    • Introducing Core ML
    • Core ML in depth
    • Vision Framework: Building on Core ML
    • iOS 11: Machine Learning for everyone
    • Why Core ML will not work for your app (most likely)
    • CoreMLͰΞΠυϧإೝࣝΞϓϦΛ࡞Ζ͏
    43
    

    View Slide

44. Thanks!
    44
    

    View Slide