Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Core ML 🏃 iOS Engineer

Core ML 🏃 iOS Engineer

at #tokyoiosmeetup
codes: https://github.com/shingt/BeerClassifier

Shinichi Goto

December 16, 2017
Tweet

More Decks by Shinichi Goto

Other Decks in Programming

Transcript

  1. Core ML ! iOS Engineer
    ɹ
    2017/12/16 - Tokyo iOS meetup
    @shingt (Shinichi Goto)

    View Slide

  2. https://developer.apple.com/videos/play/wwdc2017/710/
    2

    View Slide

  3. shingt (Shinichi Goto)
    GitHub: @shingt
    Twitter: @_shingt
    iOS Engineer at Mercari, Inc.
    3

    View Slide

  4. Core ML
    4

    View Slide

  5. ML (Machine Learning)
    5

    View Slide

  6. ML (Machine Learning) tasks
    • Regression
    • Classification
    • Clustering
    • Dimension reduction
    • etc.
    6

    View Slide

  7. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
    convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
    7

    View Slide

  8. 8

    View Slide

  9. 9

    View Slide

  10. 10

    View Slide

  11. Core ML
    11

    View Slide

  12. Core ML
    ɹ
    A new foundational machine learning framework used
    across Apple products, including Siri, Camera, and
    QuickType.
    https://developer.apple.com/machine-learning/
    12

    View Slide

  13. Core ML
    • Framework to integrate pre-trained models to apps
    • Inference only
    • Introduced Core ML model format (**.mlmodel)
    • Xcode automatically generates Swift interface for model
    • coremltools
    13

    View Slide

  14. ɹ
    let animalModel = AnimalModel()
    if let prediction = try? animalModel.prediction(animalImage: image) {
    return prediction.animalType
    }
    14

    View Slide

  15. 15

    View Slide

  16. Model conversion
    • coremltools (provided by Apple)
    • Enables converting models trained by major tools to mlmodel
    format
    • tools: Keras, Caffe, scikit-learn, etc.
    • TensorFlow model converter
    • MXNet Model converter
    16

    View Slide

  17. Lets' make an app!
    17

    View Slide

  18. BeerClassifier
    • Classify beer (bottles) in real time
    • B*dweiser / Co*ona / Hei*eken
    • Input: image
    • Output: beer name
    • https://github.com/shingt/BeerClassifier
    18

    View Slide

  19. Where is my "model"?
    19

    View Slide

  20. https://developer.apple.com/machine-learning/
    20

    View Slide

  21. 21

    View Slide

  22. Step1: Collect images
    • Drink beer (and take photos)
    • Crawl beer (bottle) images
    • Bing Image Search API (v7)
    • Query: "B*dweiser bottle"
    • Need to exclude wrong images by myself
    • ~= 1,600 images (not big)
    • scraper.py
    22

    View Slide

  23. • Step 1: Collect images
    • Step 2: Pre-process images
    • Step 3: Training
    • Step 4: Convert model to mlmodel format
    23

    View Slide

  24. Step 2: Pre-process
    images
    • Process images for algorithms input
    • In my case
    • Resize (to 32x32)
    • Center-crop
    • preprocess.py
    24

    View Slide

  25. • Step 1: Collect images
    • Step 2: Pre-process images
    • Step 3: Training
    • Step 4: Convert model to mlmodel format
    25

    View Slide

  26. Step 3: Training
    • Using Keras
    • Construct the same network as cifar10_cnn.py
    • Available in Keras example networks
    • cifar10 (dataset): 80 million images, 10 classes
    • create_data_csv.py, train.py
    • (Skipped data augmentation)
    26

    View Slide

  27. model = Sequential()
    model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))
    model.add(Activation('relu'))
    model.add(Conv2D(32, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(Activation('relu'))
    model.add(Conv2D(64, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes))
    model.add(Activation('softmax'))
    opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)
    model.compile(loss='categorical_crossentropy',
    optimizer=opt,
    metrics=['accuracy'])
    27

    View Slide

  28. • Step 1: Collect images
    • Step 2: Pre-process images
    • Step 3: Training
    • Step 4: Convert model to mlmodel format
    28

    View Slide

  29. Step 4: Convert model to mlmodel format
    # convert.py
    coreml_model = coremltools.converters.keras.convert('./models/beer_model.h5',
    input_names = 'image',
    image_input_names = 'image',
    class_labels = './texts/beer_labels.txt'
    )
    coreml_model.license = 'MIT'
    coreml_model.short_description = 'Beer Classifier'
    coreml_model.author = 'shingt'
    coreml_model.save('./models/BeerClassifier.mlmodel')
    29

    View Slide

  30. Demo
    30

    View Slide

  31. Demo 2 (dynamic model loading)
    31

    View Slide

  32. To replace model
    • Update model and release app again !
    OR
    • Download your model and compile it on device !
    • MLModel#compileModel(at:)
    • Ref: Downloading and Compiling a Model on the User's Device |
    Apple Developer Documentation
    32

    View Slide

  33. Download / compile model on device
    // On background thread
    // Assume in `do` block
    let url = URL(string: "https://your_model_url.mlmodel")!
    let data = try Data(contentsOf: url)
    try data.write(to: filePathUrl)
    let compiledUrl = try MLModel.compileModel(at: filePathUrl)
    let model = try MLModel(contentsOf: compiledUrl)
    // And use model
    // ...
    33

    View Slide

  34. Concerns / Thoughts
    34

    View Slide

  35. Learning cost
    • Great if you work with your team !
    • And if Core ML satisfies your business requirements
    • if you're an individual developer?
    • Easy to copy-and-paste, but how to improve?
    • Understainding of statistics, learning algorithms, evaluation, ...
    • life is limited (it's great if you want to learn ML of course)
    35

    View Slide

  36. "You do not need to be a machine learning expert to make use of
    these models. You continue to focus on the use case you're trying
    to enable and let Core ML handle those low-level details."
    https://developer.apple.com/videos/play/wwdc2017/703/
    36

    View Slide

  37. 37

    View Slide

  38. apple/turicreate
    • Application-oriented toolkit
    • Developers can focus on tasks instead of algorithms
    • Supported tasks
    • Image classification, Image similarity, Object detection
    • Recommender systems, Text classifier, Activity classification
    • Apple acquires Turi, a machine learning company | TechCrunch
    (posted Aug 5, 2016)
    38

    View Slide

  39. # https://apple.github.io/turicreate/docs/userguide/image_classifier/introduction.html
    import turicreate as tc
    data = tc.SFrame('cats-dogs.sframe')
    train_data, test_data = data.random_split(0.8)
    # Automatically picks the right model based on your data.
    # Based on Transfer Learning
    model = tc.image_classifier.create(train_data, target='label')
    predictions = model.predict(test_data)
    metrics = model.evaluate(test_data)
    model.save('mymodel.model')
    model.export_coreml('MyCustomImageClassifier.mlmodel')
    39

    View Slide

  40. Model size
    • BeerClassifier...5MB
    • Provided models by apple
    • ResNet50...102.6MB, VGG16...553.5 MB !
    • Integrating / replacing is not realistic
    • No model compression technique in Core ML (currently)
    • Depends on your task / application
    • Reducing the Size of Your Core ML App | Apple Developer Documentation
    40

    View Slide

  41. Other concerns
    • Unsupported machine learning components
    • You cannot convert your model if it include components Core ML
    doesn't support
    • Model data protection
    • Parameters are not encrypted / can be seen
    • etc.
    • Why Core ML will not work for your app (most likely)
    41

    View Slide

  42. Recap
    • Core ML basics
    • BeerClassifier
    • Concerns
    • learning cost, model size, etc.
    • apple/turicreate?
    42

    View Slide

  43. Reference
    • Introducing Core ML
    • Core ML in depth
    • Vision Framework: Building on Core ML
    • iOS 11: Machine Learning for everyone
    • Why Core ML will not work for your app (most likely)
    • CoreMLͰΞΠυϧإೝࣝΞϓϦΛ࡞Ζ͏
    43

    View Slide

  44. Thanks!
    44

    View Slide