Core ML ! iOS Engineer ɹ 2017/12/16 - Tokyo iOS meetup @shingt (Shinichi Goto) 

https://developer.apple.com/videos/play/wwdc2017/710/ 2 

shingt (Shinichi Goto) GitHub: @shingt Twitter: @_shingt iOS Engineer at Mercari, Inc. 3 

Core ML 4 

ML (Machine Learning) 5 

ML (Machine Learning) tasks • Regression • Classification • Clustering • Dimension reduction • etc. 6 

Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 7 

8 

9 

10 

Core ML 11 

Core ML ɹ A new foundational machine learning framework used across Apple products, including Siri, Camera, and QuickType. https://developer.apple.com/machine-learning/ 12 

Core ML • Framework to integrate pre-trained models to apps • Inference only • Introduced Core ML model format (**.mlmodel) • Xcode automatically generates Swift interface for model • coremltools 13 

ɹ let animalModel = AnimalModel() if let prediction = try? animalModel.prediction(animalImage: image) { return prediction.animalType } 14 

15 

Model conversion • coremltools (provided by Apple) • Enables converting models trained by major tools to mlmodel format • tools: Keras, Caffe, scikit-learn, etc. • TensorFlow model converter • MXNet Model converter 16 

Lets' make an app! 17 

BeerClassifier • Classify beer (bottles) in real time • B*dweiser / Co*ona / Hei*eken • Input: image • Output: beer name • https://github.com/shingt/BeerClassifier 18 

Where is my "model"? 19 

https://developer.apple.com/machine-learning/ 20 

21 

Step1: Collect images • Drink beer (and take photos) • Crawl beer (bottle) images • Bing Image Search API (v7) • Query: "B*dweiser bottle" • Need to exclude wrong images by myself • ~= 1,600 images (not big) • scraper.py 22 

• Step 1: Collect images • Step 2: Pre-process images • Step 3: Training • Step 4: Convert model to mlmodel format 23 

Step 2: Pre-process images • Process images for algorithms input • In my case • Resize (to 32x32) • Center-crop • preprocess.py 24 

• Step 1: Collect images • Step 2: Pre-process images • Step 3: Training • Step 4: Convert model to mlmodel format 25 

Step 3: Training • Using Keras • Construct the same network as cifar10_cnn.py • Available in Keras example networks • cifar10 (dataset): 80 million images, 10 classes • create_data_csv.py, train.py • (Skipped data augmentation) 26 

model = Sequential() model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:])) model.add(Activation('relu')) model.add(Conv2D(32, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Conv2D(64, (3, 3), padding='same')) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(512)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(num_classes)) model.add(Activation('softmax')) opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6) model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) 27 

• Step 1: Collect images • Step 2: Pre-process images • Step 3: Training • Step 4: Convert model to mlmodel format 28 

Step 4: Convert model to mlmodel format # convert.py coreml_model = coremltools.converters.keras.convert('./models/beer_model.h5', input_names = 'image', image_input_names = 'image', class_labels = './texts/beer_labels.txt' ) coreml_model.license = 'MIT' coreml_model.short_description = 'Beer Classifier' coreml_model.author = 'shingt' coreml_model.save('./models/BeerClassifier.mlmodel') 29 

Demo 30 

Demo 2 (dynamic model loading) 31 

To replace model • Update model and release app again ! OR • Download your model and compile it on device ! • MLModel#compileModel(at:) • Ref: Downloading and Compiling a Model on the User's Device | Apple Developer Documentation 32 

Download / compile model on device // On background thread // Assume in `do` block let url = URL(string: "https://your_model_url.mlmodel")! let data = try Data(contentsOf: url) try data.write(to: filePathUrl) let compiledUrl = try MLModel.compileModel(at: filePathUrl) let model = try MLModel(contentsOf: compiledUrl) // And use model // ... 33 

Concerns / Thoughts 34 

Learning cost • Great if you work with your team ! • And if Core ML satisfies your business requirements • if you're an individual developer? • Easy to copy-and-paste, but how to improve? • Understainding of statistics, learning algorithms, evaluation, ... • life is limited (it's great if you want to learn ML of course) 35 

"You do not need to be a machine learning expert to make use of these models. You continue to focus on the use case you're trying to enable and let Core ML handle those low-level details." https://developer.apple.com/videos/play/wwdc2017/703/ 36 

37 

apple/turicreate • Application-oriented toolkit • Developers can focus on tasks instead of algorithms • Supported tasks • Image classification, Image similarity, Object detection • Recommender systems, Text classifier, Activity classification • Apple acquires Turi, a machine learning company | TechCrunch (posted Aug 5, 2016) 38 

# https://apple.github.io/turicreate/docs/userguide/image_classifier/introduction.html import turicreate as tc data = tc.SFrame('cats-dogs.sframe') train_data, test_data = data.random_split(0.8) # Automatically picks the right model based on your data. # Based on Transfer Learning model = tc.image_classifier.create(train_data, target='label') predictions = model.predict(test_data) metrics = model.evaluate(test_data) model.save('mymodel.model') model.export_coreml('MyCustomImageClassifier.mlmodel') 39 

Model size • BeerClassifier...5MB • Provided models by apple • ResNet50...102.6MB, VGG16...553.5 MB ! • Integrating / replacing is not realistic • No model compression technique in Core ML (currently) • Depends on your task / application • Reducing the Size of Your Core ML App | Apple Developer Documentation 40 

Other concerns • Unsupported machine learning components • You cannot convert your model if it include components Core ML doesn't support • Model data protection • Parameters are not encrypted / can be seen • etc. • Why Core ML will not work for your app (most likely) 41 

Recap • Core ML basics • BeerClassifier • Concerns • learning cost, model size, etc. • apple/turicreate? 42 

Reference • Introducing Core ML • Core ML in depth • Vision Framework: Building on Core ML • iOS 11: Machine Learning for everyone • Why Core ML will not work for your app (most likely) • CoreMLͰΞΠυϧإೝࣝΞϓϦΛ࡞Ζ͏ 43 

Thanks! 44