Pragmatic Machine Learning for iOS apps

About • Khoa • iOS developer, at Shortcut • https:/ /onmyway133.github.io

Agenda • Deep Learning • Layer in Neural Network • Activation Functions • Training in Neural Network • Loss, Learning Rate • Train, Test & Validation Sets • Predicting • Overfitting

Agenda (cont) • Convolutional Neural networks & visualizing • Zero padding, max pooling • Explain back propagation • Vanishing & Exploding Gradient • Weight Initialization & Bias • Learnable parameters • Regularizationm, batch size, fine tuning and batch normalization

Pragmatic

My Sentosa

Pragmatic

Pragmatic

Pragmatic How I replicated an $86 million project in 57 lines of code

Farming

CoreML

CoreML • Computer Vision: image classification • Natural Language: language idenification, tokenization • Speech: speech recognition

Image classification Preprocess photos using the Vision framework and classify them with a Core ML model. iOS app Avengers https:/ /github.com/onmyway133/avengers

Avengers • SwiftUI • CoreML • Vision

Tools • IBM Watson • Azure Custom Vision • Google AutoML • Firebase Vision Edge • CreateML • Turi Create

Data set • Google images download https:/ /github.com/hardikvasa/google- images-download • Data augmentation

IBM Watson Visual Recognition • https:/ /www.ibm.com/watson/services/visual-recognition/ • https:/ /cloud.ibm.com/developer/watson/services • https:/ /www.ibm.com/cloud/watson-studio • https:/ /dataplatform.cloud.ibm.com/home

IBM Watson Visual Recognition - Service

IBM Watson Visual Recognition - Assets

IBM Watson Visual Recognition - Train

IBM Watson Visual Recognition - Watson SDK let classifierID = "your-classifier-id" let failure = { (error: Error) in print(error) } visualRecognition.updateLocalModel(classifierID: classifierID, failure: failure) { print("model updated") }

Vision

Vision

CoreML + Vision let model = try VNCoreMLModel(for: IBMWatson().model) let request = VNCoreMLRequest(model: model, completionHandler: { request, error in let results = request.results as? [VNClassificationObservation], let handler = VNImageRequestHandler(cgImage: image.cgImage!, options: [:]) try handler.perform([request])

Azure Custom Vision • https:/ /www.customvision.ai/

Azure Custom Vision - Project

Azure Custom Vision - Assets

Azure Custom Vision - Train

Google Cloud AutoML Vision • https:/ /cloud.google.com/automl/ • https:/ /console.cloud.google.com/vision

Google Cloud AutoML Vision

Google Cloud AutoML Vision - Dataset

Google Cloud AutoML Vision - Google Cloud Storage https:/ /console.cloud.google.com/storage

Firebase AutoML Vision Edge

Firebase AutoML Vision Edge - Dataset

Firebase AutoML Vision Edge - Dataset

Firebase AutoML Vision Edge - Train pod 'Firebase/MLModelInterpreter'

Fritz AI

CreateML • Activity, Sound, Image, Text, Tabular Classification • Word tagger • Recommendor • Object detection

CreateML - Create Document

CreateML - Data

CreateML - Train

CreateMLUI Playground • macOS Playground

Turi Create • https:/ /github.com/apple/turicreate • Open source Python framework • Latest version 5.8

Turi Create

Turi Create - SFrame import turicreate as tc import os # 1. Load images data = tc.image_analysis.load_images('dataset', with_path=True) # 2. Create label column based on folder name data['hero_name'] = data['path'].apply(lambda path: os.path.basename(os.path.dirname(path))) # 3. Save as .sframe data.save('turi.sframe') # 4. Explore data.explore()

Turi Create - Visualization

Turi Create - Training import turicreate as tc # 1. Load the data data = tc.SFrame('turi.sframe') # 2. Split to train and test data train_data, test_data = data.random_split(0.8) # 3. Create model model = tc.image_classifier.create(train_data, target='hero_name') # 4. Predictions predictions = model.predict(test_data) # 5. Evaluate the model and show metrics metrics = model.evaluate(test_data) print(metrics['accuracy']) # 6. Save the model model.save('turi.model') # 7. Export to CoreML format model.export_coreml('model/TuriCreate.mlmodel')

Turi Create - Transfer Learning resnet-50 model = tc.image_classifier.create(train_data, target='hero_name', model='squeezenet_v1.1')

Writing • Machine Learning in iOS: IBM Watson and CoreML • Machine Learning in iOS: Azure Custom Vision and CoreML • Machine Learning in iOS: Turi Create and CoreML • Vision in iOS: Text detection and Tesseract recognition

Thank you May your code continue to compile