Core ML Overview

Core ML Framework and coremltools

Transcript

1. New framework for on-device inference. It is available on macOS, iOS, watchOS, and tvOS. Core ML
2. Core ML is the foundation for domain specific frameworks and functionality. Vision framework: to be used for vision based apps. It can do Object Tracking - Object Detection. NLP framework: to be used for text based apps. It can do Language Identification - Name, Location, Organization Identification. Core ML: to be used for purposes other than those two. Core ML is built on top of low-level primitives. ML Frameworks App Vision / NLP (Domain Specific Frameworks) Core ML Accelerator / MPS (ML Performance Primitives)
3. Core ML is optimized for hardware, and switching between CPU and GPU is managed by the framework so can do both memory / compute heavy tasks efficiently. How on-device machine learning can be useful? — User Privacy — No extra cost for user mobile data — No extra costs for server hosting that maybe used to get prediction — App is always available and functional even when a network connection is unavailable — Real Time ML
4. Core ML is optimized for on-device performance, which minimizes — memory footprint and — power consumption.
5. A common type of function is performing some sort of classification. It's taking a set of inputs then assigning some categorical label. We need to know the model functional description, its inputs, outputs, types, and the details that Core ML needs to actually execute the function. What is a model? Since Core ML, a model is simply a function.
6. The logic for this function is to be learned from data. Source: WWDC 2017- session 703
7. Source: WWDC 2017- session 703
8. From where we get the Core ML format models? — Apple ready-to-use Core ML Models. — Apple tool (Python Package) to convert known model formats to Core ML model format.
9. Core ML represents a model in a single document. This document has the high level information that a developer needs to program against. It has a set of converters, one for each popular training library. Underneath that, there are Core ML bindings and a converter library. Core ML bindings allow you to call directly into Core ML from python. Converter library is a high-level API for building converters for new formats. Source: WWDC 2017- session 703
10. Prepare environment to work with Python and install coremltools — Install a package and environment manager; Anaconda
11. It comes with conda (which is package and environment manager), Python plus over 150 scientific packages and their dependencies. If you donʼt need all of that you can use Miniconda, a smaller distribution that includes only conda and Python, then you can install any package you need individually. But why use environments? Environments enable you to isolate the packages you are using for different projects. So you can have both python 3 and 2 on the same machine. coremltools supports python 2. Prepare environment to work with Python and install coremltools — Create the environments with the packages you need conda create -n py3 python=3 conda create -n py2 python=2 conda env list # conda environments: # py2 /anaconda/envs/py2 py3 /anaconda/envs/py3 root * /anaconda
12. pip which is the default package manager for Python libraries. We may still use pip alongside conda to install packages because the available packages from conda are focused around data science while pip is for general use. -U in pip command: Upgrade package if exists Prepare environment to work with Python and install coremltools — Install coremltools source activate py2 pip install -U coremltools
13. A model in source format, ex: Caffe — .prototxt the structure of the neural network — .caffemodel learned weights in that model when caffe is doing inference, it may take an image and give back an index of a class label, so we need a third file — labels.txt that maps these indices to a string class label.
14. Parameter imageinputnames means that we want the model to take an image as an input instead of multi-array. You will see a message stating that “Starting Conversion from Caffe to CoreML” then after some time depending on the model size, you will get the output file of model EmotiWVGGS.mlmodel Example: Convert Emotion Recognition trained model Caffe format ➡ Core ML Model Format save the following script to a python file (conversion.py) import coremltools caffe_model = ('EmotiW_VGG_S.caffemodel', 'deploy.prototxt') labels = 'labels.txt' coreml_model = coremltools.converters.caffe.convert(caffe_model, class_labels=labels, image_input_names='data') coreml_model.save('EmotiW_VGG_S.mlmodel') Then run the script in terminal python conversion.py
15. That will give you a programmatic interface to this model using data types and structures you're already familiar programming against. It is optimized for run time on the device. How to use Core ML Model in your app? when import a model into project, Xcode will generate interface for it. Also the model will be compiled and bundled into the app.
16. Make sure that model is added to the app target. These inputs and outputs can be of five different types; numeric, categorical, images, arrays, and dictionaries. numerics and categories/discrete exposed as doubles, integers or strings. images —> CVPixelBuffers. For the more complex things like gestures, audio and video, we have a new type called MLMultiArray to encapsulate a multidimensional array. For a lot of text-based applications, we may be interacting with dictionaries; The keys can be strings or integers, and the values are doubles. Development Flow Drag the Core ML model to Xcode
17. This is the interface CVPixelBuffer : raw image format in CoreVideo internal format; an image buffer that holds pixels in main memory. It may be used for applications for generating frames, compressing or decompressing video, or using Core Image. we have 3 classes; input, output, and the model class itself. Swift generated code interface
18. we see the initializers and the prediction methods as well. we have access to the underlying model. The model class
19. Source: WWDC 2017- session 703
20. How to use the model in code to do prediction?
21. We convert the image to CVPixelBuffer format using CVPixelBufferCreate, indicating the desired image size; width and height.
22. Vision Framework
23. Vision is recommended to be used whenever youʼre using Core ML with images or video. When using only Core ML you need to make sure your input image is in the format the model expects, but with Vision the framework takes care of resizing the image, etc. Note that VNImageRequestHandler takes an array of request objects. Vision Framework
24. Prediction Results
25. app may run fine on simulator but crash on device; cause in some cases simulator may use the Accelerate framework but the device uses Metal Performance Shaders. So unexpected results may happen. If you face such a case, you should report a bug to Apple. Performing Tests — Should always be tested on real devices.
26. compileModel() Compile a model on device to install or update the model over the air. Recent updates: As of iOS 11 beta 4: — MLModel class has a new compileModel() function — MLPredictionOptions with a usesCPUOnly property
27. not supporting all other types, including clustering, ranking, anomaly and novelty detection, etc. Model files are not encrypted or secured. Ex: Caffe neural network gets converted to several JSON files with layers description and binaries with weights data. The argument for the privacy point is that if you want to train the model, userʼs data have to be collected and uploaded to the servers anyway. But Apple published a research paper that focuses on making realistic fake images mostly of humans to train facial recognition AI. Thatʼs important progress for Apple and any company interested in generating good data for training models and not to violate user privacy in the same time.. Core ML Limitations — Core ML is not a machine learning framework. — Core ML supports only two types of ML; regression and classification. — If your models are proprietary or contain sensitive information, you can’t use CoreML. — Core ML doesn’t really solve the privacy concerns.
28. Modern neural networks can easily be hundreds of MB and even GBs. We should figure out model compression on our own; such as network pruning. Core ML Limitations — Core ML doesn’t compress models.
29. What’s next? Learn how to build and train my own models. Learn about machine learning APIs other than Core ML.
30. Resources Apple Documentation Core ML Apple ready-to-use Core ML Models Apple AI Research Paper The Paper Summary
31. Resources Videos Introducing Core ML - WWDC 2017 Core ML in depth - WWDC 2017 Articles CoreML and coremltools iOS 11: Machine Learning for everyone Why Core ML will not work for your app (most likely)
32. Resources Tools coremltools 0.4.0 Anaconda Miniconda Emotion Recognition Trained Model
33. Thank you ❤ Let’s learn about Machine Learning.