Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Firebase ML Kit for iOS Developer
Search
Kajornsak Peerapathananont
October 07, 2018
Technology
0
62
Firebase ML Kit for iOS Developer
Firebase Dev Day 2018 @Bangkok, Thailand
Kajornsak Peerapathananont
October 07, 2018
Tweet
Share
More Decks by Kajornsak Peerapathananont
See All by Kajornsak Peerapathananont
Understanding your Android build
kajornsakp
0
29
iOSDevTH #21
kajornsakp
0
35
What's new in Flutter (Google I/O Extended Bangkok 22)
kajornsakp
0
52
Mobile Design System at scale
kajornsakp
0
77
What's new in Flutter 2020
kajornsakp
0
54
Mobile Machine Learning for All Skill Levels
kajornsakp
0
28
What's new in Flutter 1.9
kajornsakp
0
47
Kotlin meets Web
kajornsakp
0
19
From design to develop with Material Components
kajornsakp
0
130
Other Decks in Technology
See All in Technology
AI Agent時代なのでAWSのLLMs.txtが欲しい!
watany
2
220
30→150人のエンジニア組織拡大に伴うアジャイル文化を醸成する役割と取り組みの変化
nagata03
0
150
遷移の高速化 ヤフートップの試行錯誤
narirou
6
1.1k
Visualize, Visualize, Visualize and rclone
tomoaki0705
9
82k
Raycast AI APIを使ってちょっと便利な拡張機能を作ってみた / created-a-handy-extension-using-the-raycast-ai-api
kawamataryo
0
210
短縮URLをお手軽に導入しよう
nakasho
0
150
Raycast Favorites × Script Command で実現するお手軽情報チェック
smasato
1
140
株式会社Awarefy(アウェアファイ)会社説明資料 / Awarefy-Company-Deck
awarefy
3
11k
AIエージェント入門
minorun365
PRO
31
17k
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
17
45k
役員・マネージャー・著者・エンジニアそれぞれの立場から見たAWS認定資格
nrinetcom
PRO
3
5.9k
AWSではじめる Web APIテスト実践ガイド / A practical guide to testing Web APIs on AWS
yokawasa
7
660
Featured
See All Featured
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Site-Speed That Sticks
csswizardry
4
400
Faster Mobile Websites
deanohume
306
31k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
GraphQLとの向き合い方2022年版
quramy
44
14k
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
The Cult of Friendly URLs
andyhume
78
6.2k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Optimising Largest Contentful Paint
csswizardry
34
3.1k
Visualization
eitanlees
146
15k
Automating Front-end Workflow
addyosmani
1368
200k
Transcript
ML Kit for iOS developers Kajornsak Peerapathananont Agoda
Machine Learning
#FirebaseDevDay
Google Lens
Smart Reply
On-device Machine Learning
#FirebaseDevDay Doable, but hard.
#FirebaseDevDay
#FirebaseDevDay Get Image Image Classification Transform Interpret Get Result
#FirebaseDevDay Transform unsigned char *sourceBaseAddr = (unsigned char *)(CVPixelBufferGetBaseAddress(pixelBuffer)); int
image_height; unsigned char *sourceStartAddr; if (fullHeight <= image_width) { image_height = fullHeight; sourceStartAddr = sourceBaseAddr; } else { image_height = image_width; const int marginY = ((fullHeight - image_width) / 2); sourceStartAddr = (sourceBaseAddr + (marginY * sourceRowBytes)); } const int image_channels = 4; assert(image_channels >= wanted_input_channels); tensorflow::Tensor image_tensor( tensorflow::DT_FLOAT, tensorflow::TensorShape( {1, wanted_input_height, wanted_input_width, wanted_input_channels})); auto image_tensor_mapped = image_tensor.tensor<float, 4>(); tensorflow::uint8 *in = sourceStartAddr; float *out = image_tensor_mapped.data(); for (int y = 0; y < wanted_input_height; ++y) { float *out_row = out + (y * wanted_input_width * wanted_input_channels); for (int x = 0; x < wanted_input_width; ++x) { const int in_x = (y * image_width) / wanted_input_width; const int in_y = (x * image_height) / wanted_input_height; tensorflow::uint8 *in_pixel = in + (in_y * image_width * image_channels) + (in_x * image_channels); float *out_pixel = out_row + (x * wanted_input_channels); for (int c = 0; c < wanted_input_channels; ++c) { out_pixel[c] = (in_pixel[c] - input_mean) / input_std; } } }
#FirebaseDevDay Interpret if (tf_session.get()) { std::vector<tensorflow::Tensor> outputs; tensorflow::Status run_status =
tf_session->Run( {{input_layer_name, image_tensor}}, {output_layer_name}, {}, &outputs); if (!run_status.ok()) { LOG(ERROR) << "Running model failed:" << run_status; } else { tensorflow::Tensor *output = &outputs[0]; auto predictions = output->flat<float>(); NSMutableDictionary *newValues = [NSMutableDictionary dictionary]; for (int index = 0; index < predictions.size(); index += 1) { const float predictionValue = predictions(index); if (predictionValue > 0.05f) { std::string label = labels[index % predictions.size()]; NSString *labelObject = [NSString stringWithUTF8String:label.c_str()]; NSNumber *valueObject = [NSNumber numberWithFloat:predictionValue]; [newValues setObject:valueObject forKey:labelObject]; } } dispatch_async(dispatch_get_main_queue(), ^(void) { [self setPredictionValues:newValues]; }); } }
None
#FirebaseDevDay
#FirebaseDevDay Real-world Common Use Cases
#FirebaseDevDay FIRVisionImage | VisionImage NS_SWIFT_NAME(VisionImage) @interface FIRVisionImage : NSObject @property(nonatomic,
nullable) FIRVisionImageMetadata *metadata; - (instancetype)initWithImage:(UIImage *)image NS_DESIGNATED_INITIALIZER; - (instancetype)initWithBuffer:(CMSampleBufferRef)sampleBuffer NS_DESIGNATED_INITIALIZER; - (instancetype)init NS_UNAVAILABLE; @end
Text Recognition - On-device - On-cloud
#FirebaseDevDay https://firebase.google.com/docs/ml-kit/recognize-text
#FirebaseDevDay FIRVisionText | VisionText NS_SWIFT_NAME(VisionText) @interface FIRVisionText : NSObject @property(nonatomic,
readonly) NSString *text; @property(nonatomic, readonly) NSArray<FIRVisionTextBlock *> *blocks; - (instancetype)init NS_UNAVAILABLE; @end
#FirebaseDevDay On-device Usage let textRecognizer = vision.onDeviceTextRecognizer() textRecognizer.process(visionImage) { (text,
error) in guard let text = text else { return } // do something with your text }
#FirebaseDevDay On-cloud Usage let textRecognizer = vision.cloudTextRecognize() textRecognizer.process(visionImage) { (text,
error) in guard let text = text else { return } // do something with your text }
Image Labeling - On-device (400+ labels) - On-cloud (10,000+ labels)
#FirebaseDevDay https://firebase.google.com/docs/ml-kit/label-images
#FirebaseDevDay FIRVisionLabel | VisionLabel NS_SWIFT_NAME(VisionLabel) @interface FIRVisionLabel : NSObject @property(nonatomic,
readonly) CGRect frame; @property(nonatomic, readonly) float confidence; @property(nonatomic, copy, readonly) NSString *entityID; @property(nonatomic, copy, readonly) NSString *label; @end
#FirebaseDevDay On-device Usage let labelDetector = vision.labelDetector() labelDetector.detect(in: visionImage) {
(labels, error) in guard let error == nill, let labels = labels, !labels.isEmpty else { return } // do something with your labels }
#FirebaseDevDay On-cloud Usage let labelDetector = vision.cloudLabelDetector() labelDetector.detect(in: visionImage) {
(labels, error) in guard let error == nill, let labels = labels, !labels.isEmpty else { return } // do something with your labels }
Face detection - On-device
#FirebaseDevDay
#FirebaseDevDay FIRVisionFace | VisionFace NS_SWIFT_NAME(VisionFace) @interface FIRVisionFace : NSObject @property(nonatomic,
readonly) CGRect frame; @property(nonatomic, readonly) BOOL hasTrackingID; @property(nonatomic, readonly) NSInteger trackingID; @property(nonatomic, readonly) BOOL hasHeadEulerAngleY; @property(nonatomic, readonly) CGFloat headEulerAngleY; @property(nonatomic, readonly) BOOL hasHeadEulerAngleZ; @property(nonatomic, readonly) CGFloat headEulerAngleZ; @property(nonatomic, readonly) BOOL hasSmilingProbability; @property(nonatomic, readonly) CGFloat smilingProbability; @property(nonatomic, readonly) BOOL hasLeftEyeOpenProbability; @property(nonatomic, readonly) CGFloat leftEyeOpenProbability; @property(nonatomic, readonly) BOOL hasRightEyeOpenProbability; @property(nonatomic, readonly) CGFloat rightEyeOpenProbability; - (instancetype)init NS_UNAVAILABLE; - (nullable FIRVisionFaceLandmark *)landmarkOfType:(FIRFaceLandmarkType)type; #ifdef ENABLE_FACE_CONTOUR - (nullable FIRVisionFaceContour *)contourOfType:(FIRFaceContourType)type; #endif // ENABLE_FACE_CONTOUR @end
#FirebaseDevDay On-device Usage let faceDetector = vision.faceDetector() faceDetector.detect(in: visionImage) {
(faces, error) in guard let error == nill, let faces = faces, !faces.isEmpty else { return } // do something with your faces }
#FirebaseDevDay Face Contour?
Landmark recognition - On-cloud
#FirebaseDevDay
#FirebaseDevDay FIRVisionCloudLandmark | VisionCloudLandmark NS_SWIFT_NAME(VisionCloudLandmark) @interface FIRVisionCloudLandmark : NSObject @property(nonatomic,
copy, readonly, nullable) NSString *entityId; @property(nonatomic, copy, readonly, nullable) NSString *landmark; @property(nonatomic, readonly, nullable) NSNumber *confidence; @property(nonatomic, readonly) CGRect frame; @property(nonatomic, readonly, nullable) NSArray<FIRVisionLatitudeLongitude *> *locations; - (instancetype)init NS_UNAVAILABLE; @end
#FirebaseDevDay On-cloud Usage let landmarkDetector = vision.cloudLandmarkDetector() landmarkDetector.detect(in: visionImage) {
(landmarks, error) in guard let error == nill, let landmarks = landmarks, !landmarks.isEmpty else { return } // do something with your landmarks }
Barcode scanning - On-device
#FirebaseDevDay https://firebase.google.com/docs/ml-kit/label-images
#FirebaseDevDay FIRVisionBarcode | VisionBarcode NS_SWIFT_NAME(VisionBarcode) @interface FIRVisionBarcode : NSObject @property(nonatomic,
readonly) CGRect frame; @property(nonatomic, readonly, nullable) NSString *rawValue; @property(nonatomic, readonly, nullable) NSString *displayValue; @property(nonatomic, readonly) FIRVisionBarcodeFormat format; @property(nonatomic, readonly, nullable) NSArray<NSValue *> *cornerPoints; @property(nonatomic, readonly) FIRVisionBarcodeValueType valueType; @property(nonatomic, readonly, nullable) FIRVisionBarcodeEmail *email; @property(nonatomic, readonly, nullable) FIRVisionBarcodePhone *phone; @property(nonatomic, readonly, nullable) FIRVisionBarcodeSMS *sms; @property(nonatomic, readonly, nullable) FIRVisionBarcodeURLBookmark *URL; @property(nonatomic, readonly, nullable) FIRVisionBarcodeWiFi *wifi; @property(nonatomic, readonly, nullable) FIRVisionBarcodeGeoPoint *geoPoint; @property(nonatomic, readonly, nullable) FIRVisionBarcodeContactInfo *contactInfo; @property(nonatomic, readonly, nullable) FIRVisionBarcodeCalendarEvent *calendarEvent; @property(nonatomic, readonly, nullable) FIRVisionBarcodeDriverLicense *driverLicense; - (instancetype)init NS_UNAVAILABLE; @end
#FirebaseDevDay FIRVisionBarcodeCalendarEvent | VisionBarcodeCalendarEvent NS_SWIFT_NAME(VisionBarcodeCalendarEvent) @interface FIRVisionBarcodeCalendarEvent : NSObject @property(nonatomic,
readonly, nullable) NSString *eventDescription; @property(nonatomic, readonly, nullable) NSString *location; @property(nonatomic, readonly, nullable) NSString *organizer; @property(nonatomic, readonly, nullable) NSString *status; @property(nonatomic, readonly, nullable) NSString *summary; @property(nonatomic, readonly, nullable) NSDate *start; @property(nonatomic, readonly, nullable) NSDate *end; - (instancetype)init NS_UNAVAILABLE; @end
#FirebaseDevDay On-device Usage let barcodeDetector = vision.barcodeDetector() barcodeDetector.detect(in: visionImage) {
(barcodes, error) in guard let error == nill, let barcodes = barcodes, !barcodes.isEmpty else { return } // do something with your barcodes }
Custom model - Tensorflow Lite
#FirebaseDevDay let conditions = ModelDownloadConditions(isWiFiRequired: true, canDownloadInBackground: true) let cloudModelSource
= CloudModelSource( modelName: "my_cloud_model", enableModelUpdates: true, initialConditions: conditions, updateConditions: conditions ) let registrationSuccessful = ModelManager.modelManager().register(cloudModelSource)
Demo
Thank You! #FirebaseDevDay Helpful resources fb.com/FirebaseThailand fb.com/groups/FirebaseDevTH medium.com/FirebaseThailand Kajornsak Peerapathananont