WWDC19 Recap of ML

WWDC19 Recap of ML

Event: https://mercaridev.connpass.com/event/132676/
Some demos and images are removed for NDA.

79abd7eb5642d2b3c8dcf9bcc6f581f3?s=128

kagemiku

June 13, 2019
Tweet

Transcript

  1. WWDC19 RECAP OF ML @kagemiku (Akira Fukunaga) / Recap of

    WWDC19 at Mercari
  2. ▸ kagemiku (Akira Fukunaga) ▸ GitHub: @kagemiku ▸ Twitter: @kagemiku_en

    ▸ iOS Engineer (19’s new grad) at Mercari JP ▸ First time participation in WWDC!!! 
 (and also, first time application) ABOUT ME
  3. None
  4. ML??

  5. SESSIONS ▸ 209: What’s New in Machine Learning ▸ 704:

    Core ML 3 Framework ▸ 406: Create ML for Object Detection and Sound Classification ▸ 222: Understanding Images in Vision Framework ▸ 228: Creating Great Apps Using Core ML and ARKit ▸ 407: Create ML for Activity, Text, and Recommendations ▸ 232: Advances in Natural Language Framework ▸ 234: Text Recognition in Vision Framework ▸ 420: Drawing Classification and One-Shot Object Detection in Turi Create ▸ 803: Designing Great ML Experiences ▸ 614: Metal for Machine Learning
  6. WHAT’S NEW IN ML ▸ Create ML ▸ Domain APIs

    ▸ Core ML
  7. WHAT’S NEW IN ML ▸ Create ML ▸ Domain APIs

    ▸ Core ML
  8. CREATE ML WHAT’S CREATE ML? ▸ Framework for creating ML

    model with Swift, appeared in Xcode 10 let data = try! MLDataTable(contentsOf: URL(fileURLWithPath: "/path/to/dataset.json")) let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5) let sentimentClassifier = try! MLTextClassifier(trainingData: trainingData, textColumn: "text", labelColumn: "label")
  9. CREATE ML ▸ In Xcode10, we can create ML model

    in Playground GUI
  10. CREATE ML NEW APPLICATION ▸ Now, the feature has been

    cut off as an independent GUI app
  11. CREATE ML ▸ 9 templates were described in session ▸

    Image Classifier ▸ Sound Classifier ▸ Activity Classifier ▸ Tabular Classifier ▸ and so on… ▸ But now, there are only 2 templates
 at the first seed
  12. CREATE ML ▸ 9 templates were described in session ▸

    Image Classifier ▸ Sound Classifier ▸ Activity Classifier ▸ Tabular Classifier ▸ and so on… ▸ But now, there are only 2 templates
 at the first seed
  13. CREATE ML ▸ Dataset notes ▸ Balanced quantity ▸ :

    10 s, 100 s, 1000 s ▸ : 100 s, 100 s, 100 s ▸ At least 10 data for each label ▸ At least 299 × 299 px
  14. CREATE ML DEMO ▸ (I cannot record the screen in

    macOS Catalina beta….)
  15. WHAT’S NEW IN ML ▸ Create ML ▸ Domain APIs

    ▸ Core ML
  16. DOMAIN APIS WHAT’S DOMAIN API ▸ Useful ML models prepared

    by Apple ▸ We don’t have to collect data and build model ▸ Main frameworks ▸ Computer Vision (Vision Framework) ▸ Natural Language Processing (NaturalLanguage Framework)
  17. DOMAIN APIS NEW DOMAIN APIS ▸ Many new apis appear

    ▸ Image Saliency ▸ Image Similarity ▸ Sentiment Analysis ▸ Text Recognition ▸ NL Transfer Learning ▸ and so on…
  18. DOMAIN APIS NEW DOMAIN APIS ▸ Many new apis appear

    ▸ Image Saliency ▸ Image Similarity ▸ Sentiment Analysis ▸ Text Recognition ▸ NL Transfer Learning ▸ and so on…
  19. DOMAIN APIS IMAGE SALIENCY ▸ Saliency: the salient points or

    features of something are 
 the most important or most noticeable parts of it ▸ There are 2 types of saliency ▸ Attention based ▸ Objectness based
  20. DOMAIN APIS IMAGE SALIENCY ▸ Attention based ▸ Training data:

    Human eye movement ▸ App example: image cropping ▸ Objectness based ▸ Training data: distinguished foreground object from background ▸ App example: object tracking
  21. DOMAIN APIS DEMO

  22. DOMAIN APIS ▸ code // 1. prepare request and handler

    let request: VNRequest = VNGenerateAttentionBasedSaliencyImageRequest() let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:]) // 2. perform requests and get results try? requestHandler.perform([request]) let observation = request.results?.first as? VNSaliencyImageObservation // 3. do something using results if let salientObjects = observation?.salientObjects { for object in salientObjects { let boundingBox = object.boundingBox // do something } } IMAGE SALIENCY
  23. DOMAIN APIS NEW DOMAIN APIS ▸ Many new apis appear

    ▸ Image Saliency ▸ Image Similarity ▸ Sentiment Analysis ▸ Text Recognition ▸ NL Transfer Learning ▸ and so on…
  24. DOMAIN APIS SENTIMENT ANALYSIS ▸ Analyze sentiment of text, positive

    or negative
  25. DOMAIN APIS SENTIMENT ANALYSIS ▸ App example: ▸ Change review

    text color based on analysis
  26. DOMAIN APIS DEMO

  27. DOMAIN APIS ▸ code ▸ Support 7 languages at the

    moment ▸ English/French/Italian/German/Spanish/Portuguese/Simplified Chinese ▸ Japanese is not be included now // 1. Prepare NLTagger with .sentimentScore scheme let tagger = NLTagger(tagSchemes: [.sentimentScore]) // 2. Set text you want to analyze tagger.string = text // 3. Get result let (sentiment, _) = tagger.tag(at: text.startIndex, unit: .paragraph, scheme: .sentimentScore) print(sentiment!.rawValue) SENTIMENT ANALYSIS
  28. WHAT’S NEW IN ML ▸ Create ML ▸ Domain APIs

    ▸ Core ML
  29. CORE ML WHAT’S CORE ML ▸ Multi platform framework for

    ML ▸ Optimized for on-device performance ▸ low memory footprint ▸ low power consumption ▸ Protect security and privacy
  30. CORE ML WHAT’S CORE ML ▸ Support many models ▸

    Generalized Linear Model ▸ SVM ▸ CNN/RNN ▸ Tree Ensembles ▸ and so on…
  31. CORE ML CORE ML 3 ▸ Model Flexibility ▸ Model

    Personalization
  32. CORE ML MODEL FLEXIBILITY ▸ Support 100+ NN layers

  33. CORE ML MODEL FLEXIBILITY ▸ Model Galleries 
 You can

    start immediately using these models
  34. CORE ML MODEL PERSONALIZATION ▸ You can fine-tune created models

    on device
  35. CORE ML MODEL PERSONALIZATION ▸ Supporting models ▸ NN ▸

    Nearest Neighbor ▸ Fine-tune can be done in background process
  36. CORE ML MODEL PERSONALIZATION ▸ Application example: ▸ User can

    train the existing model with user hand-written for drawing sticker automatically train
  37. CORE ML MODEL PERSONALIZATION ▸ Code

  38. SUMMARY ▸ Create ML ▸ A brand new app ▸

    Domain APIs ▸ Significant expansion ▸ Core ML3 ▸ More flexible ▸ On-device personalization
  39. REFERENCES ▸ Create ML - Apple ▸ Core ML -

    Apple