Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WWDC19 Recap of ML

WWDC19 Recap of ML

Event: https://mercaridev.connpass.com/event/132676/
Some demos and images are removed for NDA.

kagemiku

June 13, 2019
Tweet

More Decks by kagemiku

Other Decks in Technology

Transcript

  1. WWDC19 RECAP OF ML
    @kagemiku (Akira Fukunaga) / Recap of WWDC19 at Mercari

    View Slide

  2. ▸ kagemiku (Akira Fukunaga)
    ▸ GitHub: @kagemiku
    ▸ Twitter: @kagemiku_en
    ▸ iOS Engineer (19’s new grad) at Mercari JP
    ▸ First time participation in WWDC!!!

    (and also, first time application)
    ABOUT ME

    View Slide

  3. View Slide

  4. ML??

    View Slide

  5. SESSIONS
    ▸ 209: What’s New in Machine Learning
    ▸ 704: Core ML 3 Framework
    ▸ 406: Create ML for Object Detection and Sound Classification
    ▸ 222: Understanding Images in Vision Framework
    ▸ 228: Creating Great Apps Using Core ML and ARKit
    ▸ 407: Create ML for Activity, Text, and Recommendations
    ▸ 232: Advances in Natural Language Framework
    ▸ 234: Text Recognition in Vision Framework
    ▸ 420: Drawing Classification and One-Shot Object Detection in Turi Create
    ▸ 803: Designing Great ML Experiences
    ▸ 614: Metal for Machine Learning

    View Slide

  6. WHAT’S NEW IN ML
    ▸ Create ML
    ▸ Domain APIs
    ▸ Core ML

    View Slide

  7. WHAT’S NEW IN ML
    ▸ Create ML
    ▸ Domain APIs
    ▸ Core ML

    View Slide

  8. CREATE ML
    WHAT’S CREATE ML?
    ▸ Framework for creating ML model with Swift, appeared in Xcode 10
    let data = try! MLDataTable(contentsOf: URL(fileURLWithPath: "/path/to/dataset.json"))
    let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5)
    let sentimentClassifier = try! MLTextClassifier(trainingData: trainingData,
    textColumn: "text",
    labelColumn: "label")

    View Slide

  9. CREATE ML
    ▸ In Xcode10, we can create ML model in Playground GUI

    View Slide

  10. CREATE ML
    NEW APPLICATION
    ▸ Now, the feature has been cut off as an independent GUI app

    View Slide

  11. CREATE ML
    ▸ 9 templates were described in session
    ▸ Image Classifier
    ▸ Sound Classifier
    ▸ Activity Classifier
    ▸ Tabular Classifier
    ▸ and so on…
    ▸ But now, there are only 2 templates

    at the first seed

    View Slide

  12. CREATE ML
    ▸ 9 templates were described in session
    ▸ Image Classifier
    ▸ Sound Classifier
    ▸ Activity Classifier
    ▸ Tabular Classifier
    ▸ and so on…
    ▸ But now, there are only 2 templates

    at the first seed

    View Slide

  13. CREATE ML
    ▸ Dataset notes
    ▸ Balanced quantity
    ▸ : 10
    s, 100
    s, 1000
    s
    ▸ : 100
    s, 100
    s, 100
    s
    ▸ At least 10 data for each label
    ▸ At least 299 × 299 px

    View Slide

  14. CREATE ML
    DEMO
    ▸ (I cannot record the screen in macOS Catalina beta….)

    View Slide

  15. WHAT’S NEW IN ML
    ▸ Create ML
    ▸ Domain APIs
    ▸ Core ML

    View Slide

  16. DOMAIN APIS
    WHAT’S DOMAIN API
    ▸ Useful ML models prepared by Apple
    ▸ We don’t have to collect data and build model
    ▸ Main frameworks
    ▸ Computer Vision (Vision Framework)
    ▸ Natural Language Processing (NaturalLanguage Framework)

    View Slide

  17. DOMAIN APIS
    NEW DOMAIN APIS
    ▸ Many new apis appear
    ▸ Image Saliency
    ▸ Image Similarity
    ▸ Sentiment Analysis
    ▸ Text Recognition
    ▸ NL Transfer Learning
    ▸ and so on…

    View Slide

  18. DOMAIN APIS
    NEW DOMAIN APIS
    ▸ Many new apis appear
    ▸ Image Saliency
    ▸ Image Similarity
    ▸ Sentiment Analysis
    ▸ Text Recognition
    ▸ NL Transfer Learning
    ▸ and so on…

    View Slide

  19. DOMAIN APIS
    IMAGE SALIENCY
    ▸ Saliency: the salient points or features of something are 

    the most important or most noticeable parts of it
    ▸ There are 2 types of saliency
    ▸ Attention based
    ▸ Objectness based

    View Slide

  20. DOMAIN APIS
    IMAGE SALIENCY
    ▸ Attention based
    ▸ Training data: Human eye movement
    ▸ App example: image cropping
    ▸ Objectness based
    ▸ Training data: distinguished foreground object from background
    ▸ App example: object tracking

    View Slide

  21. DOMAIN APIS
    DEMO

    View Slide

  22. DOMAIN APIS
    ▸ code
    // 1. prepare request and handler
    let request: VNRequest = VNGenerateAttentionBasedSaliencyImageRequest()
    let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: [:])
    // 2. perform requests and get results
    try? requestHandler.perform([request])
    let observation = request.results?.first as? VNSaliencyImageObservation
    // 3. do something using results
    if let salientObjects = observation?.salientObjects {
    for object in salientObjects {
    let boundingBox = object.boundingBox
    // do something
    }
    }
    IMAGE SALIENCY

    View Slide

  23. DOMAIN APIS
    NEW DOMAIN APIS
    ▸ Many new apis appear
    ▸ Image Saliency
    ▸ Image Similarity
    ▸ Sentiment Analysis
    ▸ Text Recognition
    ▸ NL Transfer Learning
    ▸ and so on…

    View Slide

  24. DOMAIN APIS
    SENTIMENT ANALYSIS
    ▸ Analyze sentiment of text, positive or negative

    View Slide

  25. DOMAIN APIS
    SENTIMENT ANALYSIS
    ▸ App example:
    ▸ Change review text color based on analysis

    View Slide

  26. DOMAIN APIS
    DEMO

    View Slide

  27. DOMAIN APIS
    ▸ code
    ▸ Support 7 languages at the moment
    ▸ English/French/Italian/German/Spanish/Portuguese/Simplified Chinese
    ▸ Japanese is not be included now
    // 1. Prepare NLTagger with .sentimentScore scheme
    let tagger = NLTagger(tagSchemes: [.sentimentScore])
    // 2. Set text you want to analyze
    tagger.string = text
    // 3. Get result
    let (sentiment, _) = tagger.tag(at: text.startIndex, unit: .paragraph, scheme: .sentimentScore)
    print(sentiment!.rawValue)
    SENTIMENT ANALYSIS

    View Slide

  28. WHAT’S NEW IN ML
    ▸ Create ML
    ▸ Domain APIs
    ▸ Core ML

    View Slide

  29. CORE ML
    WHAT’S CORE ML
    ▸ Multi platform framework for ML
    ▸ Optimized for on-device performance
    ▸ low memory footprint
    ▸ low power consumption
    ▸ Protect security and privacy

    View Slide

  30. CORE ML
    WHAT’S CORE ML
    ▸ Support many models
    ▸ Generalized Linear Model
    ▸ SVM
    ▸ CNN/RNN
    ▸ Tree Ensembles
    ▸ and so on…

    View Slide

  31. CORE ML
    CORE ML 3
    ▸ Model Flexibility
    ▸ Model Personalization

    View Slide

  32. CORE ML
    MODEL FLEXIBILITY
    ▸ Support 100+ NN layers

    View Slide

  33. CORE ML
    MODEL FLEXIBILITY
    ▸ Model Galleries 

    You can start immediately using these models

    View Slide

  34. CORE ML
    MODEL PERSONALIZATION
    ▸ You can fine-tune created models on device

    View Slide

  35. CORE ML
    MODEL PERSONALIZATION
    ▸ Supporting models
    ▸ NN
    ▸ Nearest Neighbor
    ▸ Fine-tune can be done in background process

    View Slide

  36. CORE ML
    MODEL PERSONALIZATION
    ▸ Application example:
    ▸ User can train the existing model with user hand-written for drawing sticker
    automatically
    train

    View Slide

  37. CORE ML
    MODEL PERSONALIZATION
    ▸ Code

    View Slide

  38. SUMMARY
    ▸ Create ML
    ▸ A brand new app
    ▸ Domain APIs
    ▸ Significant expansion
    ▸ Core ML3
    ▸ More flexible
    ▸ On-device personalization

    View Slide

  39. REFERENCES
    ▸ Create ML - Apple
    ▸ Core ML - Apple

    View Slide