Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Sharing experience of adopting machine learning...

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →

Sharing experience of adopting machine learning to LINE app

LINE DevDay 2020

November 26, 2020

Transcript

26 slides
  1. Agenda › Machine learning on LINE app › Approaches to

    adopt machine learning › An example of the on-device machine learning › Video highlight wizard › Other examples › CardOCR and Auto pin chat › Lesson learned
  2. Approaches to adopt machine learning Traditional machine learning or Deep

    learning Real time or Ready-made Server or On-device
  3. Machine learning in LINE app Sticker recommendation • Deep learning

    + Traditional ML • On-device • Real-time OCR • Deep learning • Server • Real-time Video highlight wizard • Traditional ML • On-device • Ready-made
  4. What is the goal? › Supporting capabilities › Expected accuracy

    Performance of machine learning model Target user experience › Required immediate responses › New feature or improvement of existing features Difficulty in training › Privacy issue › Size of training data Case by case No general solution
  5. Video highlight wizard Video editing is bothersome › User created

    videos › Might be shaky or messy › TL;DR (Too long, didn’t read) › à TL;DW (Too long, didn’t watch) › 15sec limit of ‘LINE Timeline story’
  6. Video highlight wizard Port HECATE to Android › Cross compile

    to build a shared library › Select only necessary shared libraries (libopencv_core, highgui, imgproc, ml, objdetect) OpenCV build from source codes Substitute FFmpeg with Android’s media library › HECATE uses FFmpeg for video frame extraction › FFmpeg has a patent issue with H.264 HECATE › Open source video processing library (https://github.com/yahoo/hecate) › Implemented with C++, dependency to OpenCV and FFmpeg
  7. Video highlight wizard Video analysis based on eliminating low quality

    video frames Original Summary Highlight Filtering
  8. Video highlight wizard Video analysis details 3 1 2 Original

    Filtering Shots Extraction Summary Highlight • Find max 1st order derivation between frames • Cluster with color and edge histogram of frames for finding sub-shots • Filter low quality and transition frames • Cluster visually similar shots • Pick most dynamic shots in each cluster • Adjust length of each shots • Pick the most representative shot A B
  9. Video highlight wizard requirements Traditional ML, On-device, Ready-made Privacy issue

    High computing resource requirement Background analysis without degrading user experience
  10. Video highlight wizard development summary › Too complicated UI for

    multiple shots › 15sec limit of ‘Timeline story’ Challenge & Opportunity Actual results › Released at Early 2020 as a Labs feature › Highlight wizard with trim range recommendation › Improve existing `trim a video` feature Initial motivation › Make a summary video with several shots Highlight trim range is suggested automatically When highlight is available
  11. Auto pin chat (work in progress) Pain points at chat

    list › Heavy users have more difficulty › Naïve chat sort policies Not easy to find a chat room › Focusing on message writings › Notification mitigates issues on readings Automatically pin/unpin chatrooms › " # = %&',) %*+*,-,) × %&',) %*+*,-,,-- ×/0123 › 4ℎ262 789,3 0: ;ℎ<= ;>?@= >A B0@2 0@ ;ℎ<=6>>B # › Message weight reduction over time Calculate activeness of chat rooms
  12. Auto pin chat requirements Traditional ML, On-device, Ready-made Privacy issue

    Rapid response for frequent requests Incremental processing (maintain intermediate results)
  13. Auto pin chat development summary › Conflicted with ‘Chat folders’

    › Simplify user experience Challenge & Opportunity Actual results › Improve ‘Pin chat’ by automating ’pin/unpin a chatroom’ › Rejected Initial motivation › Sort whole chat list with various features › Contextual score using time and location, and activeness score
  14. CardOCR requirements Traditional ML + Deep learning, On-device, Real time

    Instant feedback Standalone feature without server cooperation Increase of LINE app size as small as possible
  15. CardOCR development summary › No offline card on ‘LINE My

    card’ feature › Urgent call from Taiwan Challenge & Opportunity Actual results › Improvement of credit card recognition › Released on Late 2017 Initial motivation › General card recognition including QR, barcode and numeric characters › ‘LINE My card’ feature
  16. Lesson learned › Developing and applying machine learning are different

    › Mediator between technology and user experience Check a roadmap › Keep track a goal of the related features › Intuit and target user requirements A means to an ends
  17. Lesson learned › Minimal changes to user experience Resilient user

    experiences › Workaround of the feature › Recommendation Simple and easy approach: improvement Platform supports for machine learning › Google ML kit / Apple Core ML (pre-trained model) › Android WorkManager / iOS BGTaskScheduler (background task support)

More Decks by LINE DevDay 2020

Other Decks in Technology