Share
Embed
Sharing experience of adopting machine learning to LINE app
Hyukjae Jang
LINE Plus App Dev 9 Software engineer
https://linedevday.linecorp.com/2020/ja/sessions/5123
https://linedevday.linecorp.com/2020/en/sessions/5123
LINE DevDay 2020
November 26, 2020
Transcript
26 slides
-
Agenda › Machine learning on LINE app › Approaches to
adopt machine learning › An example of the on-device machine learning › Video highlight wizard › Other examples › CardOCR and Auto pin chat › Lesson learned -
Approaches to adopt machine learning Traditional machine learning or Deep
learning Real time or Ready-made Server or On-device -
Machine learning in LINE app Sticker recommendation • Deep learning
+ Traditional ML • On-device • Real-time OCR • Deep learning • Server • Real-time Video highlight wizard • Traditional ML • On-device • Ready-made -
What is the goal? › Supporting capabilities › Expected accuracy
Performance of machine learning model Target user experience › Required immediate responses › New feature or improvement of existing features Difficulty in training › Privacy issue › Size of training data Case by case No general solution -
On-device machine learning › Mobile clients ßà Server › Mobile
app’s point of view › Android / iOS -
Video highlight wizard Video editing is bothersome › User created
videos › Might be shaky or messy › TL;DR (Too long, didn’t read) › à TL;DW (Too long, didn’t watch) › 15sec limit of ‘LINE Timeline story’ -
Video highlight wizard Generate video summary and highlights from video
analysis Original Summary Highlight -
Video highlight wizard Port HECATE to Android › Cross compile
to build a shared library › Select only necessary shared libraries (libopencv_core, highgui, imgproc, ml, objdetect) OpenCV build from source codes Substitute FFmpeg with Android’s media library › HECATE uses FFmpeg for video frame extraction › FFmpeg has a patent issue with H.264 HECATE › Open source video processing library (https://github.com/yahoo/hecate) › Implemented with C++, dependency to OpenCV and FFmpeg -
Video highlight wizard Video analysis based on eliminating low quality
video frames Original Summary Highlight Filtering -
Video highlight wizard Video analysis details 3 1 2 Original
Filtering Shots Extraction Summary Highlight • Find max 1st order derivation between frames • Cluster with color and edge histogram of frames for finding sub-shots • Filter low quality and transition frames • Cluster visually similar shots • Pick most dynamic shots in each cluster • Adjust length of each shots • Pick the most representative shot A B -
Video highlight wizard requirements Traditional ML, On-device, Ready-made Privacy issue
High computing resource requirement Background analysis without degrading user experience -
Video highlight wizard development summary › Too complicated UI for
multiple shots › 15sec limit of ‘Timeline story’ Challenge & Opportunity Actual results › Released at Early 2020 as a Labs feature › Highlight wizard with trim range recommendation › Improve existing `trim a video` feature Initial motivation › Make a summary video with several shots Highlight trim range is suggested automatically When highlight is available -
Auto pin chat (work in progress) Pain points at chat
list › Heavy users have more difficulty › Naïve chat sort policies Not easy to find a chat room › Focusing on message writings › Notification mitigates issues on readings Automatically pin/unpin chatrooms › " # = %&',) %*+*,-,) × %&',) %*+*,-,,-- ×/0123 › 4ℎ262 789,3 0: ;ℎ<= ;>?@= >A B0@2 0@ ;ℎ<=6>>B # › Message weight reduction over time Calculate activeness of chat rooms -
Auto pin chat requirements Traditional ML, On-device, Ready-made Privacy issue
Rapid response for frequent requests Incremental processing (maintain intermediate results) -
Auto pin chat development summary › Conflicted with ‘Chat folders’
› Simplify user experience Challenge & Opportunity Actual results › Improve ‘Pin chat’ by automating ’pin/unpin a chatroom’ › Rejected Initial motivation › Sort whole chat list with various features › Contextual score using time and location, and activeness score -
CardOCR Limitation of 3rd party library 4182 3087 5964 Google
Mobile Vision 4000 1234 5678 9010 -
CardOCR requirements Traditional ML + Deep learning, On-device, Real time
Instant feedback Standalone feature without server cooperation Increase of LINE app size as small as possible -
CardOCR development summary › No offline card on ‘LINE My
card’ feature › Urgent call from Taiwan Challenge & Opportunity Actual results › Improvement of credit card recognition › Released on Late 2017 Initial motivation › General card recognition including QR, barcode and numeric characters › ‘LINE My card’ feature -
Common requirements for on-device ML › Rapid response › Error
resilience › Privacy issue -
Lesson learned › Developing and applying machine learning are different
› Mediator between technology and user experience Check a roadmap › Keep track a goal of the related features › Intuit and target user requirements A means to an ends -
Lesson learned › Minimal changes to user experience Resilient user
experiences › Workaround of the feature › Recommendation Simple and easy approach: improvement Platform supports for machine learning › Google ML kit / Apple Core ML (pre-trained model) › Android WorkManager / iOS BGTaskScheduler (background task support)