Sharing experience of adopting machine learning to LINE app

Hyukjae Jang
LINE Plus App Dev 9 Software engineer
https://linedevday.linecorp.com/2020/ja/sessions/5123
https://linedevday.linecorp.com/2020/en/sessions/5123

Transcript

2. Agenda › Machine learning on LINE app › Approaches to adopt machine learning › An example of the on-device machine learning › Video highlight wizard › Other examples › CardOCR and Auto pin chat › Lesson learned
3. Machine learning in LINE app
4. Approaches to adopt machine learning Traditional machine learning or Deep learning Real time or Ready-made Server or On-device
5. Machine learning in LINE app Sticker recommendation • Deep learning + Traditional ML • On-device • Real-time OCR • Deep learning • Server • Real-time Video highlight wizard • Traditional ML • On-device • Ready-made
6. What is the goal? › Supporting capabilities › Expected accuracy Performance of machine learning model Target user experience › Required immediate responses › New feature or improvement of existing features Difficulty in training › Privacy issue › Size of training data Case by case No general solution
7. On-device machine learning › Mobile clients ßà Server › Mobile app’s point of view › Android / iOS
8. Examples of the on-device machine learning Brief overview of each example cases
9. Video highlight wizard Video editing is bothersome › User created videos › Might be shaky or messy › TL;DR (Too long, didn’t read) › à TL;DW (Too long, didn’t watch) › 15sec limit of ‘LINE Timeline story’
10. Video highlight wizard Generate video summary and highlights from video analysis Original Summary Highlight
11. Video highlight wizard Port HECATE to Android › Cross compile to build a shared library › Select only necessary shared libraries (libopencv_core, highgui, imgproc, ml, objdetect) OpenCV build from source codes Substitute FFmpeg with Android’s media library › HECATE uses FFmpeg for video frame extraction › FFmpeg has a patent issue with H.264 HECATE › Open source video processing library (https://github.com/yahoo/hecate) › Implemented with C++, dependency to OpenCV and FFmpeg
12. Video highlight wizard Video analysis based on eliminating low quality video frames Original Summary Highlight Filtering
13. Video highlight wizard Video analysis details 3 1 2 Original Filtering Shots Extraction Summary Highlight • Find max 1st order derivation between frames • Cluster with color and edge histogram of frames for finding sub-shots • Filter low quality and transition frames • Cluster visually similar shots • Pick most dynamic shots in each cluster • Adjust length of each shots • Pick the most representative shot A B
14. Video highlight wizard requirements Traditional ML, On-device, Ready-made Privacy issue High computing resource requirement Background analysis without degrading user experience
15. Video highlight wizard development summary › Too complicated UI for multiple shots › 15sec limit of ‘Timeline story’ Challenge & Opportunity Actual results › Released at Early 2020 as a Labs feature › Highlight wizard with trim range recommendation › Improve existing `trim a video` feature Initial motivation › Make a summary video with several shots Highlight trim range is suggested automatically When highlight is available
16. Examples of the on-device machine learning Brief overview of each example cases
17. Auto pin chat (work in progress) Pain points at chat list › Heavy users have more difficulty › Naïve chat sort policies Not easy to find a chat room › Focusing on message writings › Notification mitigates issues on readings Automatically pin/unpin chatrooms › " # = %&',) %*+*,-,) × %&',) %*+*,-,,-- ×/0123 › 4ℎ262 789,3 0: ;ℎ<= ;>?@= >A B0@2 0@ ;ℎ<=6>>B # › Message weight reduction over time Calculate activeness of chat rooms
18. Auto pin chat requirements Traditional ML, On-device, Ready-made Privacy issue Rapid response for frequent requests Incremental processing (maintain intermediate results)
19. Auto pin chat development summary › Conflicted with ‘Chat folders’ › Simplify user experience Challenge & Opportunity Actual results › Improve ‘Pin chat’ by automating ’pin/unpin a chatroom’ › Rejected Initial motivation › Sort whole chat list with various features › Contextual score using time and location, and activeness score
20. CardOCR Limitation of 3rd party library 4182 3087 5964 Google Mobile Vision 4000 1234 5678 9010
21. CardOCR requirements Traditional ML + Deep learning, On-device, Real time Instant feedback Standalone feature without server cooperation Increase of LINE app size as small as possible
22. CardOCR development summary › No offline card on ‘LINE My card’ feature › Urgent call from Taiwan Challenge & Opportunity Actual results › Improvement of credit card recognition › Released on Late 2017 Initial motivation › General card recognition including QR, barcode and numeric characters › ‘LINE My card’ feature
23. Common requirements for on-device ML › Rapid response › Error resilience › Privacy issue
24. Lesson learned › Developing and applying machine learning are different › Mediator between technology and user experience Check a roadmap › Keep track a goal of the related features › Intuit and target user requirements A means to an ends
25. Lesson learned › Minimal changes to user experience Resilient user experiences › Workaround of the feature › Recommendation Simple and easy approach: improvement Platform supports for machine learning › Google ML kit / Apple Core ML (pre-trained model) › Android WorkManager / iOS BGTaskScheduler (background task support)
26. Thank you