The Art of Smart Channel

Naoki Watanabe
LINE AD Dev Team1 Manager
https://linedevday.linecorp.com/jp/2019/sessions/B1-2

Transcript

1. 2019 DevDay The Art of Smart Channel > Naoki Watanabe > LINE AD Dev Team1 Manager
2. Smart Channel
3. Variation
4. Global
5. Concept Contents Personalize
6. User Experience Three Steps To Complete Purchase
7. Stats Impression / Day 500M Request / Second 250k Global DAU 100M+
8. Architecture
9. Flow CRS Engine Event Tracker Joiner Learning worker Importer Imp/Click Log Parameter Log Contents LINE APP Contents
10. 2-Layered Personalization Online Contextual Bandits Recommendation Today’s Tomorrow’s Zodiac Notification Recommendation Notification Recommendation News Weather Fortune Sticker Manga Layer #2 Layer #1
11. Layer #1 CRS Engine Event Tracker Joiner Learning worker Importer Contents Imp/Click Log Parameter Log Contents LINE APP
12. How To Integrate With Service Service Side Mapping Provider Data Provider Redis Cluster HDFS User1, News_A User2, News_B Smart Channel Importer Get News Title, URL, Image
13. 2-Layer Personalization(Again) Online Contextual Bandits Recommendation Today’s Tomorrow’s Zodiac Notification Recommendation Notification Recommendation News Weather Fortune Sticker Manga Layer #2 Layer #1
14. Layer #2 CRS Engine Event Tracker Joiner Learning worker Importer Contents Imp/Click Log Parameter Log Contents LINE APP
15. Which One Is the Best? Less Knowledge 35-39 Man Frequently Updated ?
16. Bandit Algorithms
17. Bandit Algorithms > Rewards is user happiness > Combination of exploration and exploitation > Algorithms to maximize the rewards (= earnings)
18. Exploration & Exploitation Exploitation Exploration Time Data Collection/ Sample Apply Learning https://conversionxl.com/blog/bandit-tests/
19. Bandit Selection https://conductrics.com/balancing-earning-with-learning-bandits-and-adaptive-optimization/ Content B Content C Content A
20. Supported Algorithms LinUCB Bayesian Factorization Machines Bayesian Liner Regression Bayesian Neural Factorization Machines
21. How To Evaluate Contents Click Do Nothing Mute
22. How To Evaluate Contents CRS Engine Event Tracker Joiner Learning worker Importer Contents Imp/Click Log Parameter Log Contents LINE APP
23. Reaction Timeline Impression 10min Evaluate point Mute Do Nothing Click Click
24. Delayed Consumer Imp/Click Joined Kafka Redis Joiner EventTracker Delayed Imp Kafka Click/Mute Kafka Imp Kafka Imp Consumer Click/Mute Consumer 10min later Imp Click/Mute Timer
25. Learning CRS Engine Event Tracker Joiner Learning worker Importer Contents Imp/Click Log Parameter Log Contents LINE APP
26. 2-Sided Features User side > Age > Gender Content side > Time > Id > Type > etc..
27. Bandit Arms: News Contents 20-24 25-29 30-34 35-39 40-44 M F M F M F M F M F 4af3e1d 0.57 0.52 0.59 0.57 0.64 0.65 0.68 0.64 0.76 0.73 2ulf3a4 0.73 0.38 0.67 0.39 0.61 0.40 0.58 0.40 0.51 0.41 1a0ew3 0.42 0.71 0.46 0.66 0.48 0.61 0.54 0.54 0.58 0.50 w2ag67 0.51 0.61 0.53 0.67 0.51 0.68 0.56 0.69 0.57 0.70 xr02h1 0.65 0.55 0.61 0.58 0.63 0.59 0.59 0.60 0.58 0.60 age x gender x id
28. Bandit Arms: Weather Contents 20-24 25-29 30-34 35-39 40-44 M F M F M F M F M F Weather 0.48 0.56 0.47 0.58 0.51 0.57 0.52 0.58 0.55 0.59 0.53 0.51 0.51 0.52 0.55 0.53 0.55 0.57 0.54 0.54 0.61 0.62 0.62 0.63 0.62 0.66 0.62 0.64 0.62 0.66 0.68 0.67 0.65 0.67 0.67 0.67 0.66 0.67 0.68 0.67 0.72 0.73 0.71 0.72 0.71 0.75 0.74 0.72 0.73 0.74 age x gender x type x hour
29. Bandit Arms: Weather Contents 20-24 25-29 30-34 35-39 40-44 M F M F M F M F M F Weather 0.48 0.56 0.47 0.58 0.51 0.57 0.52 0.58 0.55 0.59 0.53 0.51 0.51 0.52 0.55 0.53 0.55 0.57 0.54 0.54 0.61 0.62 0.62 0.63 0.62 0.66 0.62 0.64 0.62 0.66 0.68 0.67 0.65 0.67 0.67 0.67 0.66 0.67 0.68 0.67 0.72 0.73 0.71 0.72 0.71 0.75 0.74 0.72 0.73 0.74 age x gender x type x hour
30. Bandit Arms Contents 20-24 25-29 30-34 35-39 40-44 M F M F M F M F M F 0.52 0.53 0.51 0.54 0.52 0.54 0.53 0.54 0.53 0.54 0.53 0.52 0.55 0.51 0.56 0.54 0.57 0.53 0.58 0.54 0.58 0.59 0.57 0.58 0.55 0.54 0.54 0.54 0.51 0.52 0.58 0.56 0.57 0.56 0.55 0.54 0.55 0.54 0.52 0.53 0.58 0.57 0.58 0.56 0.52 0.54 0.51 0.52 0.52 0.51
31. Ideas for Learning Huge Traffic Stability Learning Efficiency Distributed Architecture
32. For Huge Traffic CRS Engine Imp/Click Joined Kafka Model Parameter Redis Ranker Learning Dispatcher
33. Load Balancing > Store data for each arm Arms > Split every K into Partition P=1 P=2 P=3 > Always use the same instance for arm and partition id pairs > Consistent hash by Proxy(NGINX) 1 2 3 4 5 6 7 Arm=3 Partition=3 Arm=2 Partition=1
34. For Huge Traffic CRS Engine Imp/Click Joined Kafka Learning Worker Worker Proxy Parameter Proxy Parameter Server Model Parameter Redis Trainer Ranker Learning Dispatcher
35. Hetero-Architecture CRS Engine Imp/Click Joined Kafka Learning Worker Worker Proxy Parameter Proxy Parameter Server Model Parameter Redis Learning Dispatcher Realtime Async
36. Hetero-Architecture CRS Engine Imp/Click Joined Kafka Learning Worker Worker Proxy Parameter Proxy Parameter Server Model Parameter Redis Learning Dispatcher Java Python
37. Flow CRS Engine Event Tracker Joiner Learning worker Importer Contents Imp/Click Log Parameter Log Contents LINE APP
38. Operation & Improvement
39. Test Problem > Test with limited users in the Production environment > Verify the performance of recommendation > Hard to test recommendation on test environment
40. Gradual Rollout > A/B test support tool called Libra > Exposure is limited to n% users > ID can also be specified
41. Health Check of Service Mute Click Click / Mute = Score
42. A/B Test > A/B test also uses Libra > Easy to configure
43. A/B Analysis > A/B result is analyzed by Data scientist
44. Disadvantages of Bandit > Bandit arms are only age x gender. > Should optimize by user basis > It takes a certain time(10min~) to learn Content B Content C Content a Personalize??
45. 2-Sided Features+ User side > Age > Gender Contents side > Time > Id > Score > etc.. > Past CTR > Past xCTR
46. User Feature User CTR xCTR CTR xCTR CTR xCTR CTR xCTR CTR xCTR UserA 0.57 0.47 0.87 0.37 0.47 0.57 0.67 0.47 0.37 0.47 UserB 0.53 0.37 0.47 0.57 0.67 0.47 0.51 0.52 0.67 0.37 UserC 0.51 0.52 - - 0.57 0.52 0.57 0.53 - - UserD 0.52 0.51 0.57 0.53 0.54 0.53 - - 0.56 0.53 UserE 0.55 0.54 0.57 0.51 0.53 0.51 0.68 0.34 0.53 0.47
47. Personalize by User User A User B User C User D User E
48. Future
49. Life Information
50. Life Information
51. Bandit API
52. Bandit API Flow Bandit API Other Service Event Tracker Learning Worker UserID, IDs A,B,C,D UserID, Ranked IDs C,D,B,A Imp/Click Log Parameter
53. Recommend 2 Other Services
54. Recommend 2 Other Service CRS Talk Tab Home Tab Wallet Tab Sticker News Manga
55. Location & Beacon
56. Location & Beacon >Limited content in specific places > Make LINE Beacon experience mach more fun > Show local weather by user location
57. > Day2: C2-1 12:00-12:40 > LINE-Like Product Development Using Smart Channel as an Example "LINE-Like" Product Management > Poster Session 13:40-14:20/15:30-16:10 (2days) > Case Studies on Smart Channel Platforms and How To Improve Content > Day1: C1-7 18:10-18:50 > Explanation of SmartChannel From the Perspective of ML / DA Engineer Building a Smart Recommender System Across LINE Services Continuous Improvements in Smart Channel Platform/Contents Related Presentation
58. Thank You