The Art of Smart Channel

The Art of Smart Channel

Naoki Watanabe
LINE AD Dev Team1 Manager
https://linedevday.linecorp.com/jp/2019/sessions/B1-2

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay The Art of Smart Channel > Naoki Watanabe

    > LINE AD Dev Team1 Manager
  2. Smart Channel

  3. Variation

  4. Global

  5. Concept Contents Personalize

  6. User Experience Three Steps To Complete Purchase

  7. Stats Impression / Day 500M Request / Second 250k Global

    DAU 100M+
  8. Architecture

  9. Flow CRS Engine Event Tracker Joiner Learning worker Importer Imp/Click

    Log Parameter Log Contents LINE APP Contents
  10. 2-Layered Personalization Online Contextual Bandits Recommendation Today’s Tomorrow’s Zodiac Notification

    Recommendation Notification Recommendation News Weather Fortune Sticker Manga Layer #2 Layer #1
  11. Layer #1 CRS Engine Event Tracker Joiner Learning worker Importer

    Contents Imp/Click Log Parameter Log Contents LINE APP
  12. How To Integrate With Service Service Side Mapping Provider Data

    Provider Redis Cluster HDFS User1, News_A User2, News_B Smart Channel Importer Get News Title, URL, Image
  13. 2-Layer Personalization(Again) Online Contextual Bandits Recommendation Today’s Tomorrow’s Zodiac Notification

    Recommendation Notification Recommendation News Weather Fortune Sticker Manga Layer #2 Layer #1
  14. Layer #2 CRS Engine Event Tracker Joiner Learning worker Importer

    Contents Imp/Click Log Parameter Log Contents LINE APP
  15. Which One Is the Best? Less Knowledge 35-39 Man Frequently

    Updated ?
  16. Bandit Algorithms

  17. Bandit Algorithms > Rewards is user happiness > Combination of

    exploration and exploitation > Algorithms to maximize the rewards (= earnings)
  18. Exploration & Exploitation Exploitation Exploration Time Data Collection/ Sample Apply

    Learning https://conversionxl.com/blog/bandit-tests/
  19. Bandit Selection https://conductrics.com/balancing-earning-with-learning-bandits-and-adaptive-optimization/ Content B Content C Content A

  20. Supported Algorithms LinUCB Bayesian Factorization Machines Bayesian Liner Regression Bayesian

    Neural Factorization Machines
  21. How To Evaluate Contents Click Do Nothing Mute

  22. How To Evaluate Contents CRS Engine Event Tracker Joiner Learning

    worker Importer Contents Imp/Click Log Parameter Log Contents LINE APP
  23. Reaction Timeline Impression 10min Evaluate point Mute Do Nothing Click

    Click
  24. Delayed Consumer Imp/Click Joined Kafka Redis Joiner EventTracker Delayed Imp

    Kafka Click/Mute Kafka Imp Kafka Imp Consumer Click/Mute Consumer 10min later Imp Click/Mute Timer
  25. Learning CRS Engine Event Tracker Joiner Learning worker Importer Contents

    Imp/Click Log Parameter Log Contents LINE APP
  26. 2-Sided Features User side > Age > Gender Content side

    > Time > Id > Type > etc..
  27. Bandit Arms: News Contents 20-24 25-29 30-34 35-39 40-44 M

    F M F M F M F M F 4af3e1d 0.57 0.52 0.59 0.57 0.64 0.65 0.68 0.64 0.76 0.73 2ulf3a4 0.73 0.38 0.67 0.39 0.61 0.40 0.58 0.40 0.51 0.41 1a0ew3 0.42 0.71 0.46 0.66 0.48 0.61 0.54 0.54 0.58 0.50 w2ag67 0.51 0.61 0.53 0.67 0.51 0.68 0.56 0.69 0.57 0.70 xr02h1 0.65 0.55 0.61 0.58 0.63 0.59 0.59 0.60 0.58 0.60 age x gender x id
  28. Bandit Arms: Weather Contents 20-24 25-29 30-34 35-39 40-44 M

    F M F M F M F M F Weather 0.48 0.56 0.47 0.58 0.51 0.57 0.52 0.58 0.55 0.59 0.53 0.51 0.51 0.52 0.55 0.53 0.55 0.57 0.54 0.54 0.61 0.62 0.62 0.63 0.62 0.66 0.62 0.64 0.62 0.66 0.68 0.67 0.65 0.67 0.67 0.67 0.66 0.67 0.68 0.67 0.72 0.73 0.71 0.72 0.71 0.75 0.74 0.72 0.73 0.74 age x gender x type x hour
  29. Bandit Arms: Weather Contents 20-24 25-29 30-34 35-39 40-44 M

    F M F M F M F M F Weather 0.48 0.56 0.47 0.58 0.51 0.57 0.52 0.58 0.55 0.59 0.53 0.51 0.51 0.52 0.55 0.53 0.55 0.57 0.54 0.54 0.61 0.62 0.62 0.63 0.62 0.66 0.62 0.64 0.62 0.66 0.68 0.67 0.65 0.67 0.67 0.67 0.66 0.67 0.68 0.67 0.72 0.73 0.71 0.72 0.71 0.75 0.74 0.72 0.73 0.74 age x gender x type x hour
  30. Bandit Arms Contents 20-24 25-29 30-34 35-39 40-44 M F

    M F M F M F M F 0.52 0.53 0.51 0.54 0.52 0.54 0.53 0.54 0.53 0.54 0.53 0.52 0.55 0.51 0.56 0.54 0.57 0.53 0.58 0.54 0.58 0.59 0.57 0.58 0.55 0.54 0.54 0.54 0.51 0.52 0.58 0.56 0.57 0.56 0.55 0.54 0.55 0.54 0.52 0.53 0.58 0.57 0.58 0.56 0.52 0.54 0.51 0.52 0.52 0.51
  31. Ideas for Learning Huge Traffic Stability Learning Efficiency Distributed Architecture

  32. For Huge Traffic CRS Engine Imp/Click Joined Kafka Model Parameter

    Redis Ranker Learning Dispatcher
  33. Load Balancing > Store data for each arm Arms >

    Split every K into Partition P=1 P=2 P=3 > Always use the same instance for arm and partition id pairs > Consistent hash by Proxy(NGINX) 1 2 3 4 5 6 7 Arm=3 Partition=3 Arm=2 Partition=1
  34. For Huge Traffic CRS Engine Imp/Click Joined Kafka Learning Worker

    Worker Proxy Parameter Proxy Parameter Server Model Parameter Redis Trainer Ranker Learning Dispatcher
  35. Hetero-Architecture CRS Engine Imp/Click Joined Kafka Learning Worker Worker Proxy

    Parameter Proxy Parameter Server Model Parameter Redis Learning Dispatcher Realtime Async
  36. Hetero-Architecture CRS Engine Imp/Click Joined Kafka Learning Worker Worker Proxy

    Parameter Proxy Parameter Server Model Parameter Redis Learning Dispatcher Java Python
  37. Flow CRS Engine Event Tracker Joiner Learning worker Importer Contents

    Imp/Click Log Parameter Log Contents LINE APP
  38. Operation & Improvement

  39. Test Problem > Test with limited users in the Production

    environment > Verify the performance of recommendation > Hard to test recommendation on test environment
  40. Gradual Rollout > A/B test support tool called Libra >

    Exposure is limited to n% users > ID can also be specified
  41. Health Check of Service Mute Click Click / Mute =

    Score
  42. A/B Test > A/B test also uses Libra > Easy

    to configure
  43. A/B Analysis > A/B result is analyzed by Data scientist

  44. Disadvantages of Bandit > Bandit arms are only age x

    gender. > Should optimize by user basis > It takes a certain time(10min~) to learn Content B Content C Content a Personalize??
  45. 2-Sided Features+ User side > Age > Gender Contents side

    > Time > Id > Score > etc.. > Past CTR > Past xCTR
  46. User Feature User CTR xCTR CTR xCTR CTR xCTR CTR

    xCTR CTR xCTR UserA 0.57 0.47 0.87 0.37 0.47 0.57 0.67 0.47 0.37 0.47 UserB 0.53 0.37 0.47 0.57 0.67 0.47 0.51 0.52 0.67 0.37 UserC 0.51 0.52 - - 0.57 0.52 0.57 0.53 - - UserD 0.52 0.51 0.57 0.53 0.54 0.53 - - 0.56 0.53 UserE 0.55 0.54 0.57 0.51 0.53 0.51 0.68 0.34 0.53 0.47
  47. Personalize by User User A User B User C User

    D User E
  48. Future

  49. Life Information

  50. Life Information

  51. Bandit API

  52. Bandit API Flow Bandit API Other Service Event Tracker Learning

    Worker UserID, IDs A,B,C,D UserID, Ranked IDs C,D,B,A Imp/Click Log Parameter
  53. Recommend 2 Other Services

  54. Recommend 2 Other Service CRS Talk Tab Home Tab Wallet

    Tab Sticker News Manga
  55. Location & Beacon

  56. Location & Beacon >Limited content in specific places > Make

    LINE Beacon experience mach more fun > Show local weather by user location
  57. > Day2: C2-1 12:00-12:40 > LINE-Like Product Development Using Smart

    Channel as an Example "LINE-Like" Product Management > Poster Session 13:40-14:20/15:30-16:10 (2days) > Case Studies on Smart Channel Platforms and How To Improve Content > Day1: C1-7 18:10-18:50 > Explanation of SmartChannel From the Perspective of ML / DA Engineer Building a Smart Recommender System Across LINE Services Continuous Improvements in Smart Channel Platform/Contents Related Presentation
  58. Thank You