Technologies that support the distribution of LINE NEWS articles

Technologies that support the distribution of LINE NEWS articles

Daiki Inaba
LINE Development Team I Software engineer
https://linedevday.linecorp.com/jp/2019/sessions/C1-3

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay Technologies That Support the Distribution of LINE NEWS

    Articles > Daiki Inaba > LINE Development Team I Software engineer
  2. Agenda > Overview of LINE NEWS > Personalized Recommendation Architecture

    • How To Process User Request • How To Import Recommendations
  3. Overview of LINE NEWS

  4. Service History Launch App & Web 2013

  5. Launch News Tab 2017 Launch App & Web 2013 Service

    History
  6. Now Launch News Tab 2017 Launch App & Web 2013

    Service History
  7. News Sports What Contents We Provide Train Info J-Alert /

    J-Alert Earthquake
  8. How Many Users MAU 68M MPV 12B

  9. How Much Traffic Req / Day 17.3B Req / Sec

    - Avg 200K Req / Sec - Peak 375K
  10. 00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00

    20:00 22:00
  11. Application Server Infrastructure Spring Boot (Java) × 30 Amon2 (Perl)

    × 44
  12. DB Server Infrastructure MongoDB × 21 MySQL × 3 Cache

    Server Redis Sentinel × 3 Memcached × 44 Redis Cluster × 1
  13. Realtime Article RSS/ SFTP/ SOAP Import Personalized Recommendation Original Body

    Format Deliver Message
  14. Realtime Article RSS/ SFTP/ SOAP Import Personalized Recommendation Original Body

    Format Deliver Message
  15. Personalized Recommendation Architecture

  16. Recommendations

  17. Machine Learning × Manual

  18. Machine Learning × Manual

  19. > Generated by Machine Learning Team called as Data Labs

    > Provided as files • Contain recommendations for 100 million users in all • Each recommendations contain 200 article • Updated hourly Machine Learning Recommendation
  20. > Generated by Machine Learning Team called as Data Labs

    > Provided as files • Contain recommendations for 100 million users in all • Each recommendations contain 200 article • Updated hourly Machine Learning Recommendation
  21. > Set by operators • Targets are specified by many

    attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation
  22. > Set by operators • Targets are specified by many

    attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation
  23. Recommendations 1. Machine Learning 2. Machine Learning + Manual

  24. Machine Learning

  25. Not Personalized CDN Origin Request Same Response for Every User

  26. ? Personalized Request With User Identifier Personalized Response

  27. Web Redis (Recommendation) Redis (Article Cache) MySQL

  28. Web Redis (Recommendation) Redis (Article Cache) MySQL Import

  29. Web Redis (Recommendation) Redis (Article Cache) MySQL Import

  30. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

  31. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

    Get Cache(MGET)
  32. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

    Search MySQL Get Cache(MGET)
  33. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

    Search MySQL Get Cache(MGET) Cache
  34. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

    Search MySQL Get Cache(MGET) Cache
  35. None
  36. None
  37. Request Process > Actually, processing personalized request is not so

    difficult > Issues are in the process of importing recommendations
  38. Import Issue Need to import recommendations for over 100 million

    users hourly
  39. API Data Labs Batch Redis Datahub ML Importer × 3

  40. API Data Labs Batch Redis Datahub ML Importer × 3

    Upload
  41. API Data Labs Batch Redis Datahub ML Importer × 3

    Upload API Call
  42. API Data Labs Batch Redis Datahub ML Importer × 3

    Upload API Call Distribute
  43. API ML Importer 1 ML Importer 2 ML Importer 3

    Datahub
  44. API Datahub LIMIT : 100 OFFSET : 0 LIMIT :

    100 OFFSET : 100 LIMIT : 100 OFFSET : 200 ML Importer 1 ML Importer 2 ML Importer 3
  45. API Datahub LIMIT : 100 OFFSET : 0 LIMIT :

    100 OFFSET : 100 LIMIT : 100 OFFSET : 200 Import ML Importer 1 ML Importer 2 ML Importer 3
  46. Datahub Redis Download File Parse file data Store on Redis

    Importer
  47. ${user_id}\t${article_id},${article_id}\t${score},${score} ${user_id}\t${article_id},${article_id}\t${score},${score} ${user_id}\t${article_id},${article_id}\t${score},${score} …

  48. Datahub Redis Download File Parse file data Store on Redis

    Importer
  49. ${user_id}\t${article_id},${article_id}\t${score},${score} { "user_id": ${user_id} "articles": [ { "id": ${article_id}, "score":

    ${score} } … ] }
  50. Datahub Redis Download File Parse file data Store on Redis

    Importer
  51. Datahub Redis Download File Parse file data Store on Redis

    Importer Async/Parallel × 20
  52. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

    Search MySQL Search Cache(MGET) Cache
  53. API Data Labs Batch Redis Datahub Importer × 3 Upload

    API Call Distribute Import Store
  54. ML + Manual

  55. > Set by operators • Targets are specified by many

    attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation
  56. 1st Approach > Import target user ids as CSV that

    extracted by operator in advance
  57. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL Operator

  58. Central Dogma Redis Web Article Data

  59. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL Operator

  60. What Is Central Dogma? > Highly-available version-controlled service configuration repository

    > Store your configuration files such as .json, .yaml and .xml into a centralized > Let your servers get notified immediately when the configuration files are updated and the new settings are applied without server restarts
  61. Datalake Batch Operator CMS Data Analyst Central Dogma Redis MySQL

    User Ids
  62. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User

    Ids CSV Operator
  63. Datalake Batch Operator CMS Data Analyst Central Dogma Redis MySQL

    User Ids CSV Store Mapping Data
  64. article_id_1 SET(user_id, user_id, user_id,…) article_id_2 SET(user_id, user_id, user_id,…) … …

  65. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User

    Ids CSV Store Mapping Data Operator
  66. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User

    Ids CSV Store Mapping Data Upload Article Data Operator
  67. [ { "title": "ถΞοϓϧ͸৽αʔϏεͳͲ…", "image_url": "https://scdn.line…", "position": 10 … },

    … ]
  68. Central Dogma Redis Sync Web Article Data

  69. Central Dogma Redis Sync Web Article Data

  70. Central Dogma Redis Sync Web Article Data Get Article Ids

  71. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …)

    article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …)
  72. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …)

    article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …) SISMEMBER (article_id_n, user_id_1)
  73. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …)

    article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …) SISMEMBER (article_id_n, user_id_1)
  74. Central Dogma Redis Sync Web Article Data Get Article Ids

  75. Issues >Less scalability due to mapping data structure >Too high

    operating cost
  76. Issues >Less scalability due to mapping data structure >Too high

    operating cost
  77. 2nd Approach > Less scalability due mapping data structure •

    Change mapping data structure > Too high operating cost • Enable to extract target user ids via CMS
  78. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

  79. Central Dogma Redis Web Article Data

  80. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

  81. Central Dogma Redis Web Article Data

  82. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input
  83. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input
  84. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input
  85. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input User Ids
  86. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input User Ids Store Mapping Data
  87. article_id SET(user_id_1, user_id_2, …) user_id_1 SET(article_id, article_id, article_id, …) user_id_2

    SET(article_id, article_id, article_id, …) … …
  88. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input User Ids Store Mapping Data
  89. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

    Input User Ids Store Mapping Data
  90. Central Dogma Redis Web Article Data Sync

  91. Central Dogma Redis Web Article Data Sync Get Article Ids

  92. Central Dogma Redis Web Article Data Sync

  93. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …)

    article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …) SISMEMBER (article_id_n, user_id_1)
  94. user_id_1 SET(article_id, article_id, article_id,…) user_id_2 SET(article_id, article_id, article_id,…) user_id_3 SET(article_id,

    article_id, article_id,…) GET(user_id_2)
  95. Price of 2nd Approach >Difficult to generate and delete mapping

    data
  96. article_id_5 for user_id_3 user_id_4 user_id_5

  97. article_id_5 for user_id_3 user_id_4 user_id_5 article_id_1 SET(user_id, user_id_, user_id, …)

    article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id, user_id, user_id, …) article_id_4 SET(user_id, user_id, user_id, …)
  98. article_id_5 for user_id_3 user_id_4 user_id_5 article_id_1 SET(user_id, user_id_, user_id, …)

    article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id, user_id, user_id, …) article_id_4 SET(user_id, user_id, user_id, …) article_id_5 SET(user_id_1, user_id_2, user_id_3)
  99. article_id_5 for user_id_3 user_id_4 user_id_5 user_id_1 SET(article_id, article_id) user_id_2 SET(article_id,

    article_id) user_id_3 SET(article_id, article_id) user_id_4 SET(article_id, article_id)
  100. article_id_5 for user_id_3 user_id_4 user_id_5 user_id_5 SET(article_id_5) user_id_1 SET(article_id, article_id)

    user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id) user_id_4 SET(article_id, article_id)
  101. article_id_5 for user_id_3 user_id_4 user_id_5 user_id_1 SET(article_id, article_id) user_id_2 SET(article_id,

    article_id) user_id_3 SET(article_id, article_id, article_id_5) user_id_4 SET(article_id, article_id, article_id_5) user_id_5 SET(article_id_5)
  102. Future Plans

  103. Future Plans >Improve recommendation quality

  104. Improve Quality Future Plans User Recommendation Response

  105. Improve Quality Future Plans User Recommendation Any Time

  106. Improve Quality Future Plans User Recommendation 1 Hour Later Any

    Time
  107. Improve Quality Future Plans User Recommendation Realtime Any Time

  108. Future Plans >Apply recommendation to other part >Improve recommendation quality

  109. Recommendation In Digest

  110. Recommendation In Digest

  111. Conclusion > Focused on delivering recommendation • Machine Learning &

    Manual • Request & Import Process > But we are delivering many other kinds of contents • So…
  112. Please Use LINE NEWS :)

  113. Thank You