Technologies that support the distribution of LINE NEWS articles

Daiki Inaba
LINE Development Team I Software engineer
https://linedevday.linecorp.com/jp/2019/sessions/C1-3

Transcript

1. 2019 DevDay Technologies That Support the Distribution of LINE NEWS Articles > Daiki Inaba > LINE Development Team I Software engineer
2. Agenda > Overview of LINE NEWS > Personalized Recommendation Architecture • How To Process User Request • How To Import Recommendations
3. Overview of LINE NEWS
4. Service History Launch App & Web 2013
5. Launch News Tab 2017 Launch App & Web 2013 Service History
6. Now Launch News Tab 2017 Launch App & Web 2013 Service History
7. News Sports What Contents We Provide Train Info J-Alert / J-Alert Earthquake
8. How Many Users MAU 68M MPV 12B
9. How Much Traffic Req / Day 17.3B Req / Sec - Avg 200K Req / Sec - Peak 375K
10. 00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00
11. Application Server Infrastructure Spring Boot (Java) × 30 Amon2 (Perl) × 44
12. DB Server Infrastructure MongoDB × 21 MySQL × 3 Cache Server Redis Sentinel × 3 Memcached × 44 Redis Cluster × 1
13. Realtime Article RSS/ SFTP/ SOAP Import Personalized Recommendation Original Body Format Deliver Message
14. Realtime Article RSS/ SFTP/ SOAP Import Personalized Recommendation Original Body Format Deliver Message
15. Personalized Recommendation Architecture
16. Recommendations
17. Machine Learning × Manual
18. Machine Learning × Manual
19. > Generated by Machine Learning Team called as Data Labs > Provided as files • Contain recommendations for 100 million users in all • Each recommendations contain 200 article • Updated hourly Machine Learning Recommendation
20. > Generated by Machine Learning Team called as Data Labs > Provided as files • Contain recommendations for 100 million users in all • Each recommendations contain 200 article • Updated hourly Machine Learning Recommendation
21. > Set by operators • Targets are specified by many attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation
22. > Set by operators • Targets are specified by many attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation
23. Recommendations 1. Machine Learning 2. Machine Learning + Manual
24. Machine Learning
25. Not Personalized CDN Origin Request Same Response for Every User
26. ? Personalized Request With User Identifier Personalized Response
27. Web Redis (Recommendation) Redis (Article Cache) MySQL
28. Web Redis (Recommendation) Redis (Article Cache) MySQL Import
29. Web Redis (Recommendation) Redis (Article Cache) MySQL Import
30. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations
31. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Get Cache(MGET)
32. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Get Cache(MGET)
33. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Get Cache(MGET) Cache
34. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Get Cache(MGET) Cache
37. Request Process > Actually, processing personalized request is not so difficult > Issues are in the process of importing recommendations
38. Import Issue Need to import recommendations for over 100 million users hourly
39. API Data Labs Batch Redis Datahub ML Importer × 3
40. API Data Labs Batch Redis Datahub ML Importer × 3 Upload
41. API Data Labs Batch Redis Datahub ML Importer × 3 Upload API Call
42. API Data Labs Batch Redis Datahub ML Importer × 3 Upload API Call Distribute
43. API ML Importer 1 ML Importer 2 ML Importer 3 Datahub
44. API Datahub LIMIT : 100 OFFSET : 0 LIMIT : 100 OFFSET : 100 LIMIT : 100 OFFSET : 200 ML Importer 1 ML Importer 2 ML Importer 3
45. API Datahub LIMIT : 100 OFFSET : 0 LIMIT : 100 OFFSET : 100 LIMIT : 100 OFFSET : 200 Import ML Importer 1 ML Importer 2 ML Importer 3
46. Datahub Redis Download File Parse file data Store on Redis Importer
47. ${user_id}\t${article_id},${article_id}\t${score},${score} ${user_id}\t${article_id},${article_id}\t${score},${score} ${user_id}\t${article_id},${article_id}\t${score},${score} …
48. Datahub Redis Download File Parse file data Store on Redis Importer
49. ${user_id}\t${article_id},${article_id}\t${score},${score} { "user_id": ${user_id} "articles": [ { "id": ${article_id}, "score": ${score} } … ] }
50. Datahub Redis Download File Parse file data Store on Redis Importer
51. Datahub Redis Download File Parse file data Store on Redis Importer Async/Parallel × 20
52. Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Search Cache(MGET) Cache
53. API Data Labs Batch Redis Datahub Importer × 3 Upload API Call Distribute Import Store
54. ML + Manual
55. > Set by operators • Targets are specified by many attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation
56. 1st Approach > Import target user ids as CSV that extracted by operator in advance
57. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL Operator
58. Central Dogma Redis Web Article Data
59. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL Operator
60. What Is Central Dogma? > Highly-available version-controlled service configuration repository > Store your configuration files such as .json, .yaml and .xml into a centralized > Let your servers get notified immediately when the configuration files are updated and the new settings are applied without server restarts
61. Datalake Batch Operator CMS Data Analyst Central Dogma Redis MySQL User Ids
62. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Operator
63. Datalake Batch Operator CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Store Mapping Data
64. article_id_1 SET(user_id, user_id, user_id,…) article_id_2 SET(user_id, user_id, user_id,…) … …
65. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Store Mapping Data Operator
66. Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Store Mapping Data Upload Article Data Operator
67. [ { "title": "ถΞοϓϧ͸৽αʔϏεͳͲ…", "image_url": "https://scdn.line…", "position": 10 … }, … ]
68. Central Dogma Redis Sync Web Article Data
69. Central Dogma Redis Sync Web Article Data
70. Central Dogma Redis Sync Web Article Data Get Article Ids
71. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …)
72. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …) SISMEMBER (article_id_n, user_id_1)
73. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …) SISMEMBER (article_id_n, user_id_1)
74. Central Dogma Redis Sync Web Article Data Get Article Ids
75. Issues >Less scalability due to mapping data structure >Too high operating cost
76. Issues >Less scalability due to mapping data structure >Too high operating cost
77. 2nd Approach > Less scalability due mapping data structure • Change mapping data structure > Too high operating cost • Enable to extract target user ids via CMS
78. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer
79. Central Dogma Redis Web Article Data
80. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer
81. Central Dogma Redis Web Article Data
82. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input
83. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input
84. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input
85. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids
86. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids Store Mapping Data
87. article_id SET(user_id_1, user_id_2, …) user_id_1 SET(article_id, article_id, article_id, …) user_id_2 SET(article_id, article_id, article_id, …) … …
88. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids Store Mapping Data
89. Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids Store Mapping Data
90. Central Dogma Redis Web Article Data Sync
91. Central Dogma Redis Web Article Data Sync Get Article Ids
92. Central Dogma Redis Web Article Data Sync
93. article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …) SISMEMBER (article_id_n, user_id_1)
94. user_id_1 SET(article_id, article_id, article_id,…) user_id_2 SET(article_id, article_id, article_id,…) user_id_3 SET(article_id, article_id, article_id,…) GET(user_id_2)
95. Price of 2nd Approach >Difficult to generate and delete mapping data
96. article_id_5 for user_id_3 user_id_4 user_id_5
97. article_id_5 for user_id_3 user_id_4 user_id_5 article_id_1 SET(user_id, user_id_, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id, user_id, user_id, …) article_id_4 SET(user_id, user_id, user_id, …)
98. article_id_5 for user_id_3 user_id_4 user_id_5 article_id_1 SET(user_id, user_id_, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id, user_id, user_id, …) article_id_4 SET(user_id, user_id, user_id, …) article_id_5 SET(user_id_1, user_id_2, user_id_3)
99. article_id_5 for user_id_3 user_id_4 user_id_5 user_id_1 SET(article_id, article_id) user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id) user_id_4 SET(article_id, article_id)
100. article_id_5 for user_id_3 user_id_4 user_id_5 user_id_5 SET(article_id_5) user_id_1 SET(article_id, article_id) user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id) user_id_4 SET(article_id, article_id)
101. article_id_5 for user_id_3 user_id_4 user_id_5 user_id_1 SET(article_id, article_id) user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id, article_id_5) user_id_4 SET(article_id, article_id, article_id_5) user_id_5 SET(article_id_5)
102. Future Plans
103. Future Plans >Improve recommendation quality
104. Improve Quality Future Plans User Recommendation Response
105. Improve Quality Future Plans User Recommendation Any Time
106. Improve Quality Future Plans User Recommendation 1 Hour Later Any Time
107. Improve Quality Future Plans User Recommendation Realtime Any Time
108. Future Plans >Apply recommendation to other part >Improve recommendation quality
109. Recommendation In Digest
110. Recommendation In Digest
111. Conclusion > Focused on delivering recommendation • Machine Learning & Manual • Request & Import Process > But we are delivering many other kinds of contents • So…
112. Please Use LINE NEWS :)
113. Thank You