Technologies that support the distribution of LINE NEWS articles

by LINE DevDay 2019

Slide 1

Slide 1 text

2019 DevDay Technologies That Support the Distribution of LINE NEWS Articles > Daiki Inaba > LINE Development Team I Software engineer

Slide 2

Slide 2 text

Agenda > Overview of LINE NEWS > Personalized Recommendation Architecture • How To Process User Request • How To Import Recommendations

Slide 3

Slide 3 text

Overview of LINE NEWS

Slide 4

Slide 4 text

Service History Launch App & Web 2013

Slide 5

Slide 5 text

Launch News Tab 2017 Launch App & Web 2013 Service History

Slide 6

Slide 6 text

Now Launch News Tab 2017 Launch App & Web 2013 Service History

Slide 7

Slide 7 text

News Sports What Contents We Provide Train Info J-Alert / J-Alert Earthquake

Slide 8

Slide 8 text

How Many Users MAU 68M MPV 12B

Slide 9

Slide 9 text

How Much Traffic Req / Day 17.3B Req / Sec - Avg 200K Req / Sec - Peak 375K

Slide 10

Slide 10 text

00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00

Slide 11

Slide 11 text

Application Server Infrastructure Spring Boot (Java) × 30 Amon2 (Perl) × 44

Slide 12

Slide 12 text

DB Server Infrastructure MongoDB × 21 MySQL × 3 Cache Server Redis Sentinel × 3 Memcached × 44 Redis Cluster × 1

Slide 13

Slide 13 text

Realtime Article RSS/ SFTP/ SOAP Import Personalized Recommendation Original Body Format Deliver Message

Slide 14

Slide 14 text

Realtime Article RSS/ SFTP/ SOAP Import Personalized Recommendation Original Body Format Deliver Message

Slide 15

Slide 15 text

Personalized Recommendation Architecture

Slide 16

Slide 16 text

Recommendations

Slide 17

Slide 17 text

Machine Learning × Manual

Slide 18

Slide 18 text

Machine Learning × Manual

Slide 19

Slide 19 text

> Generated by Machine Learning Team called as Data Labs > Provided as files • Contain recommendations for 100 million users in all • Each recommendations contain 200 article • Updated hourly Machine Learning Recommendation

Slide 20

Slide 20 text

Slide 21

Slide 21 text

> Set by operators • Targets are specified by many attributes • Age, Gender, Residence, and more… > Can be set in parallel > Reservable > Mixed in ML recommendations at specified position Manual Recommendation

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Recommendations 1. Machine Learning 2. Machine Learning + Manual

Slide 24

Slide 24 text

Machine Learning

Slide 25

Slide 25 text

Not Personalized CDN Origin Request Same Response for Every User

Slide 26

Slide 26 text

? Personalized Request With User Identifier Personalized Response

Slide 27

Slide 27 text

Web Redis (Recommendation) Redis (Article Cache) MySQL

Slide 28

Slide 28 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import

Slide 29

Slide 29 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import

Slide 30

Slide 30 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations

Slide 31

Slide 31 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Get Cache(MGET)

Slide 32

Slide 32 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Get Cache(MGET)

Slide 33

Slide 33 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Get Cache(MGET) Cache

Slide 34

Slide 34 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Get Cache(MGET) Cache

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Request Process > Actually, processing personalized request is not so difficult > Issues are in the process of importing recommendations

Slide 38

Slide 38 text

Import Issue Need to import recommendations for over 100 million users hourly

Slide 39

Slide 39 text

API Data Labs Batch Redis Datahub ML Importer × 3

Slide 40

Slide 40 text

API Data Labs Batch Redis Datahub ML Importer × 3 Upload

Slide 41

Slide 41 text

API Data Labs Batch Redis Datahub ML Importer × 3 Upload API Call

Slide 42

Slide 42 text

API Data Labs Batch Redis Datahub ML Importer × 3 Upload API Call Distribute

Slide 43

Slide 43 text

API ML Importer 1 ML Importer 2 ML Importer 3 Datahub

Slide 44

Slide 44 text

API Datahub LIMIT : 100 OFFSET : 0 LIMIT : 100 OFFSET : 100 LIMIT : 100 OFFSET : 200 ML Importer 1 ML Importer 2 ML Importer 3

Slide 45

Slide 45 text

API Datahub LIMIT : 100 OFFSET : 0 LIMIT : 100 OFFSET : 100 LIMIT : 100 OFFSET : 200 Import ML Importer 1 ML Importer 2 ML Importer 3

Slide 46

Slide 46 text

Datahub Redis Download File Parse file data Store on Redis Importer

Slide 47

Slide 47 text

${user_id}\t${article_id},${article_id}\t${score},${score} ${user_id}\t${article_id},${article_id}\t${score},${score} ${user_id}\t${article_id},${article_id}\t${score},${score} …

Slide 48

Slide 48 text

Datahub Redis Download File Parse file data Store on Redis Importer

Slide 49

Slide 49 text

${user_id}\t${article_id},${article_id}\t${score},${score} { "user_id": ${user_id} "articles": [ { "id": ${article_id}, "score": ${score} } … ] }

Slide 50

Slide 50 text

Datahub Redis Download File Parse file data Store on Redis Importer

Slide 51

Slide 51 text

Datahub Redis Download File Parse file data Store on Redis Importer Async/Parallel × 20

Slide 52

Slide 52 text

Web Redis (Recommendation) Redis (Article Cache) MySQL Import Fetch Recommendations Search MySQL Search Cache(MGET) Cache

Slide 53

Slide 53 text

API Data Labs Batch Redis Datahub Importer × 3 Upload API Call Distribute Import Store

Slide 54

Slide 54 text

ML + Manual

Slide 55

Slide 55 text

Slide 56

Slide 56 text

1st Approach > Import target user ids as CSV that extracted by operator in advance

Slide 57

Slide 57 text

Datalake Batch CMS Data Analyst Central Dogma Redis MySQL Operator

Slide 58

Slide 58 text

Central Dogma Redis Web Article Data

Slide 59

Slide 59 text

Datalake Batch CMS Data Analyst Central Dogma Redis MySQL Operator

Slide 60

Slide 60 text

What Is Central Dogma? > Highly-available version-controlled service configuration repository > Store your configuration files such as .json, .yaml and .xml into a centralized > Let your servers get notified immediately when the configuration files are updated and the new settings are applied without server restarts

Slide 61

Slide 61 text

Datalake Batch Operator CMS Data Analyst Central Dogma Redis MySQL User Ids

Slide 62

Slide 62 text

Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Operator

Slide 63

Slide 63 text

Datalake Batch Operator CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Store Mapping Data

Slide 64

Slide 64 text

article_id_1 SET(user_id, user_id, user_id,…) article_id_2 SET(user_id, user_id, user_id,…) … …

Slide 65

Slide 65 text

Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Store Mapping Data Operator

Slide 66

Slide 66 text

Datalake Batch CMS Data Analyst Central Dogma Redis MySQL User Ids CSV Store Mapping Data Upload Article Data Operator

Slide 67

Slide 67 text

[ { "title": "ถΞοϓϧ͸৽αʔϏεͳͲ…", "image_url": "https://scdn.line…", "position": 10 … }, … ]

Slide 68

Slide 68 text

Central Dogma Redis Sync Web Article Data

Slide 69

Slide 69 text

Central Dogma Redis Sync Web Article Data

Slide 70

Slide 70 text

Central Dogma Redis Sync Web Article Data Get Article Ids

Slide 71

Slide 71 text

article_id_1 SET(user_id, user_id_1, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id_1, user_id, user_id, …) article_id_4 SET(user_id_1, user_id, user_id, …) article_id_5 SET(user_id, user_id, user_id, …)

Slide 72

Slide 72 text

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Central Dogma Redis Sync Web Article Data Get Article Ids

Slide 75

Slide 75 text

Issues >Less scalability due to mapping data structure >Too high operating cost

Slide 76

Slide 76 text

Issues >Less scalability due to mapping data structure >Too high operating cost

Slide 77

Slide 77 text

2nd Approach > Less scalability due mapping data structure • Change mapping data structure > Too high operating cost • Enable to extract target user ids via CMS

Slide 78

Slide 78 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

Slide 79

Slide 79 text

Central Dogma Redis Web Article Data

Slide 80

Slide 80 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer

Slide 81

Slide 81 text

Central Dogma Redis Web Article Data

Slide 82

Slide 82 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input

Slide 83

Slide 83 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input

Slide 84

Slide 84 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input

Slide 85

Slide 85 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids

Slide 86

Slide 86 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids Store Mapping Data

Slide 87

Slide 87 text

article_id SET(user_id_1, user_id_2, …) user_id_1 SET(article_id, article_id, article_id, …) user_id_2 SET(article_id, article_id, article_id, …) … …

Slide 88

Slide 88 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids Store Mapping Data

Slide 89

Slide 89 text

Datalake Batch CMS Central Dogma Redis MySQL Operator Manual Importer Input User Ids Store Mapping Data

Slide 90

Slide 90 text

Central Dogma Redis Web Article Data Sync

Slide 91

Slide 91 text

Central Dogma Redis Web Article Data Sync Get Article Ids

Slide 92

Slide 92 text

Central Dogma Redis Web Article Data Sync

Slide 93

Slide 93 text

Slide 94

Slide 94 text

user_id_1 SET(article_id, article_id, article_id,…) user_id_2 SET(article_id, article_id, article_id,…) user_id_3 SET(article_id, article_id, article_id,…) GET(user_id_2)

Slide 95

Slide 95 text

Price of 2nd Approach >Difficult to generate and delete mapping data

Slide 96

Slide 96 text

article_id_5 for user_id_3 user_id_4 user_id_5

Slide 97

Slide 97 text

article_id_5 for user_id_3 user_id_4 user_id_5 article_id_1 SET(user_id, user_id_, user_id, …) article_id_2 SET(user_id, user_id, user_id, …) article_id_3 SET(user_id, user_id, user_id, …) article_id_4 SET(user_id, user_id, user_id, …)

Slide 98

Slide 98 text

Slide 99

Slide 99 text

article_id_5 for user_id_3 user_id_4 user_id_5 user_id_1 SET(article_id, article_id) user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id) user_id_4 SET(article_id, article_id)

Slide 100

Slide 100 text

article_id_5 for user_id_3 user_id_4 user_id_5 user_id_5 SET(article_id_5) user_id_1 SET(article_id, article_id) user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id) user_id_4 SET(article_id, article_id)

Slide 101

Slide 101 text

article_id_5 for user_id_3 user_id_4 user_id_5 user_id_1 SET(article_id, article_id) user_id_2 SET(article_id, article_id) user_id_3 SET(article_id, article_id, article_id_5) user_id_4 SET(article_id, article_id, article_id_5) user_id_5 SET(article_id_5)

Slide 102

Slide 102 text

Future Plans

Slide 103

Slide 103 text

Future Plans >Improve recommendation quality

Slide 104

Slide 104 text

Improve Quality Future Plans User Recommendation Response

Slide 105

Slide 105 text

Improve Quality Future Plans User Recommendation Any Time

Slide 106

Slide 106 text

Improve Quality Future Plans User Recommendation 1 Hour Later Any Time

Slide 107

Slide 107 text

Improve Quality Future Plans User Recommendation Realtime Any Time

Slide 108

Slide 108 text

Future Plans >Apply recommendation to other part >Improve recommendation quality

Slide 109

Slide 109 text

Recommendation In Digest

Slide 110

Slide 110 text

Recommendation In Digest

Slide 111

Slide 111 text

Conclusion > Focused on delivering recommendation • Machine Learning & Manual • Request & Import Process > But we are delivering many other kinds of contents • So…