Problems and Solutions for Two Billion Recommendations Per Day

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Agenda - About Smart Channel - Architecture - Troubles & Solutions - Lessons learned

Slide 3

Slide 3 text

Smart Channel

Slide 4

Slide 4 text

Smart Channel

Slide 5

Slide 5 text

Smart Channel

Slide 6

Slide 6 text

Service Concept

Slide 7

Slide 7 text

Contents Service Concept

Slide 8

Slide 8 text

Contents Service Concept

Slide 9

Slide 9 text

Contents Personalize Service Concept

Slide 10

Slide 10 text

Stats Request / day MAU Items / day

Slide 11

Slide 11 text

Stats Request / day 2 billion MAU Items / day

Slide 12

Slide 12 text

Stats Request / day 2 billion MAU 167 million Items / day

Slide 13

Slide 13 text

Stats Request / day 2 billion MAU 167 million Items / day 16 million

Slide 14

Slide 14 text

Architecture

Slide 15

Slide 15 text

Flow CRS Engine Event Tracker Learning worker Services LINE APP

Slide 16

Slide 16 text

Flow CRS Engine Event Tracker Learning worker Services Contents LINE APP

Slide 17

Slide 17 text

Flow CRS Engine Event Tracker Learning worker Request Services Contents LINE APP

Slide 18

Slide 18 text

Flow CRS Engine Event Tracker Learning worker Services Contents LINE APP

Slide 19

Slide 19 text

Flow CRS Engine Event Tracker Learning worker Services Contents LINE APP Contents

Slide 20

Slide 20 text

Flow CRS Engine Event Tracker Learning worker Imp/Click Services Contents LINE APP Contents

Slide 21

Slide 21 text

Flow CRS Engine Event Tracker Learning worker Imp/Click Log Services Contents LINE APP Contents

Slide 22

Slide 22 text

Flow CRS Engine Event Tracker Learning worker Imp/Click Log Services Parameter Contents LINE APP Contents

Slide 23

Slide 23 text

Key function

Slide 24

Slide 24 text

Gathers recommended contents from services Key function

Slide 25

Slide 25 text

Gathers recommended contents from services Select best ones of the many contents Key function

Slide 26

Slide 26 text

Gathers recommended contents from services Learning users preferences & use them to make the next selection Select best ones of the many contents Key function

Slide 27

Slide 27 text

https://linedevday.linecorp.com/jp/2019/sessions/B1-2

Slide 28

Slide 28 text

Peak Traffic

Slide 29

Slide 29 text

Peak Traffic - LINE’s peak traffic is New Year

Slide 30

Slide 30 text

Peak Traffic - LINE’s peak traffic is New Year - Especially JP (GTM+9)

Slide 31

Slide 31 text

Peak Traffic - LINE’s peak traffic is New Year - Especially JP (GTM+9) - 250k request / second

Slide 32

Slide 32 text

Peak Traffic - LINE’s peak traffic is New Year - Especially JP (GTM+9) - 250k request / second - Client side cache is effective

Slide 33

Slide 33 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 New Year Peek of 2018

Slide 34

Slide 34 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 60 148,000 New Year Peek of 2018

Slide 35

Slide 35 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 60 148,000 180 121,000 New Year Peek of 2018

Slide 36

Slide 36 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 60 148,000 180 121,000 300 107,000 New Year Peek of 2018

Slide 37

Slide 37 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 60 148,000 180 121,000 300 107,000 600 71,000 New Year Peek of 2018

Slide 38

Slide 38 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 60 148,000 180 121,000 300 107,000 600 71,000 1800 58,000 New Year Peek of 2018

Slide 39

Slide 39 text

Request with client-side cache TTL (seconds) Estimated RPS 0 250,000 60 148,000 180 121,000 300 107,000 600 71,000 1800 58,000 New Year Peek of 2018

Slide 40

Slide 40 text

Core system LINE App Services CRS Engine

Slide 41

Slide 41 text

Core system LINE App Services CRS Engine

Slide 42

Slide 42 text

Core system LINE App Services CRS Engine

Slide 43

Slide 43 text

Core system LINE App Services Importer CRS Engine

Slide 44

Slide 44 text

Core system LINE App Services Importer CRS Engine Redis Cluster

Slide 45

Slide 45 text

Data Modeling

Slide 46

Slide 46 text

Data Modeling

Slide 47

Slide 47 text

Data Modeling

Slide 48

Slide 48 text

Data Modeling

Slide 49

Slide 49 text

Data Modeling 167Million times

Slide 50

Slide 50 text

Data Modeling

Slide 51

Slide 51 text

Data Modeling Item-A ID Item-B ID Item-C ID Item-A ID Item-D ID

Slide 52

Slide 52 text

Data Modeling Item-A ID Item-B ID Item-C ID Item-A ID Item-D ID

Slide 53

Slide 53 text

Targeting Data Modeling Item-A ID Item-B ID Item-C ID Item-A ID Item-D ID

Slide 54

Slide 54 text

Information Targeting Data Modeling Item-A ID Item-B ID Item-C ID Item-A ID Item-D ID

Slide 55

Slide 55 text

Redis Cluster Redis Cluster Key: User-A ID Value: Item-A ID, Item-B ID, ….

Slide 56

Slide 56 text

Redis Cluster Redis Cluster Key: User-A ID Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, ….

Slide 57

Slide 57 text

Redis Cluster Redis Cluster Key: User-A ID Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, ….

Slide 58

Slide 58 text

Redis Cluster Redis Cluster Key: User-A ID Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. + Service X

Slide 59

Slide 59 text

Redis Cluster Redis Cluster Key: User-A ID Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. Key: User-A ID Value: Item-X ID, Item-Y ID, …. + Service X + Service Y

Slide 60

Slide 60 text

Redis Cluster Redis Cluster Key: User-A ID Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. Key: User-A ID Value: Item-X ID, Item-Y ID, …. + Service X + Service Y

Slide 61

Slide 61 text

Redis Cluster Redis Cluster Key: User-A ID Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. Key: User-A ID Value: Item-X ID, Item-Y ID, …. + Service X + Service Y

Slide 62

Slide 62 text

Redis Cluster Redis Cluster Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. Value: Item-X ID, Item-Y ID, …. + Service X + Service Y Key: {User-A ID} Key: {User-A ID}

Slide 63

Slide 63 text

Redis Cluster Redis Cluster Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. Value: Item-X ID, Item-Y ID, …. + Service X + Service Y Key: {User-A ID} Key: {User-A ID}

Slide 64

Slide 64 text

Redis Cluster Redis Cluster Node 3 Node 2 Node 1 Value: Item-A ID, Item-B ID, …. Value: Item-X ID, Item-Y ID, …. + Service X + Service Y Key: {User-A ID} Key: {User-A ID} Value: Item-C ID, Item-D ID, …. + Service X Key: {User-B ID}

Slide 65

Slide 65 text

Data Modeling Type Data format Distribution Key Targeting Redis {User-A}:Service x => Item-A ID, Item-B ID, … {User-A}:Service Y => Item-X ID, Item-Y ID, … {User-B}:Service X => Item-C ID, Item-D, … User ID

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Delivery flow Information

Slide 68

Slide 68 text

Delivery flow Targeting Redis Information Fetch Targeting

Slide 69

Slide 69 text

Delivery flow Targeting Redis Fetch Information Information Redis Information Fetch Targeting

Slide 70

Slide 70 text

Delivery flow Targeting Redis Fetch Information Information Redis Information Fetch Targeting Ranking

Slide 71

Slide 71 text

Look before you leap - Measure performance per web server

Slide 72

Slide 72 text

Look before you leap - Measure performance per web server

Slide 73

Slide 73 text

Look before you leap - Measure performance per web server

Slide 74

Slide 74 text

Look before you leap - Release gradually - Measure performance per web server

Slide 75

Slide 75 text

Look before you leap - Release gradually - Always A/B test if anything is unclear - Measure performance per web server

Slide 76

Slide 76 text

Troubles & Solutions

Slide 77

Slide 77 text

1st trouble Targeting Redis Fetch Information Information Redis Information Fetch Targeting Ranking

Slide 78

Slide 78 text

1st trouble Targeting Redis Fetch Information Information Redis Information Fetch Targeting Ranking

Slide 79

Slide 79 text

Sudden slowdown Response 2020/11/10

Slide 80

Slide 80 text

Sudden slowdown Response 99%tile 50ms 2020/11/10

Slide 81

Slide 81 text

Sudden slowdown Response 99%tile 50ms 2020/11/10 2020/12/1

Slide 82

Slide 82 text

Sudden slowdown Response 99%tile 50ms 99%tile 200ms 2020/11/10 2020/12/1

Slide 83

Slide 83 text

Information Redis Stats Time

Slide 84

Slide 84 text

Information Redis Stats Count Time

Slide 85

Slide 85 text

Recommended Item Increasing ↓11/1 8.3 items/user 12/5 20.5 items/user↓

Slide 86

Slide 86 text

Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID

Slide 87

Slide 87 text

Fetch Information Process Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID

Slide 88

Slide 88 text

Fetch Information Process Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F

Slide 89

Slide 89 text

Fetch Information Process Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Lettuce (mget)

Slide 90

Slide 90 text

Fetch Information Process Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Lettuce (mget)

Slide 91

Slide 91 text

Fetch Information Process Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID Item F ID Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Lettuce (mget)

Slide 92

Slide 92 text

Slide 93

Slide 93 text

Fetch Information Process Hypothesis - 1 Fetch Target Process Targeting Redis Item a ID Item B ID Item D ID Item C ID Item F ID Item e ID Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Lettuce (mget)

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Verification -1 Execution time (99%tile)

Slide 96

Slide 96 text

Verification -1 1-9 items Execution time (99%tile)

Slide 97

Slide 97 text

Verification -1 1-9 items 10-19 items Execution time (99%tile)

Slide 98

Slide 98 text

Verification -1 1-9 items 10-19 items 20-29 items Execution time (99%tile)

Slide 99

Slide 99 text

Verification -1 1-9 items 10-19 items 20-29 items 30-39 items Execution time (99%tile)

Slide 100

Slide 100 text

Verification -1 1-9 items 10-19 items 20-29 items 30-39 items Over 40 items Execution time (99%tile)

Slide 101

Slide 101 text

Verification -1 1-9 items 10-19 items 20-29 items 30-39 items Over 40 items Execution time (99%tile) Command count

Slide 102

Slide 102 text

Verification -1 1-9 items 10-19 items 20-29 items 30-39 items Over 40 items Execution time (99%tile) Command count 1-9 items 10-19 items 20-29 items 30-39 items Over 40 items

Slide 103

Slide 103 text

Targeting Hypothesis - 2 X Y Z

Slide 104

Slide 104 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 X Y Z

Slide 105

Slide 105 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID X Y Z

Slide 106

Slide 106 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID X Y Z

Slide 107

Slide 107 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID Item a ID Item F ID X Y Z

Slide 108

Slide 108 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID Item a ID Item F ID X Y Z

Slide 109

Slide 109 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID Item a ID Item F ID Item D ID Item B ID X Y Z

Slide 110

Slide 110 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID Item a ID Item F ID Item D ID Item B ID X Y Z

Slide 111

Slide 111 text

Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Fetch Information Process Hypothesis - 2 Item a ID Item B ID Item a ID Item F ID Item D ID Item B ID X Y Z

Slide 112

Slide 112 text

Problem & Solution # Issue Problem

Slide 113

Slide 113 text

Problem & Solution # Issue Problem 1 Burst of MGET command • Recommend items increased • Need access to multiple Redis nodes to retrieve information

Slide 114

Slide 114 text

Problem & Solution # Issue Problem 1 Burst of MGET command • Recommend items increased • Need access to multiple Redis nodes to retrieve information 2 Concentration of Targeting • There is content recommended by many people • Concentration of access to specific node

Slide 115

Slide 115 text

Solution1 - Burst of MGET command Fetch Information Information Redis Information Targeting Redis Fetch Targeting Ranking

Slide 116

Slide 116 text

Solution1 - Burst of MGET command Fetch Information Information Redis Information Targeting Redis Fetch Targeting Ranking

Slide 117

Slide 117 text

Solution1 - Burst of MGET command Fetch Information Information Redis Information Targeting Redis Fetch Targeting Ranking Limit

Slide 118

Slide 118 text

Solution1 - Burst of MGET command Fetch Information Information Redis Information Targeting Redis Fetch Targeting Ranking Limit ↓ Not Smart

Slide 119

Slide 119 text

Solution1 - Burst of MGET command Fetch Information Information Redis Information Targeting Redis Fetch Targeting Ranking

Slide 120

Slide 120 text

Solution1 - Burst of MGET command Fetch Information Information Redis Information Targeting Redis Fetch Targeting Ranking Smart Filter

Slide 121

Slide 121 text

Smart Filter Information Fetch Targeting Ranking Filter Fetch Information

Slide 122

Slide 122 text

Smart Filter Information Fetch Targeting Ranking Filter Information Filter Targeting Filter Fetch Information

Slide 123

Slide 123 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Fetch Information

Slide 124

Slide 124 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Fetch Information

Slide 125

Slide 125 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information

Slide 126

Slide 126 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information

Slide 127

Slide 127 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information Tier 1 (user log based)

Slide 128

Slide 128 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information Tier 1 (user log based) Tier 2 (user attribute based)

Slide 129

Slide 129 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information Tier 1 (user log based) Tier 2 (user attribute based) Limit N = 5

Slide 130

Slide 130 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information Tier 1 (user log based) Tier 2 (user attribute based) Limit N = 5 Item Item Item

Slide 131

Slide 131 text

Smart Filter Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information Tier 1 (user log based) Tier 2 (user attribute based) Limit N = 5 Item Item Item Item Item Item Item Item Item

Slide 132

Slide 132 text

Slide 133

Slide 133 text

Slide 134

Slide 134 text

Smart Filter Tier 1 (user log based) Tier 2 (user attribute based) Limit N = 5 Item Item Item Item Item Item Item Item Item Item Item Item Information Fetch Targeting Ranking Information Filter Targeting Filter Limit Fetch Information

Slide 135

Slide 135 text

Slide 136

Slide 136 text

Solution2 - Concentration of Targeting Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Item a ID Item B ID Item a ID Item F ID Item D ID Item B ID X Y Z

Slide 137

Slide 137 text

Solution2 - Concentration of Targeting Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Item a ID Item B ID Item a ID Item F ID Item D ID Item B ID X Y Z

Slide 138

Slide 138 text

Solution2 - Concentration of Targeting Information Redis Node 1 Node 3 Node 2 Item A Item B Item C Item D Item E Item F Targeting Item a ID Item B ID Item a ID Item F ID Item D ID Item B ID Cache (size=4) X Y Z

Slide 139

Slide 139 text

Slide 140

Slide 140 text

Slide 141

Slide 141 text

Slide 142

Slide 142 text

Slide 143

Slide 143 text

Slide 144

Slide 144 text

Slide 145

Slide 145 text

Slide 146

Slide 146 text

Slide 147

Slide 147 text

Slide 148

Slide 148 text

Slide 149

Slide 149 text

Slide 150

Slide 150 text

Results Percentile Response Time Before After 99.9 352ms 67ms 99.0 176ms 29ms 95.0 126ms 22ms 75.0 92ms 16ms

Slide 151

Slide 151 text

Results Percentile Response Time Before After 99.9 352ms 67ms 99.0 176ms 29ms 95.0 126ms 22ms 75.0 92ms 16ms 80% faster

Slide 152

Slide 152 text

Earthquake 2nd trouble

Slide 153

Slide 153 text

Earthquake CHIBA M 5.9 2021/10/7 22:41 https://www.jma.go.jp/jma/press/2110/08c/202110080050.html 2nd trouble

Slide 154

Slide 154 text

Traffic explosion

Slide 155

Slide 155 text

Traffic explosion Request per second

Slide 156

Slide 156 text

Traffic explosion Request per second Application server threads

Slide 157

Slide 157 text

Service effect Earthquake notice expected

Slide 158

Slide 158 text

Service effect Earthquake notice expected Some request time outed…

Slide 159

Slide 159 text

Prioritized Delivery Fetch Information Information Fetch Targeting Ranking Prioritized Contents Information

Slide 160

Slide 160 text

Prioritized Delivery Fetch Information Information Fetch Targeting Ranking Prioritized Contents Information Disaster information threshold

Slide 161

Slide 161 text

Prioritized Delivery Fetch Information Information Fetch Targeting Ranking Prioritized Contents Information Disaster information threshold Japanese earthquake scale >= 6

Slide 162

Slide 162 text

Prioritized Delivery Fetch Information Information Fetch Targeting Ranking Prioritized Contents Information Disaster information threshold Japanese earthquake scale >= 6 10/7 earthquake scale = 5+

Slide 163

Slide 163 text

Scale-out Change Threshold Solution

Slide 164

Slide 164 text

Scale-out Change Threshold Solution Servers 25 → 50

Slide 165

Slide 165 text

Scale-out Change Threshold Solution Japanese earthquake scale 6 → 5 Servers 25 → 50

Slide 166

Slide 166 text

Earthquake Result

Slide 167

Slide 167 text

Earthquake FUKUSIHIMA M7.3 2022/3/16 23:36 https://www.jma.go.jp/jma/press/2203/17a/202203170130.html Result

Slide 168

Slide 168 text

Result 105k req/sec

Slide 169

Slide 169 text

Result 105k req/sec Response time 88ms(99.9%)

Slide 170

Slide 170 text

Result 105k req/sec Response time 88ms(99.9%) MGET 367µs

Slide 171

Slide 171 text

Result 105k req/sec Response time 88ms(99.9%) MGET 367µs

Slide 172

Slide 172 text

Unexpected issue

Slide 173

Slide 173 text

Unexpected issue - My house lost power

Slide 174

Slide 174 text

Unexpected issue - My house lost power - Candles and tethering were used

Slide 175

Slide 175 text

Lessons learned

Slide 176

Slide 176 text

In the design - Define system requirements for the system in numerical terms

Slide 177

Slide 177 text

In the design - Be prepared for errors in assumptions and unexpected behavior that can be easily controlled - Define system requirements for the system in numerical terms

Slide 178

Slide 178 text

In the design - Be prepared for errors in assumptions and unexpected behavior that can be easily controlled - The growth of the service sometimes be a problem. And it usually comes just when you’ve forgotten about it. - Define system requirements for the system in numerical terms

Slide 179

Slide 179 text

In the trouble - Logs and metrics are very important. If they are missing, add them as needed.

Slide 180

Slide 180 text

In the trouble - We have to make hypotheses and test them one by one. - Logs and metrics are very important. If they are missing, add them as needed.

Slide 181

Slide 181 text

In the trouble - We have to make hypotheses and test them one by one. - The evolution of technology bring us another difficulty. There is no silver bullet yet. - Logs and metrics are very important. If they are missing, add them as needed.

Slide 182

Slide 182 text

No content

Slide 183

Slide 183 text

I have not failed.

Slide 184

Slide 184 text

I have not failed. I’ve just found 10,000 ways that won’t work.

Slide 185

Slide 185 text

I have not failed. I’ve just found 10,000 ways that won’t work. - Thomas Edison