あけおめLINE 傾向とその対策 The Preparation for the "New Year Bursting" Developer Platform CBU Reliability Engineering Unit SRE9 Division @nasa9084 © LY Corporation 

$ whoami - @nasa9084 - Work: - 2017/04 - 2018/09: Python/Kubernetes on OpenStack/AWS/Azure - 2018/10 - 2021/09: Jenkins, CircleCI:e, internal metrics agent - 2021/10 - now: Current team - SRE for Stickershop, Wallet tab, LYP Premium - Terraform provider development - Personal Projects/Hobbies: - Kubernetes SIG Docs Localization Subproject Japanese Localization Team Lead - Music Instruments - Indoor Gardening - Home automation © LY Corporation 2 

LINE Sticker / Theme / Emoji You can buy stickers on LINE sticker shop and then you can send them © LY Corporation 3 

LINE Shop Sticker / Theme / Emoji shop on LINE App © LY Corporation 4 

LINE Store Sticker / Theme / Emoji store on web © LY Corporation 5 

SRE-related Projects/Tasks for Stickershop Routine / Adhoc - Prepare for New Year Bursting - Hardware / OS / Middleware upgrade/replacement - Kubernetes migration - Introduce new softwares - SLI/SLO - Security Handling - CI/CD - Budget Planning © LY Corporation 6 

SRE-related Projects for Stickershop Routine / Adhoc - Prepare for New Year Bursting - Hardware / OS / Middleware upgrade/replacement - Kubernetes migration - Introduce new softwares - SLI/SLO - Security Handling - CI/CD - Budget Planning © LY Corporation 7 

The New Year Bursting - Users send "A Happy New Year" Greetings at 01/01 00:00 - Also important for business: a lot of stickers for the New Year event are sold © LY Corporation 8 

Preparing for the "New Year Bursting" © LY Corporation 9 

Preparing for the "New Year Bursting" - Summarize changes in the year - Summarize outages in the year - Throttling workshop - Prepare dashboards - Try to reduce requests © LY Corporation 10 

Preparing for the "New Year Bursting" - Check handleable #requests for each microservice by the "CPU 70% experiment" - Reduce actual servers until the CPU utilization reaches 70% or the latency increases - Plan scaling down the server spec if there're too few requests © LY Corporation 11 

Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: n #a1#a2 . . . #an where ai = #reqthis year−i) #reqthis year−i−1 - Weekday growth rate - #requests in the last year - #instances = (estimated peak access / handleable #requests per server) * margin © LY Corporation 12 

Preparing for the "New Year Bursting" Example #req - Estimate peak requests using: - New Year Bursting growth rate: - Weekday growth rate - #requests in the last year 3 1.06 × 1.01 × 1.23 = 1.1 2023 3,000 2024 3,200 1.06 2025 3,250 1.01 2026 4,000 1.23 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation YoY 13 

Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: 4000 × 1.1 = 4400 - Weekday growth rate - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 14 

Preparing for the "New Year Bursting" Example #req - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 1200 = 1.2 1000 - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 2025 Weekday 2026 Weekday 2026 01/01 1,000 1,200 4,000 15 

Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4000 × 1.2 = 4800 - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 16 

Preparing for the "New Year Bursting" Example #req - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4800 2026 01/01 4,000 - #requests in the last year: 4000 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 17 

Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4800 ← Use this - #requests in the last year: 4000 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 18 

Preparing for the "New Year Bursting" - Estimate peak requests: 4800 - #instances = (4800 ÷ #handleable_request) × margin © LY Corporation 19 

Preparing for the "New Year Bursting" © LY Corporation 20 

Preparing for the "New Year Bursting" © LY Corporation 21 

Preparing for the "New Year Bursting" 🎉 © LY Corporation 22 

Preparing for the "New Year Bursting" 🎉 © LY Corporation 23 

Preparing for the "New Year Bursting" 🎉 © LY Corporation 24 

Preparing for the "New Year Bursting" 🛏 © LY Corporation 25 

Preparing for the "New Year Bursting" - Also: - Make a plan for the next year - Write a monitoring report - Hold a review/feedback meeting © LY Corporation 26 

Q? © LY Corporation