あけおめLINE 傾向とその対策

#sapporo_sre #0

Transcript

1. あけおめLINE 傾向とその対策 The Preparation for the "New Year Bursting" Developer Platform CBU Reliability Engineering Unit SRE9 Division @nasa9084 © LY Corporation
2. $ whoami - @nasa9084 - Work: - 2017/04 - 2018/09: Python/Kubernetes on OpenStack/AWS/Azure - 2018/10 - 2021/09: Jenkins, CircleCI:e, internal metrics agent - 2021/10 - now: Current team - SRE for Stickershop, Wallet tab, LYP Premium - Terraform provider development - Personal Projects/Hobbies: - Kubernetes SIG Docs Localization Subproject Japanese Localization Team Lead - Music Instruments - Indoor Gardening - Home automation © LY Corporation 2
3. LINE Sticker / Theme / Emoji You can buy stickers on LINE sticker shop and then you can send them © LY Corporation 3
4. LINE Shop Sticker / Theme / Emoji shop on LINE App © LY Corporation 4
5. LINE Store Sticker / Theme / Emoji store on web © LY Corporation 5
6. SRE-related Projects/Tasks for Stickershop Routine / Adhoc - Prepare for New Year Bursting - Hardware / OS / Middleware upgrade/replacement - Kubernetes migration - Introduce new softwares - SLI/SLO - Security Handling - CI/CD - Budget Planning © LY Corporation 6
7. SRE-related Projects for Stickershop Routine / Adhoc - Prepare for New Year Bursting - Hardware / OS / Middleware upgrade/replacement - Kubernetes migration - Introduce new softwares - SLI/SLO - Security Handling - CI/CD - Budget Planning © LY Corporation 7
8. The New Year Bursting - Users send "A Happy New Year" Greetings at 01/01 00:00 - Also important for business: a lot of stickers for the New Year event are sold © LY Corporation 8
9. Preparing for the "New Year Bursting" © LY Corporation 9
10. Preparing for the "New Year Bursting" - Summarize changes in the year - Summarize outages in the year - Throttling workshop - Prepare dashboards - Try to reduce requests © LY Corporation 10
11. Preparing for the "New Year Bursting" - Check handleable #requests for each microservice by the "CPU 70% experiment" - Reduce actual servers until the CPU utilization reaches 70% or the latency increases - Plan scaling down the server spec if there're too few requests © LY Corporation 11
12. Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: n #a1#a2 . . . #an where ai = #reqthis year−i) #reqthis year−i−1 - Weekday growth rate - #requests in the last year - #instances = (estimated peak access / handleable #requests per server) * margin © LY Corporation 12
13. Preparing for the "New Year Bursting" Example #req - Estimate peak requests using: - New Year Bursting growth rate: - Weekday growth rate - #requests in the last year 3 1.06 × 1.01 × 1.23 = 1.1 2023 3,000 2024 3,200 1.06 2025 3,250 1.01 2026 4,000 1.23 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation YoY 13
14. Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: 4000 × 1.1 = 4400 - Weekday growth rate - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 14
15. Preparing for the "New Year Bursting" Example #req - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 1200 = 1.2 1000 - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 2025 Weekday 2026 Weekday 2026 01/01 1,000 1,200 4,000 15
16. Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4000 × 1.2 = 4800 - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 16
17. Preparing for the "New Year Bursting" Example #req - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4800 2026 01/01 4,000 - #requests in the last year: 4000 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 17
18. Preparing for the "New Year Bursting" - Estimate peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4800 ← Use this - #requests in the last year: 4000 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 18
19. Preparing for the "New Year Bursting" - Estimate peak requests: 4800 - #instances = (4800 ÷ #handleable_request) × margin © LY Corporation 19
20. Preparing for the "New Year Bursting" © LY Corporation 20
21. Preparing for the "New Year Bursting" © LY Corporation 21
22. Preparing for the "New Year Bursting" 🎉 © LY Corporation 22
23. Preparing for the "New Year Bursting" 🎉 © LY Corporation 23
24. Preparing for the "New Year Bursting" 🎉 © LY Corporation 24
25. Preparing for the "New Year Bursting" 🛏 © LY Corporation 25
26. Preparing for the "New Year Bursting" - Also: - Make a plan for the next year - Write a monitoring report - Hold a review/feedback meeting © LY Corporation 26
27. Q? © LY Corporation