Share
Embed
Transcript
27 slides
-
あけおめLINE 傾向とその対策 The Preparation for the "New Year Bursting" Developer
Platform CBU Reliability Engineering Unit SRE9 Division @nasa9084 © LY Corporation -
$ whoami - @nasa9084 - Work: - 2017/04 - 2018/09:
Python/Kubernetes on OpenStack/AWS/Azure - 2018/10 - 2021/09: Jenkins, CircleCI:e, internal metrics agent - 2021/10 - now: Current team - SRE for Stickershop, Wallet tab, LYP Premium - Terraform provider development - Personal Projects/Hobbies: - Kubernetes SIG Docs Localization Subproject Japanese Localization Team Lead - Music Instruments - Indoor Gardening - Home automation © LY Corporation 2 -
LINE Sticker / Theme / Emoji You can buy stickers
on LINE sticker shop and then you can send them © LY Corporation 3 -
LINE Shop Sticker / Theme / Emoji shop on LINE
App © LY Corporation 4 -
LINE Store Sticker / Theme / Emoji store on web
© LY Corporation 5 -
SRE-related Projects/Tasks for Stickershop Routine / Adhoc - Prepare for
New Year Bursting - Hardware / OS / Middleware upgrade/replacement - Kubernetes migration - Introduce new softwares - SLI/SLO - Security Handling - CI/CD - Budget Planning © LY Corporation 6 -
SRE-related Projects for Stickershop Routine / Adhoc - Prepare for
New Year Bursting - Hardware / OS / Middleware upgrade/replacement - Kubernetes migration - Introduce new softwares - SLI/SLO - Security Handling - CI/CD - Budget Planning © LY Corporation 7 -
The New Year Bursting - Users send "A Happy New
Year" Greetings at 01/01 00:00 - Also important for business: a lot of stickers for the New Year event are sold © LY Corporation 8 -
Preparing for the "New Year Bursting" - Summarize changes in
the year - Summarize outages in the year - Throttling workshop - Prepare dashboards - Try to reduce requests © LY Corporation 10 -
Preparing for the "New Year Bursting" - Check handleable #requests
for each microservice by the "CPU 70% experiment" - Reduce actual servers until the CPU utilization reaches 70% or the latency increases - Plan scaling down the server spec if there're too few requests © LY Corporation 11 -
Preparing for the "New Year Bursting" - Estimate peak requests
using: - New Year Bursting growth rate: n #a1#a2 . . . #an where ai = #reqthis year−i) #reqthis year−i−1 - Weekday growth rate - #requests in the last year - #instances = (estimated peak access / handleable #requests per server) * margin © LY Corporation 12 -
Preparing for the "New Year Bursting" Example #req - Estimate
peak requests using: - New Year Bursting growth rate: - Weekday growth rate - #requests in the last year 3 1.06 × 1.01 × 1.23 = 1.1 2023 3,000 2024 3,200 1.06 2025 3,250 1.01 2026 4,000 1.23 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation YoY 13 -
Preparing for the "New Year Bursting" - Estimate peak requests
using: - New Year Bursting growth rate: 4000 × 1.1 = 4400 - Weekday growth rate - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 14 -
Preparing for the "New Year Bursting" Example #req - Estimate
peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 1200 = 1.2 1000 - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 2025 Weekday 2026 Weekday 2026 01/01 1,000 1,200 4,000 15 -
Preparing for the "New Year Bursting" - Estimate peak requests
using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4000 × 1.2 = 4800 - #requests in the last year - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 16 -
Preparing for the "New Year Bursting" Example #req - Estimate
peak requests using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4800 2026 01/01 4,000 - #requests in the last year: 4000 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 17 -
Preparing for the "New Year Bursting" - Estimate peak requests
using: - New Year Bursting growth rate: 4400 - Weekday growth rate: 4800 ← Use this - #requests in the last year: 4000 - #instances = (estimated peak access / handlable #requests per server) * margin © LY Corporation 18 -
Preparing for the "New Year Bursting" - Estimate peak requests:
4800 - #instances = (4800 ÷ #handleable_request) × margin © LY Corporation 19 -
Preparing for the "New Year Bursting" - Also: - Make
a plan for the next year - Write a monitoring report - Hold a review/feedback meeting © LY Corporation 26