Share
Embed
LINE Shopチームのあけおめ対応 / LINE Shop team's New Year Work
2020/5/22に行われたLINE Developer Meetup #63 での登壇資料です
https://line.connpass.com/event/174730/
LINE Developers
May 22, 2020
Transcript
51 slides
-
LINE Shop team’s New Year work 2020/05/22 ʲOnlineʳLINE Developer Meetup
#63 ( https://line.connpass.com/event/174730/ ) LINE Fukuoka Corp, Development 1 Dept. Manabu Matsuzaki -
About me @matsumana LINE Fukuoka Corp, Development 1 Dept. SRE/Server
side Engineer https://github.com/matsumana Manabu Matsuzaki -
• Introduction to LINE Shop team • LINE Shop service
architecture • LINE Shop’s Network traffic on New Year • Preparation for New Year • Result of the New Year's day • Improvements for next year Agenda -
What is LINE Shop team? •We develop Sticker/Emoji/Theme features (buy,
use, etc) for LINE app and Web • Sticker Shop and Theme Shop in LINE app • Web store named LINE STORE (https://store.line.me/) • LINE wallet in LINE app’s Wallet tab (Not Sticker/Emoji/Theme, though) -
What is LINE Shop team? •Team members (server-side engineers) •
Tokyo: about 15 • Fukuoka: about 10 -
What is LINE Shop team? •We collaborate with other teams
• Native app engineers (iOS, Android) • Frontend engineers (called UIT) • DBA • In-house platform engineers (Monitoring, Private Cloud, Data Analysis, etc) -
How big LINE Shop’s service scale? •Number of Stickers *1
• 8,550,000 LINE Sticker sets are on sale (As of Mar 2020) • 433,000,000 stickers sent per day on average (As of April 2019) •RPS(requests/sec) • Usual: ~ 50K RPS • Event/New Year: ~ 100K RPS *1 en: https://linecorp.com/en/pr/news/en/2020/3157 ja:https://linecorp.com/ja/pr/news/ja/2020/3127 -
Custom Stickers • Short text customizable Stickers • en: https://linecorp.com/ja/pr/news/en/2019/2666
• ja: https://linecorp.com/ja/pr/news/ja/2019/2664 -
Stickers Premium • Get unlimited access to over 3,000,000 creators
stickers • en: https://store.line.me/stickers-premium/landing/en • ja: https://store.line.me/stickers-premium/landing/ja http://creator-mag.line.me/ja/archives/1075007192.html -
Message Stickers • Users can add personalized messages • en:
https://linecorp.com/en/pr/news/en/2020/3157 • ja: https://linecorp.com/ja/pr/news/ja/2020/3127 -
LEGY (LINE Event Delivery Gateway) LINE App talk-server Web Browser
LINE STORE (https://store.line.me) shop-proxy (Shop API gateway) shop-server stickershop-server search-fe ownership-server capability-server LINE Shop Overview (include LEGY and talk-server) CWA (Web App in LINE) -
Overview (inside LINE Shop) LINE STORE (https://store.line.me) shop-proxy shop-server stickershop-server
search-fe ownership-server capability-server Product Ownership Used as cache -
•LINE Store: Web store •shop-proxy: Shop API gateway •shop-server: Coordinate
requests •search-fe: Search frontend •ownership-server: Products(sticker, emoji, theme) ownership •capability-server: capability check Overview (inside LINE Shop) -
Overview (Framework, monitoring) Microservice A Microservice B Microservice C Tracing
Logging Metrics IMON (in-house monitoring system) •Armeria (Microservice Framework written in Java) • REST/Thrift/gRPC, HTTP2, Service Discovery, Circuit Breaker, etc -
What do we monitor? •Prometheus+Grafana or IMON • Service latency
(50th, 90th, 99th percentile, etc) • Amount of logs (Warn, Error) • JVM (GC, Heap, etc) • Server load (CPU, Memory, Network Traffic, etc) • etc… -
What do we monitor? •Elasticsearch+Kibana • Raw log •Zipkin •
Identify performance bottlenecks in microservices -
A bunch of requests will come than usual •New Year
campaign • ྩॳͷ͓ਖ਼݄ɺLINEͷʮ͓ۄլελϯϓʯͰ৽ͷѫࡰΛɻ େਓؾΩϟϥΫλʔͳͲɺ800छྨҎ্ͷελϯϓɾֆจࣈ͕ొʂ https://linecorp.com/ja/pr/news/ja/2019/3038 -
A bunch of requests will come than usual •Messages from
LINE Sticker official account • As of Feb 2020, the account has 57,000,000 friends • Increasing continuously (YoY: x1.5) • The account sends messages to friends regularly -
Summary so far •LINE Shop team develops Sticker/Emoji/Theme features •Service
scale • 8,550,000 LINE Sticker sets are on sale (As of Mar 2020) • 433,000,000 stickers sent per day on average (As of April 2019) • RPS • Usual: ~ 50K RPS • Event/New Year: ~ 100K RPS •Introduced LINE Shop service architecture -
Summary so far • New Year is one of the
highest traffic event for our service • New Year greetings • New Year campaign • Messages from LINE Sticker official account -
Preparation for New Year • Measure capacity for each microservice
• Estimate number of servers and spec based on measured capacity and past events • Last New Year • Other high traffic events -
Measure capacity for each microservice • Measured using production traffic
• Service out existing servers little by little while monitoring -
Measure capacity for each microservice •Measured using production traffic •
Service out existing servers little by little while monitoring • RPS per one server • API latency -
Measure capacity for each microservice •Improvement ideas for capacity measurement
• Control traffic ratio for one server (e.g. Weighted Round Robin) • Continuous load test (like a CI) • etc -
Estimation based on past event’s metrics •Determine server spec and
number of servers • Based on API latency, server’s load, etc • e.g. • Supposed total RPS: 10K • Current capacity per one app server: 1K • As the result: 10K / 1K = Need 10 servers -
Estimation based on past event’s metrics •Need to care DB
server’s load and setting as well • If we add many app servers, DB servers may become bottle neck • Connection error might occur due to DB’s max connections -
Estimation based on past event’s metrics •If necessary, replace existing
app servers with higher spec servers • Advantages: • Reduce DB connections • Make faster server provisioning (we use Ansible) • Reduce server operation cost -
RPS (right after 2020/01/01 12:00am) •In several microservices, RPS was
higher than last year •YoY x1.7 ~ x4.9 on each microservice • shop-proxy: 124.1k RPS (YoY x1.7) • search-fe: 142.0k RPS (YoY x4.9) -
Service failure occurred •Unfortunately, service failure occurred right after Jan
1st 12:00am •Users couldn’t edit text, send and receive Custom Stickers •Sorry for inconvenience -
Service failure occurred • Due to lack of server resources
• Custom Stickers’ requests higher than our expectation • There was no data for 2018 • Custom Stickers has been released in 2019 • Added app servers for Custom Stickers urgently • However, it took long time due to server provisioning -
Service failure occurred • Fortunately the failure didn’t affect whole
of our service • Affected only Custom Sticker • This is one of the benefit of microservices • Well designed microservices can mitigate impact -
Postmortem Culture in LINE • If service failure occur, we
hold a postmortem meeting • Discuss with related teams to prevent future failures -
Postmortem Culture in LINE • Items of postmortem report •
Failure detection time • Affected services • Cause • Timeline • How to prevent future failures • How to improve failure detection • How to improve failure handling -
Improvements for this service failure •Implement rate limiter(throttling) for Custom
Sticker server • Rate limiter can reduce server load •Make server scale-out faster • More automation • Make Ansible execution duration faster • Consider to migrate to k8s -
Conclusion •New Year is one of the highest traffic event
for our service •Preparation for New Year • Capacity measurement • Consider scale-up or scale-out based on measured capacity and past events -
Conclusion • In several microservices, RPS was higher than last
year • The service failure occurred in part of our service • However, thanks to microservices, the failure didn't affect to whole service • We organize postmortem meeting with related teams -
Related LINE Engineering Blog article (2020 New Year campaign) •High-throughput
distributed rate limiter https://engineering.linecorp.com/en/blog/high-throughput- distributed-rate-limiter/ -
Related sessions at LINE DEV DAY 2019 (Custom Stickers) •How
Custom Sticker Development Works https://linedevday.linecorp.com/jp/2019/sessions/S1-02 •Custom Elements in Custom Stickers https://linedevday.linecorp.com/jp/2019/sessions/S2-10 -
• Long road to microservices architecture at LINE messaging platform
https://linedevday.linecorp.com/jp/2019/sessions/D1-6 • Armeria: A Microservice Framework Well-suited Everywhere https://linedevday.linecorp.com/jp/2019/sessions/D2-2 Related sessions at LINE DEV DAY 2019 (Architecture) -
•LINE's New Year campaign - Taking Control of High Traffic
https://linedevday.linecorp.com/jp/2019/sessions/S1-09 Related sessions at LINE DEV DAY 2019 (2019 New Year campaign)