Long road to microservices architecture at LINE messaging platform.

Transcript

1. 1.

   2019 DevDay Long Road to Microservices Architecture at LINE Messaging

   Platform. > Masahiro Ide > LINE Z Part Team
2. 2.

   Agenda > How did we scale our services > Weapons

   we prepared for our service microservices > How do we growth our architecture in future
3. 3.

   When we launch LINE service LINE Messaging Platform (2011) >

   A Monolithic Service • Everything are handled at talk-server > Team • 1 team, 6 members LEGY talk-server Redis MySQL Server team
4. 4.

   LINE Messaging Platform (2011~2019) Social Graph VoIP Message Sticker Theme

   LINE@ Group Search LINE NEWS Recommendation Game Timeline Image Video Payment Wallet Clova OpenChat Social Login Emoji Settings
5. 5.

   LINE Messaging Platform (2011~2019) LEGY talk-server Server team ...

6. 6.

   LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

   let’s focus on Video share feature
7. 7.

   LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

   let’s focus on Video share feature Oh, wait. I want to focus on sticker
8. 8.

   LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

   let’s focus on Video share feature Oh, wait. I want to focus on sticker Waaait!! VoIP has highest priority!!
9. 9.

   LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Image/Video

   Delivery
10. 10.

    LINE Messaging Platform (2011~2019) LEGY talk-server ... Image/Video Delivery Stickershop

    Want to make more new type of sticker!! Let’s consider next-gen protocol Let’s improve img/video quality Let’s focus on new messaging feature
11. 11.

    Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
12. 12.

    Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop x8 x25 x10 x31 x6 x9 Service x1 Service >20 Service x1 Service >40 Service >20 Service x7
13. 13.

    Microservice architecture solves everything? > Good ! • Conflicts between

    features ↓ • Development speed ↑ ↑ > Not good " • Network failure ↑ • Potential Cascading failure ↑ • Manage more service configs ↑
14. 14.

    Concepts need to build our microservices architecture Armeria Connectivity LEGY

    Routing CentralDogma Directory Service
15. 15.

    Connectivity Armeria - Our RPC layer > Asynchronous RPC/REST library

    • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC • https://github.com/line/armeria > Take care common functionality for microservice • Logging, Client side Load Balancing, Monitoring integration, Tracing • Retry, Throttling, Circuit Breaker, Health Check
16. 16.

    Structured Logging Client Connectivity Armeria - Our RPC layer gRPC/Thrift

    Client HTTP/2 Client Client side Load Balancer Circuit Breaker Metrics Integration Structured Logging Service gRPC/Thrift Service HTTP/2 Service Zipkin Service Rate Limiting Service Metrics Integration
17. 17.

    Directory Service Central Dogma > Repository service for textual configurations

    • Highly available, version controlled, advanced query mechanism > Stores start-time/run-time configuration safely > https://github.com/line/centraldogma/ JSON, YAML, XML, JS…
18. 18.

    Directory Service Central Dogma TKSVR123 Find talk-server-release servers TKSVR124 ...

    [{hostname:TKSVR123, port: 8080}, {hostname:TKSVR124, port: 8080,},... ] Armeria Client http://group:talk-server-relase/api group:talk-server-relase → TKSVR123:8080
19. 19.

    Routing - LEGY DataCenter 1 Main DataCenter App Server LEGY

    DataCenter 2 LEGY LEGY App Server > LINE Event Delivery Gateway – API Gateway written in Erlang • SPDY (& HTTP/2) • Global POP • Routing DSL
20. 20.

    Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
21. 21.

    Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
22. 22.

    talk-server and stickershop example Case study > Making sticker ownership

    validation feature as microservices > Service redesign project between talk-server & Sticker Shop at 2017 • Rearranges service boundary
23. 23.

    Motivation for reconstruction > Increase the number of stickers sold

    due to Creators Market launch > Increasing number of users who have used LINE since long times ago • Some users have 20,000 stickers. # of user # of owned stickers
24. 24.

    How to send a sticker at 2017 > Save sticker

    ownership data to Redis for each user as a Set • Refresh all user ownership when a user purchase a sticker > Check Redis data when a user sends a sticker Stickershop LEGY talk-server Redis
25. 25.

    What happens if services are not separated > Talk-server stops

    every time a mass-sticker owned user buy a sticker LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
26. 26.

    What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
27. 27.

    What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
28. 28.

    How to solve this? > Separate responsibilities and services not

    to block each development > Make feature fast-fail-able not to affect each other • Circuit breaker + automatic request retry Stickershop LEGY talk-server Sticker Capability service Sticker Capability service
29. 29.

    Retrospective > Rebuilt with microservices architecture • without degrading service

    quality thanks to our weapons > Require a mechanism that can flexibly respond to • Scale • Speed
30. 30.

    Our Futures > Make our platform more fine-grained microservices Feature

    Feature Feature To: separating by Function Function Function Function Function Function Function Function Function Function Function Function Function From: separating by Feature
31. 31.

    Wrap up of this talk > We adopt microservices architecture

    to speed up our development > Prepared our weapons for microservices architecture • Armeria, CentralDogma, LEGY > We’re continuing to transform our service to microservices architecture