Long road to microservices architecture at LINE messaging platform.

Transcript

1. 2019 DevDay Long Road to Microservices Architecture at LINE Messaging

   Platform. > Masahiro Ide > LINE Z Part Team

2. Agenda > How did we scale our services > Weapons

   we prepared for our service microservices > How do we growth our architecture in future

3. When we launch LINE service LINE Messaging Platform (2011) >

   A Monolithic Service • Everything are handled at talk-server > Team • 1 team, 6 members LEGY talk-server Redis MySQL Server team

4. LINE Messaging Platform (2011~2019) Social Graph VoIP Message Sticker Theme

   LINE@ Group Search LINE NEWS Recommendation Game Timeline Image Video Payment Wallet Clova OpenChat Social Login Emoji Settings

5. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ...

6. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

   let’s focus on Video share feature

7. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

   let’s focus on Video share feature Oh, wait. I want to focus on sticker

8. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

   let’s focus on Video share feature Oh, wait. I want to focus on sticker Waaait!! VoIP has highest priority!!

9. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Image/Video

   Delivery

10. LINE Messaging Platform (2011~2019) LEGY talk-server ... Image/Video Delivery Stickershop

    Want to make more new type of sticker!! Let’s consider next-gen protocol Let’s improve img/video quality Let’s focus on new messaging feature

11. Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop

12. Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop x8 x25 x10 x31 x6 x9 Service x1 Service >20 Service x1 Service >40 Service >20 Service x7

13. Microservice architecture solves everything? > Good ! • Conflicts between

    features ↓ • Development speed ↑ ↑ > Not good " • Network failure ↑ • Potential Cascading failure ↑ • Manage more service configs ↑

14. Concepts need to build our microservices architecture Armeria Connectivity LEGY

    Routing CentralDogma Directory Service

15. Connectivity Armeria - Our RPC layer > Asynchronous RPC/REST library

    • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC • https://github.com/line/armeria > Take care common functionality for microservice • Logging, Client side Load Balancing, Monitoring integration, Tracing • Retry, Throttling, Circuit Breaker, Health Check

16. Structured Logging Client Connectivity Armeria - Our RPC layer gRPC/Thrift

    Client HTTP/2 Client Client side Load Balancer Circuit Breaker Metrics Integration Structured Logging Service gRPC/Thrift Service HTTP/2 Service Zipkin Service Rate Limiting Service Metrics Integration

17. Directory Service Central Dogma > Repository service for textual configurations

    • Highly available, version controlled, advanced query mechanism > Stores start-time/run-time configuration safely > https://github.com/line/centraldogma/ JSON, YAML, XML, JS…

18. Directory Service Central Dogma TKSVR123 Find talk-server-release servers TKSVR124 ...

    [{hostname:TKSVR123, port: 8080}, {hostname:TKSVR124, port: 8080,},... ] Armeria Client http://group:talk-server-relase/api group:talk-server-relase → TKSVR123:8080

19. Routing - LEGY DataCenter 1 Main DataCenter App Server LEGY

    DataCenter 2 LEGY LEGY App Server > LINE Event Delivery Gateway – API Gateway written in Erlang • SPDY (& HTTP/2) • Global POP • Routing DSL

20. Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf

21. Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf

22. talk-server and stickershop example Case study > Making sticker ownership

    validation feature as microservices > Service redesign project between talk-server & Sticker Shop at 2017 • Rearranges service boundary

23. Motivation for reconstruction > Increase the number of stickers sold

    due to Creators Market launch > Increasing number of users who have used LINE since long times ago • Some users have 20,000 stickers. # of user # of owned stickers

24. How to send a sticker at 2017 > Save sticker

    ownership data to Redis for each user as a Set • Refresh all user ownership when a user purchase a sticker > Check Redis data when a user sends a sticker Stickershop LEGY talk-server Redis

25. What happens if services are not separated > Talk-server stops

    every time a mass-sticker owned user buy a sticker LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop

26. What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop

27. What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop

28. How to solve this? > Separate responsibilities and services not

    to block each development > Make feature fast-fail-able not to affect each other • Circuit breaker + automatic request retry Stickershop LEGY talk-server Sticker Capability service Sticker Capability service

29. Retrospective > Rebuilt with microservices architecture • without degrading service

    quality thanks to our weapons > Require a mechanism that can flexibly respond to • Scale • Speed

30. Our Futures > Make our platform more fine-grained microservices Feature

    Feature Feature To: separating by Function Function Function Function Function Function Function Function Function Function Function Function Function From: separating by Feature

31. Wrap up of this talk > We adopt microservices architecture

    to speed up our development > Prepared our weapons for microservices architecture • Armeria, CentralDogma, LEGY > We’re continuing to transform our service to microservices architecture