Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Long road to microservices architecture at LINE messaging platform.

Long road to microservices architecture at LINE messaging platform.

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay Long Road to Microservices Architecture at LINE Messaging

    Platform. > Masahiro Ide > LINE Z Part Team
  2. Agenda > How did we scale our services > Weapons

    we prepared for our service microservices > How do we growth our architecture in future
  3. When we launch LINE service LINE Messaging Platform (2011) >

    A Monolithic Service • Everything are handled at talk-server > Team • 1 team, 6 members LEGY talk-server Redis MySQL Server team
  4. LINE Messaging Platform (2011~2019) Social Graph VoIP Message Sticker Theme

    LINE@ Group Search LINE NEWS Recommendation Game Timeline Image Video Payment Wallet Clova OpenChat Social Login Emoji Settings
  5. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ...

  6. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

    let’s focus on Video share feature
  7. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

    let’s focus on Video share feature Oh, wait. I want to focus on sticker
  8. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

    let’s focus on Video share feature Oh, wait. I want to focus on sticker Waaait!! VoIP has highest priority!!
  9. LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Image/Video

    Delivery
  10. LINE Messaging Platform (2011~2019) LEGY talk-server ... Image/Video Delivery Stickershop

    Want to make more new type of sticker!! Let’s consider next-gen protocol Let’s improve img/video quality Let’s focus on new messaging feature
  11. Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  12. Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop x8 x25 x10 x31 x6 x9 Service x1 Service >20 Service x1 Service >40 Service >20 Service x7
  13. Microservice architecture solves everything? > Good ! • Conflicts between

    features ↓ • Development speed ↑ ↑ > Not good " • Network failure ↑ • Potential Cascading failure ↑ • Manage more service configs ↑
  14. Concepts need to build our microservices architecture Armeria Connectivity LEGY

    Routing CentralDogma Directory Service
  15. Connectivity Armeria - Our RPC layer > Asynchronous RPC/REST library

    • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC • https://github.com/line/armeria > Take care common functionality for microservice • Logging, Client side Load Balancing, Monitoring integration, Tracing • Retry, Throttling, Circuit Breaker, Health Check
  16. Structured Logging Client Connectivity Armeria - Our RPC layer gRPC/Thrift

    Client HTTP/2 Client Client side Load Balancer Circuit Breaker Metrics Integration Structured Logging Service gRPC/Thrift Service HTTP/2 Service Zipkin Service Rate Limiting Service Metrics Integration
  17. Directory Service Central Dogma > Repository service for textual configurations

    • Highly available, version controlled, advanced query mechanism > Stores start-time/run-time configuration safely > https://github.com/line/centraldogma/ JSON, YAML, XML, JS…
  18. Directory Service Central Dogma TKSVR123 Find talk-server-release servers TKSVR124 ...

    [{hostname:TKSVR123, port: 8080}, {hostname:TKSVR124, port: 8080,},... ] Armeria Client http://group:talk-server-relase/api group:talk-server-relase → TKSVR123:8080
  19. Routing - LEGY DataCenter 1 Main DataCenter App Server LEGY

    DataCenter 2 LEGY LEGY App Server > LINE Event Delivery Gateway – API Gateway written in Erlang • SPDY (& HTTP/2) • Global POP • Routing DSL
  20. Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
  21. Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
  22. talk-server and stickershop example Case study > Making sticker ownership

    validation feature as microservices > Service redesign project between talk-server & Sticker Shop at 2017 • Rearranges service boundary
  23. Motivation for reconstruction > Increase the number of stickers sold

    due to Creators Market launch > Increasing number of users who have used LINE since long times ago • Some users have 20,000 stickers. # of user # of owned stickers
  24. How to send a sticker at 2017 > Save sticker

    ownership data to Redis for each user as a Set • Refresh all user ownership when a user purchase a sticker > Check Redis data when a user sends a sticker Stickershop LEGY talk-server Redis
  25. What happens if services are not separated > Talk-server stops

    every time a mass-sticker owned user buy a sticker LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  26. What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  27. What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  28. How to solve this? > Separate responsibilities and services not

    to block each development > Make feature fast-fail-able not to affect each other • Circuit breaker + automatic request retry Stickershop LEGY talk-server Sticker Capability service Sticker Capability service
  29. Retrospective > Rebuilt with microservices architecture • without degrading service

    quality thanks to our weapons > Require a mechanism that can flexibly respond to • Scale • Speed
  30. Our Futures > Make our platform more fine-grained microservices Feature

    Feature Feature To: separating by Function Function Function Function Function Function Function Function Function Function Function Function Function From: separating by Feature
  31. Wrap up of this talk > We adopt microservices architecture

    to speed up our development > Prepared our weapons for microservices architecture • Armeria, CentralDogma, LEGY > We’re continuing to transform our service to microservices architecture