Long road to microservices architecture at LINE messaging platform.

Long road to microservices architecture at LINE messaging platform.

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 1.

    2019 DevDay Long Road to Microservices Architecture at LINE Messaging

    Platform. > Masahiro Ide > LINE Z Part Team
  2. 2.

    Agenda > How did we scale our services > Weapons

    we prepared for our service microservices > How do we growth our architecture in future
  3. 3.

    When we launch LINE service LINE Messaging Platform (2011) >

    A Monolithic Service • Everything are handled at talk-server > Team • 1 team, 6 members LEGY talk-server Redis MySQL Server team
  4. 4.

    LINE Messaging Platform (2011~2019) Social Graph VoIP Message Sticker Theme

    LINE@ Group Search LINE NEWS Recommendation Game Timeline Image Video Payment Wallet Clova OpenChat Social Login Emoji Settings
  5. 7.

    LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

    let’s focus on Video share feature Oh, wait. I want to focus on sticker
  6. 8.

    LINE Messaging Platform (2011~2019) LEGY talk-server Server team ... Ok,

    let’s focus on Video share feature Oh, wait. I want to focus on sticker Waaait!! VoIP has highest priority!!
  7. 10.

    LINE Messaging Platform (2011~2019) LEGY talk-server ... Image/Video Delivery Stickershop

    Want to make more new type of sticker!! Let’s consider next-gen protocol Let’s improve img/video quality Let’s focus on new messaging feature
  8. 11.

    Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  9. 12.

    Now LINE Messaging Platform (2019) > Simple app server became

    a huge microservice architecture LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop x8 x25 x10 x31 x6 x9 Service x1 Service >20 Service x1 Service >40 Service >20 Service x7
  10. 13.

    Microservice architecture solves everything? > Good ! • Conflicts between

    features ↓ • Development speed ↑ ↑ > Not good " • Network failure ↑ • Potential Cascading failure ↑ • Manage more service configs ↑
  11. 15.

    Connectivity Armeria - Our RPC layer > Asynchronous RPC/REST library

    • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC • https://github.com/line/armeria > Take care common functionality for microservice • Logging, Client side Load Balancing, Monitoring integration, Tracing • Retry, Throttling, Circuit Breaker, Health Check
  12. 16.

    Structured Logging Client Connectivity Armeria - Our RPC layer gRPC/Thrift

    Client HTTP/2 Client Client side Load Balancer Circuit Breaker Metrics Integration Structured Logging Service gRPC/Thrift Service HTTP/2 Service Zipkin Service Rate Limiting Service Metrics Integration
  13. 17.

    Directory Service Central Dogma > Repository service for textual configurations

    • Highly available, version controlled, advanced query mechanism > Stores start-time/run-time configuration safely > https://github.com/line/centraldogma/ JSON, YAML, XML, JS…
  14. 18.

    Directory Service Central Dogma TKSVR123 Find talk-server-release servers TKSVR124 ...

    [{hostname:TKSVR123, port: 8080}, {hostname:TKSVR124, port: 8080,},... ] Armeria Client http://group:talk-server-relase/api group:talk-server-relase → TKSVR123:8080
  15. 19.

    Routing - LEGY DataCenter 1 Main DataCenter App Server LEGY

    DataCenter 2 LEGY LEGY App Server > LINE Event Delivery Gateway – API Gateway written in Erlang • SPDY (& HTTP/2) • Global POP • Routing DSL
  16. 20.

    Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
  17. 21.

    Routing - LEGY > LINE Event Delivery Gateway – API

    Gateway written in Erlang • Routing DSL legy_dispatcher public_dispatcher ( CLIENT_TYPE = #{ <<"ANDROID">> => 'android.talk-server', <<"IOS">> => 'iphone.talk-server’, ... }. cf_select_upstream_by_client CLIENT_TYPE. cf_url_handler "^/(?P<tag>[A-Z]+)[^¥.]*$" #{ <<"THEME">> => 'theme.shop-server', <<"MESSAGE">> => default, }) LEGY /talk-server-BETA {"android": [.., "SVR01.jp", ..], “iphone”: [.., ”SVR02.jp“, ..], ...} /instance.json /talk-server-RELEASE SVR01.jp Android /MESSAGE req iOS /MESSAGE req SVR02.jp public_legy_release.conf
  18. 22.

    talk-server and stickershop example Case study > Making sticker ownership

    validation feature as microservices > Service redesign project between talk-server & Sticker Shop at 2017 • Rearranges service boundary
  19. 23.

    Motivation for reconstruction > Increase the number of stickers sold

    due to Creators Market launch > Increasing number of users who have used LINE since long times ago • Some users have 20,000 stickers. # of user # of owned stickers
  20. 24.

    How to send a sticker at 2017 > Save sticker

    ownership data to Redis for each user as a Set • Refresh all user ownership when a user purchase a sticker > Check Redis data when a user sends a sticker Stickershop LEGY talk-server Redis
  21. 25.

    What happens if services are not separated > Talk-server stops

    every time a mass-sticker owned user buy a sticker LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  22. 26.

    What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  23. 27.

    What happens if services are not separated > In worst

    case, whole messaging platform stop due to a single user LEGY talk-server VoIP Image/Video Delivery Auth LINE@/Bot backend Service Gateway Stickershop
  24. 28.

    How to solve this? > Separate responsibilities and services not

    to block each development > Make feature fast-fail-able not to affect each other • Circuit breaker + automatic request retry Stickershop LEGY talk-server Sticker Capability service Sticker Capability service
  25. 29.

    Retrospective > Rebuilt with microservices architecture • without degrading service

    quality thanks to our weapons > Require a mechanism that can flexibly respond to • Scale • Speed
  26. 30.

    Our Futures > Make our platform more fine-grained microservices Feature

    Feature Feature To: separating by Function Function Function Function Function Function Function Function Function Function Function Function Function From: separating by Feature
  27. 31.

    Wrap up of this talk > We adopt microservices architecture

    to speed up our development > Prepared our weapons for microservices architecture • Armeria, CentralDogma, LEGY > We’re continuing to transform our service to microservices architecture