Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Long road to microservices architecture at LINE messaging platform.

Long road to microservices architecture at LINE messaging platform.

LINE DevDay 2019

November 20, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay
    Long Road to Microservices
    Architecture at LINE Messaging
    Platform.
    > Masahiro Ide
    > LINE Z Part Team

    View full-size slide

  2. Agenda
    > How did we scale our services
    > Weapons we prepared for our service
    microservices
    > How do we growth our architecture in future

    View full-size slide

  3. When we launch LINE service
    LINE Messaging Platform (2011)
    > A Monolithic Service
    • Everything are handled at talk-server
    > Team
    • 1 team, 6 members
    LEGY
    talk-server
    Redis
    MySQL
    Server team

    View full-size slide

  4. LINE Messaging Platform (2011~2019)
    Social Graph
    VoIP
    Message
    Sticker
    Theme LINE@
    Group Search
    LINE NEWS
    Recommendation
    Game
    Timeline
    Image
    Video
    Payment
    Wallet Clova
    OpenChat
    Social Login
    Emoji
    Settings

    View full-size slide

  5. LINE Messaging Platform (2011~2019)
    LEGY
    talk-server
    Server team
    ...

    View full-size slide

  6. LINE Messaging Platform (2011~2019)
    LEGY
    talk-server
    Server team
    ...
    Ok, let’s focus on
    Video share feature

    View full-size slide

  7. LINE Messaging Platform (2011~2019)
    LEGY
    talk-server
    Server team
    ...
    Ok, let’s focus on
    Video share feature
    Oh, wait. I want to
    focus on sticker

    View full-size slide

  8. LINE Messaging Platform (2011~2019)
    LEGY
    talk-server
    Server team
    ...
    Ok, let’s focus on
    Video share feature
    Oh, wait. I want to
    focus on sticker
    Waaait!! VoIP has
    highest priority!!

    View full-size slide

  9. LINE Messaging Platform (2011~2019)
    LEGY
    talk-server
    Server team
    ...
    Image/Video
    Delivery

    View full-size slide

  10. LINE Messaging Platform (2011~2019)
    LEGY
    talk-server
    ...
    Image/Video
    Delivery
    Stickershop
    Want to make
    more new type
    of sticker!!
    Let’s consider
    next-gen protocol
    Let’s improve
    img/video quality
    Let’s focus on
    new messaging
    feature

    View full-size slide

  11. Now
    LINE Messaging Platform (2019)
    > Simple app server became a huge microservice architecture
    LEGY talk-server
    VoIP
    Image/Video
    Delivery
    Auth
    LINE@/Bot
    backend
    Service
    Gateway
    Stickershop

    View full-size slide

  12. Now
    LINE Messaging Platform (2019)
    > Simple app server became a huge microservice architecture
    LEGY talk-server
    VoIP
    Image/Video
    Delivery
    Auth
    LINE@/Bot
    backend
    Service
    Gateway
    Stickershop
    x8
    x25
    x10
    x31
    x6
    x9
    Service x1
    Service >20
    Service x1
    Service >40
    Service >20
    Service x7

    View full-size slide

  13. Microservice architecture solves everything?
    > Good !
    • Conflicts between features ↓
    • Development speed ↑ ↑
    > Not good "
    • Network failure ↑
    • Potential Cascading failure ↑
    • Manage more service configs ↑

    View full-size slide

  14. Concepts need to build
    our microservices architecture
    Armeria
    Connectivity
    LEGY
    Routing
    CentralDogma
    Directory
    Service

    View full-size slide

  15. Connectivity Armeria - Our RPC layer
    > Asynchronous RPC/REST library
    • built on top of Java 8, Netty, HTTP/2, Thrift and gRPC
    • https://github.com/line/armeria
    > Take care common functionality for microservice
    • Logging, Client side Load Balancing, Monitoring integration, Tracing
    • Retry, Throttling, Circuit Breaker, Health Check

    View full-size slide

  16. Structured Logging Client
    Connectivity Armeria - Our RPC layer
    gRPC/Thrift Client
    HTTP/2 Client
    Client side Load Balancer
    Circuit Breaker
    Metrics Integration
    Structured Logging Service
    gRPC/Thrift Service
    HTTP/2 Service
    Zipkin Service
    Rate Limiting Service
    Metrics Integration

    View full-size slide

  17. Directory Service Central Dogma
    > Repository service for textual configurations
    • Highly available, version controlled, advanced query mechanism
    > Stores start-time/run-time configuration safely
    > https://github.com/line/centraldogma/
    JSON, YAML, XML, JS…

    View full-size slide

  18. Directory Service Central Dogma
    TKSVR123
    Find talk-server-release servers
    TKSVR124
    ...
    [{hostname:TKSVR123, port: 8080},
    {hostname:TKSVR124, port: 8080,},... ]
    Armeria
    Client
    http://group:talk-server-relase/api group:talk-server-relase
    → TKSVR123:8080

    View full-size slide

  19. Routing - LEGY
    DataCenter 1 Main DataCenter
    App Server
    LEGY
    DataCenter 2
    LEGY
    LEGY
    App Server
    > LINE Event Delivery Gateway – API Gateway written in Erlang
    • SPDY (& HTTP/2)
    • Global POP
    • Routing DSL

    View full-size slide

  20. Routing - LEGY
    > LINE Event Delivery Gateway – API Gateway written in Erlang
    • Routing DSL
    legy_dispatcher public_dispatcher (
    CLIENT_TYPE = #{
    <<"ANDROID">> => 'android.talk-server',
    <<"IOS">> => 'iphone.talk-server’,
    ... }.
    cf_select_upstream_by_client CLIENT_TYPE.
    cf_url_handler
    "^/(?P[A-Z]+)[^¥.]*$" #{
    <<"THEME">> => 'theme.shop-server',
    <<"MESSAGE">> => default,
    })
    LEGY
    /talk-server-BETA
    {"android": [.., "SVR01.jp", ..],
    “iphone”: [.., ”SVR02.jp“, ..], ...}
    /instance.json
    /talk-server-RELEASE
    SVR01.jp
    Android /MESSAGE req
    iOS /MESSAGE req
    SVR02.jp
    public_legy_release.conf

    View full-size slide

  21. Routing - LEGY
    > LINE Event Delivery Gateway – API Gateway written in Erlang
    • Routing DSL
    legy_dispatcher public_dispatcher (
    CLIENT_TYPE = #{
    <<"ANDROID">> => 'android.talk-server',
    <<"IOS">> => 'iphone.talk-server’,
    ... }.
    cf_select_upstream_by_client CLIENT_TYPE.
    cf_url_handler
    "^/(?P[A-Z]+)[^¥.]*$" #{
    <<"THEME">> => 'theme.shop-server',
    <<"MESSAGE">> => default,
    })
    LEGY
    /talk-server-BETA
    {"android": [.., "SVR01.jp", ..],
    “iphone”: [.., ”SVR02.jp“, ..], ...}
    /instance.json
    /talk-server-RELEASE
    SVR01.jp
    Android /MESSAGE req
    iOS /MESSAGE req
    SVR02.jp
    public_legy_release.conf

    View full-size slide

  22. talk-server and stickershop example
    Case study
    > Making sticker ownership validation feature as microservices
    > Service redesign project between talk-server & Sticker Shop at 2017
    • Rearranges service boundary

    View full-size slide

  23. Motivation for reconstruction
    > Increase the number of stickers sold due to Creators Market launch
    > Increasing number of users who have used LINE since long times ago
    • Some users have 20,000 stickers.
    # of user
    # of owned stickers

    View full-size slide

  24. How to send a sticker at 2017
    > Save sticker ownership data to Redis for each user as a Set
    • Refresh all user ownership when a user purchase a sticker
    > Check Redis data when a user sends a sticker
    Stickershop
    LEGY talk-server
    Redis

    View full-size slide

  25. What happens if services are not separated
    > Talk-server stops every time a mass-sticker owned user buy a sticker
    LEGY talk-server
    VoIP
    Image/Video
    Delivery
    Auth
    LINE@/Bot
    backend
    Service
    Gateway
    Stickershop

    View full-size slide

  26. What happens if services are not separated
    > In worst case, whole messaging platform stop due to a single user
    LEGY talk-server
    VoIP
    Image/Video
    Delivery
    Auth
    LINE@/Bot
    backend
    Service
    Gateway
    Stickershop

    View full-size slide

  27. What happens if services are not separated
    > In worst case, whole messaging platform stop due to a single user
    LEGY talk-server
    VoIP
    Image/Video
    Delivery
    Auth
    LINE@/Bot
    backend
    Service
    Gateway
    Stickershop

    View full-size slide

  28. How to solve this?
    > Separate responsibilities and services not to block each development
    > Make feature fast-fail-able not to affect each other
    • Circuit breaker + automatic request retry
    Stickershop
    LEGY talk-server
    Sticker
    Capability
    service
    Sticker
    Capability
    service

    View full-size slide

  29. Retrospective
    > Rebuilt with microservices architecture
    • without degrading service quality thanks to our weapons
    > Require a mechanism that can flexibly respond to
    • Scale
    • Speed

    View full-size slide

  30. Our Futures
    > Make our platform more fine-grained microservices
    Feature
    Feature
    Feature
    To: separating by Function
    Function Function Function
    Function
    Function Function Function
    Function
    Function Function Function
    Function
    From: separating by Feature

    View full-size slide

  31. Wrap up of this talk
    > We adopt microservices architecture to speed up our development
    > Prepared our weapons for microservices architecture
    • Armeria, CentralDogma, LEGY
    > We’re continuing to transform our service to microservices architecture

    View full-size slide