Slide 1
Slide 1 text
2019 DevDay
Long Road to Microservices
Architecture at LINE Messaging
Platform.
> Masahiro Ide
> LINE Z Part Team
Slide 2
Slide 2 text
Agenda
> How did we scale our services
> Weapons we prepared for our service
microservices
> How do we growth our architecture in future
Slide 3
Slide 3 text
When we launch LINE service
LINE Messaging Platform (2011)
> A Monolithic Service
• Everything are handled at talk-server
> Team
• 1 team, 6 members
LEGY
talk-server
Redis
MySQL
Server team
Slide 4
Slide 4 text
LINE Messaging Platform (2011~2019)
Social Graph
VoIP
Message
Sticker
Theme LINE@
Group Search
LINE NEWS
Recommendation
Game
Timeline
Image
Video
Payment
Wallet Clova
OpenChat
Social Login
Emoji
Settings
Slide 5
Slide 5 text
LINE Messaging Platform (2011~2019)
LEGY
talk-server
Server team
...
Slide 6
Slide 6 text
LINE Messaging Platform (2011~2019)
LEGY
talk-server
Server team
...
Ok, let’s focus on
Video share feature
Slide 7
Slide 7 text
LINE Messaging Platform (2011~2019)
LEGY
talk-server
Server team
...
Ok, let’s focus on
Video share feature
Oh, wait. I want to
focus on sticker
Slide 8
Slide 8 text
LINE Messaging Platform (2011~2019)
LEGY
talk-server
Server team
...
Ok, let’s focus on
Video share feature
Oh, wait. I want to
focus on sticker
Waaait!! VoIP has
highest priority!!
Slide 9
Slide 9 text
LINE Messaging Platform (2011~2019)
LEGY
talk-server
Server team
...
Image/Video
Delivery
Slide 10
Slide 10 text
LINE Messaging Platform (2011~2019)
LEGY
talk-server
...
Image/Video
Delivery
Stickershop
Want to make
more new type
of sticker!!
Let’s consider
next-gen protocol
Let’s improve
img/video quality
Let’s focus on
new messaging
feature
Slide 11
Slide 11 text
Now
LINE Messaging Platform (2019)
> Simple app server became a huge microservice architecture
LEGY talk-server
VoIP
Image/Video
Delivery
Auth
LINE@/Bot
backend
Service
Gateway
Stickershop
Slide 12
Slide 12 text
Now
LINE Messaging Platform (2019)
> Simple app server became a huge microservice architecture
LEGY talk-server
VoIP
Image/Video
Delivery
Auth
LINE@/Bot
backend
Service
Gateway
Stickershop
x8
x25
x10
x31
x6
x9
Service x1
Service >20
Service x1
Service >40
Service >20
Service x7
Slide 13
Slide 13 text
Microservice architecture solves everything?
> Good !
• Conflicts between features ↓
• Development speed ↑ ↑
> Not good "
• Network failure ↑
• Potential Cascading failure ↑
• Manage more service configs ↑
Slide 14
Slide 14 text
Concepts need to build
our microservices architecture
Armeria
Connectivity
LEGY
Routing
CentralDogma
Directory
Service
Slide 15
Slide 15 text
Connectivity Armeria - Our RPC layer
> Asynchronous RPC/REST library
• built on top of Java 8, Netty, HTTP/2, Thrift and gRPC
• https://github.com/line/armeria
> Take care common functionality for microservice
• Logging, Client side Load Balancing, Monitoring integration, Tracing
• Retry, Throttling, Circuit Breaker, Health Check
Slide 16
Slide 16 text
Structured Logging Client
Connectivity Armeria - Our RPC layer
gRPC/Thrift Client
HTTP/2 Client
Client side Load Balancer
Circuit Breaker
Metrics Integration
Structured Logging Service
gRPC/Thrift Service
HTTP/2 Service
Zipkin Service
Rate Limiting Service
Metrics Integration
Slide 17
Slide 17 text
Directory Service Central Dogma
> Repository service for textual configurations
• Highly available, version controlled, advanced query mechanism
> Stores start-time/run-time configuration safely
> https://github.com/line/centraldogma/
JSON, YAML, XML, JS…
Slide 18
Slide 18 text
Directory Service Central Dogma
TKSVR123
Find talk-server-release servers
TKSVR124
...
[{hostname:TKSVR123, port: 8080},
{hostname:TKSVR124, port: 8080,},... ]
Armeria
Client
http://group:talk-server-relase/api group:talk-server-relase
→ TKSVR123:8080
Slide 19
Slide 19 text
Routing - LEGY
DataCenter 1 Main DataCenter
App Server
LEGY
DataCenter 2
LEGY
LEGY
App Server
> LINE Event Delivery Gateway – API Gateway written in Erlang
• SPDY (& HTTP/2)
• Global POP
• Routing DSL
Slide 20
Slide 20 text
Routing - LEGY
> LINE Event Delivery Gateway – API Gateway written in Erlang
• Routing DSL
legy_dispatcher public_dispatcher (
CLIENT_TYPE = #{
<<"ANDROID">> => 'android.talk-server',
<<"IOS">> => 'iphone.talk-server’,
... }.
cf_select_upstream_by_client CLIENT_TYPE.
cf_url_handler
"^/(?P[A-Z]+)[^¥.]*$" #{
<<"THEME">> => 'theme.shop-server',
<<"MESSAGE">> => default,
})
LEGY
/talk-server-BETA
{"android": [.., "SVR01.jp", ..],
“iphone”: [.., ”SVR02.jp“, ..], ...}
/instance.json
/talk-server-RELEASE
SVR01.jp
Android /MESSAGE req
iOS /MESSAGE req
SVR02.jp
public_legy_release.conf
Slide 21
Slide 21 text
Routing - LEGY
> LINE Event Delivery Gateway – API Gateway written in Erlang
• Routing DSL
legy_dispatcher public_dispatcher (
CLIENT_TYPE = #{
<<"ANDROID">> => 'android.talk-server',
<<"IOS">> => 'iphone.talk-server’,
... }.
cf_select_upstream_by_client CLIENT_TYPE.
cf_url_handler
"^/(?P[A-Z]+)[^¥.]*$" #{
<<"THEME">> => 'theme.shop-server',
<<"MESSAGE">> => default,
})
LEGY
/talk-server-BETA
{"android": [.., "SVR01.jp", ..],
“iphone”: [.., ”SVR02.jp“, ..], ...}
/instance.json
/talk-server-RELEASE
SVR01.jp
Android /MESSAGE req
iOS /MESSAGE req
SVR02.jp
public_legy_release.conf
Slide 22
Slide 22 text
talk-server and stickershop example
Case study
> Making sticker ownership validation feature as microservices
> Service redesign project between talk-server & Sticker Shop at 2017
• Rearranges service boundary
Slide 23
Slide 23 text
Motivation for reconstruction
> Increase the number of stickers sold due to Creators Market launch
> Increasing number of users who have used LINE since long times ago
• Some users have 20,000 stickers.
# of user
# of owned stickers
Slide 24
Slide 24 text
How to send a sticker at 2017
> Save sticker ownership data to Redis for each user as a Set
• Refresh all user ownership when a user purchase a sticker
> Check Redis data when a user sends a sticker
Stickershop
LEGY talk-server
Redis
Slide 25
Slide 25 text
What happens if services are not separated
> Talk-server stops every time a mass-sticker owned user buy a sticker
LEGY talk-server
VoIP
Image/Video
Delivery
Auth
LINE@/Bot
backend
Service
Gateway
Stickershop
Slide 26
Slide 26 text
What happens if services are not separated
> In worst case, whole messaging platform stop due to a single user
LEGY talk-server
VoIP
Image/Video
Delivery
Auth
LINE@/Bot
backend
Service
Gateway
Stickershop
Slide 27
Slide 27 text
What happens if services are not separated
> In worst case, whole messaging platform stop due to a single user
LEGY talk-server
VoIP
Image/Video
Delivery
Auth
LINE@/Bot
backend
Service
Gateway
Stickershop
Slide 28
Slide 28 text
How to solve this?
> Separate responsibilities and services not to block each development
> Make feature fast-fail-able not to affect each other
• Circuit breaker + automatic request retry
Stickershop
LEGY talk-server
Sticker
Capability
service
Sticker
Capability
service
Slide 29
Slide 29 text
Retrospective
> Rebuilt with microservices architecture
• without degrading service quality thanks to our weapons
> Require a mechanism that can flexibly respond to
• Scale
• Speed
Slide 30
Slide 30 text
Our Futures
> Make our platform more fine-grained microservices
Feature
Feature
Feature
To: separating by Function
Function Function Function
Function
Function Function Function
Function
Function Function Function
Function
From: separating by Feature
Slide 31
Slide 31 text
Wrap up of this talk
> We adopt microservices architecture to speed up our development
> Prepared our weapons for microservices architecture
• Armeria, CentralDogma, LEGY
> We’re continuing to transform our service to microservices architecture