No content

Agenda • What is the MessagingHub? • Messaging as a Service • Architecture • Conditions for reliability • Wrap-up

What is the MessagingHub? • MessagingHub is... • Functions • Born Background • Where is being used it? • Summary Indicators

MessagingHub is... An Independent and Universal All-in-one Messaging Platform A mechanism that operates on its own without being affected by other systems. A mechanism that can be used universally without any special difference anywhere. Meaning to take the necessary components within the MessagingHub project. I will focus on messaging. In order to meet the above requirements, it must be a platform.

MessagingHub is... Client Server MessagingHub Client Client Client Send a message from the server to the client

MessagingHub is... Client Server MessagingHub Client Client Client Send a message between clients

Functions Just like making a creation out of a toolbox. For example, there is a toolbox. Server Push App Push Email SMS Chatting

Functions Just like making a creation out of a toolbox. For example, there is a toolbox. Server Push App Push Email SMS Chatting

! Packets wasted System overload System Down Unnecessary traffic A fundamental structural improvement is needed! Born Background Client Server Periodically Polling DB In a situation where most polling responses result in a meaningless call "no change", What if the number of clients increases sharply? " # $ ☠

Push Trigger Event Client Server DB ! Packets wasted System overload System Down Unnecessary traffic Polling Push MessagingHub Born Background

Complexity How to pay off technical debt Productivity Improve little by little with maintenance. ᶄ ᶅ ᶃ Completely new from scratch. Turn it into a fundamental technology.

For universal use, Delegate roles easily through loose coupling. For operating independently, No business domain knowledge. Turn it into a fundamental technology For delivering the message, Send messages from source to destination.

Where is being used it? Delivery Merchant CS Retail User The fundamental tech. for Demae-can, the No.1 food delivery service in JPN. MessagingHub

Summary Indicators Send Message 1.5M+ Server Push Processing Time 150ms Daily Concurrent Peak Connection 50K+

M a a S • The reason to be a platform • Conditions to become a platform essaging s ervice

The reason to be a platform Maintenance Communication Stability System Fault Legacy Debt Error Common Case Duplicate Needs Spaghetti Code Product Cross Concern Difficult Handling Problem Domains

The reason to be a platform Common Case Duplicate Needs Spaghetti Code Product Cross Concern Difficult Handling Problem Domains Maintenance Communication Stability System Fault Legacy Debt Error

The reason to be a platform Maintenance Communication Stability System Fault Legacy Debt Error Common Case Duplicate Needs Spaghetti Code Product Cross Concern Difficult Handling Problem Domains

The reason to be a platform Maintenance Communication Stability System Fault Legacy Debt Error Product Cross Concern Difficult Handling Problem Domains Common Case Duplicate Needs Spaghetti Code

The reason to be a platform MessagingHub It must be a platform It doesn't have to be a platform For specific domain N : 1 1 : 1

The reason to be a platform MessagingHub It must be a platform It doesn't have to be a platform For specific domain N : 1 1 : 1

The reason to be a platform A) Delegate! Focus on your business MessagingHub Q) Do you need a messaging?

Can & Can't Conditions to become a platform Can do It is focused on messaging O Send to message from source to destination. O Send to various notification channels. O Communicate with clients. O We are tracking sending messages history. O Messages sending strategy. Can't do Do not engage in domain knowledge X It is not dependent on a specific domain. X It should not know the detailed business logic of the specific domain.

I don't know who you are. Conditions to become a platform MessagingHub 1. Request Connection Token 5. Response Connection Token 6. CONNECT with Connection Token 2. Request Token 4. Response Token 3. Token Generating Client Server

I don't know who you are. Conditions to become a platform The user id is known to the external-system. A type defined by messaging-hub. It is an array for sending the same message to multiple targets. It is like a message type or action. It should be "stringify JSON". MessagingHub doesn't care about payload and bypasses it. If the app-client is not connected to the WebSocket, it sends the app-push (a.k.a. FCM) with this data. If you want to turn off the app-push, you can turn it off. This is a message-sending strategy parameter. If the message-router does not consume the sent message to MessagingHub within the specified time, it is ignored.

Conditions to become a platform I don't know who you are. Any Server Any Client Because it is independent and universal. But I don't care.

Architecture • Overview • Modules • Flow • Generalization • Infra

$POTVM 4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ Overview message-router connection-manager chat-app ack-receiver message-app operation-app consumer-proxy notification-app operation redis ack message notification &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server Server ,BGLB chat

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM message-router message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat message-router Server redis message sourcing and routing It receives messages from an external interworking system and appropriately routes them according to business requirements. ,BGLB

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM connection-manager message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat Client connection-manager redis connection management and server push It manages WebSocket connections connected to clients and handles server pushes. ,BGLB Server

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM notification-app message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat various third-party notification channel It delivers messages to the third-party notification channel, except for server push. 4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ notification Server notification-app

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM message-app message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat message data management It manages the message data sent by the interworking server to the client. Management items also include information on the delivery results. message-app message Server

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM ack-receiver message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat message acknowledgment receiver The purpose of the ack-receiver is to receive a message reception callback from the client to know whether the message has been safely delivered to the client. ack-receiver ack Client

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM consumer-proxy message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat Kafka consumer-client layer It was added to separate Kafka's consumer clients into a separate layer. Benefits - 1) Resolving consumer rebalancing issues. 2) Eliminate the limitations of scale-out. ,BGLB consumer-proxy

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM chat-app message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat chatting business logic It processes the chat business logic and manages the data. chat-app chat

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM operation-app message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat operation It is a role for registering and processing schedulers related to service operations. operation-app operation ,BGLB

4.4 '$. &NBJM /PUJpDBUJPOT ŋŋŋ consumer-proxy notification-app notification Server $POTVM third-party solutions message-router connection-manager chat-app ack-receiver message-app operation-app operation redis ack message &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE Client Server Server ,BGLB chat Monitoring & Alert & etc $POTVM &MBTUJDTFBSDI (SBGBOB 1SPNFUIFVT , ,JCBOB BENJOJTUSBUPS *OqVY%# 'MVFOUE

Libraries MessagingHub Modules common core messaging-shared external-lib Avro IDL Protocol Buffers IDL Specific for messaging Boilerplate External System Protocol Buffers IDL for sharing

Libraries Controller Aspect Common Response Format Minimum Support Version Custom Header Logging Factory Log Annotation Security API MySQL with JPA Kafka with Avro Redis gRPC Builder Common Log Format Log Filter for security Append Custom Log Default Common Log Setup TimeChecker Logging AtomicTask InternalAPI Security Config Token Generation Token Validation Metrics Micrometer Consul Watcher Common ErrorCode Environment Payload Utility Config RestClient Retry DateTime Mapper SystemInfo Yaml Loader RestTemplate ThreadPool ObjectMapper

Message Flow message-router connection-manager ack-receiver notification-app message-app 1. send message(sourcing) 3. message uuid issued 7.1 push message by socket 8. message acknowledgement 7.2. send message by notification channels 5.1 route message 5.3 produce message if client disconnected 4. consume 5.2 route message 2. produce 6. consume update message sending status check the message with the message uuid if desired Client Server ,BGLB

Message Flow message-router connection-manager ack-receiver notification-app message-app 1. send message(sourcing) 3. message uuid issued 7.1 push message by socket 8. message acknowledgement 7.2. send message by notification channels 5.1 route message 5.3 produce message if client disconnected 4. consume 5.2 route message 2. produce 6. consume update message sending status check the message with the message uuid if desired Client Server ,BGLB message-router 3. message uuid issued 5.1 route message 4. consume 5.2 route message 2. produce Server 1. send message(sourcing) 6. consume ,BGLB

Message Flow message-router connection-manager ack-receiver notification-app message-app 1. send message(sourcing) 3. message uuid issued 7.1 push message by socket 8. message acknowledgement 7.2. send message by notification channels 5.1 route message 5.3 produce message if client disconnected 4. consume 5.2 route message 2. produce 6. consume update message sending status check the message with the message uuid if desired Client Server ,BGLB connection-manager notification-app 7.1 push message by socket 5.3 produce message if client disconnected Client ,BGLB 7.2. send message by notification channels 6. consume

Message Flow message-router connection-manager ack-receiver notification-app message-app 1. send message(sourcing) 3. message uuid issued 7.1 push message by socket 8. message acknowledgement 7.2. send message by notification channels 5.1 route message 5.3 produce message if client disconnected 4. consume 5.2 route message 2. produce 6. consume update message sending status check the message with the message uuid if desired Client Server ,BGLB ack-receiver message-app 8. message acknowledgement check the message with the message uuid if desired Client Server update message sending status

Notification Flow FCM SMS Email Another Notifications ŋŋŋ message notification message-router notification-app message-app message sourcing Server ,BGLB necessary information

Chat Flow chat-app chat Client A Client B connection-manager

Specific Needs in Food Delivery Think about chatting between drivers and users Who makes the chat room? How to automatically delete chat rooms? How do you allow input to be selected?

Chat Flow message-router chat-app notification-app chat $POTVM Server Client A Client B ,BGLB connection-manager

Chat Usecase Specific Usecase Common Usecase SYSTEM_UPDATE_CHAT_USER SYSTEM_CREATE_DIRECT_CHAT_ROOM SYSTEM_SEND_CHAT_LOG SYSTEM_DISABLE_CHAT_ROOM SYSTEM_DELETE_CHAT_ROOM UPDATE_CHAT_USER CONNECT GET_CHAT_ROOMS MARK_CHAT_READ HAS_DIRECT_CHAT_ROOM CREATE_DIRECT_CHAT_ROOM GET_CHAT_ROOM GET_CHAT_LOGS SEND_CHAT_LOG RECEIVE_CHAT_READ RECEIVE_CHAT_LOG MARK_CHAT_READ RECEIVE_CHAT_READ SET_CHAT_ROOM_NAME SET_CHAT_ROOM_PUSH SET_CHAT_ROOM_IMAGE DELETE_CHAT_ROOM GET_CHAT_MEMBER GET_CHAT_MEMBERS GET_CHAT_LOG_MANAGED_GROUP_NAMES GET_CHAT_LOG_MANAGED_GROUP Receiver has new setting delete member chat log Generalization

Generalization Specific Usecase Common Usecase SYSTEM_UPDATE_CHAT_USER SYSTEM_CREATE_DIRECT_CHAT_ROOM SYSTEM_SEND_CHAT_LOG SYSTEM_DISABLE_CHAT_ROOM SYSTEM_DELETE_CHAT_ROOM UPDATE_CHAT_USER CONNECT GET_CHAT_ROOMS MARK_CHAT_READ HAS_DIRECT_CHAT_ROOM CREATE_DIRECT_CHAT_ROOM GET_CHAT_ROOM GET_CHAT_LOGS SEND_CHAT_LOG RECEIVE_CHAT_READ RECEIVE_CHAT_LOG MARK_CHAT_READ RECEIVE_CHAT_READ SET_CHAT_ROOM_NAME SET_CHAT_ROOM_PUSH SET_CHAT_ROOM_IMAGE DELETE_CHAT_ROOM GET_CHAT_MEMBER GET_CHAT_MEMBERS GET_CHAT_LOG_MANAGED_GROUP_NAMES GET_CHAT_LOG_MANAGED_GROUP Receiver has new setting delete member chat log Generalization Generalization is a way to extend universality without compromising independence.

Infra

Conditions for reliability • Connection Management • Message Acknowledgement • Connection Confirmation • How to get missing messages?

Connection Management internal socket server internal socket client message sourcing message routing Host event operation-app message-router Storage connection-manager bootstrap Initializer handler codec messaging-shared connection meta internal connection scheduler channel kafka bootstrap Initializer handler codec messaging-shared channel internal connection connection meta kafka scheduler bootstrap Initializer handler codec Client ,BGLB websocket server

Connection Management internal socket server internal socket client message sourcing message routing Host event operation-app message-router Storage connection-manager bootstrap Initializer handler codec messaging-shared connection meta internal connection scheduler channel kafka bootstrap Initializer handler codec messaging-shared channel internal connection connection meta kafka scheduler bootstrap Initializer handler codec Client ,BGLB websocket server message-router Storage connection-manager connection meta connection meta websocket server internal socket server internal socket client

Connection Management internal socket server internal socket client message sourcing message routing Host event operation-app message-router Storage connection-manager bootstrap Initializer handler codec messaging-shared connection meta internal connection scheduler channel kafka bootstrap Initializer handler codec messaging-shared channel internal connection connection meta kafka scheduler bootstrap Initializer handler codec Client ,BGLB websocket server internal socket server internal socket client Host event operation-app message-router connection-manager connection meta connection meta ,BGLB Storage

Include ack authorization Include ack authorization Message Acknowledgement notification-app connection-manager chat-app ack-receiver messaging-shared MessagingHub ack Client App Push Chat Server Push

Connection Confirmation MessagingHub External System Send OK, But NO. ᶄ ᶅ ᶆ ᶇ ᶈ Network Client ᶃ

Connection Confirmation MessagingHub PING PONG_ACK PONG ᶃ Periodically send a ping. ᶄ Receive Pong. If there is no response within a specific time, the connection is terminated. If the client does not receive the server PONG_ACK within a specific time, it is good to try to reconnect. ᶅ Client

Client Still, there must be a way to send it NO connection OFF app push OFFLINE temporarily A push message can also be sent when the user agent is temporarily offline. In support of this, the push service stores messages for the user agent until the user agent becomes available. Q. How to get missing messages? A.

How to get missing messages? Client message-app RECENT_MESSAGES message-router connection-manager notification-app message chat-app chat

Wrap-up • Next Plan • Tech Blog

Next Plan Stability Optimizing Advancement Observability Efficiency More Simple, More Powerful! Widen the usage point Reliability &

Tech Blog Until you take the first steps to pay off your technical debt Take a look at the messaging system (a.k.a MessagingHub) MessagingHub troubleshooting https://engineering.linecorp.com/ko/blog/about-messaging-hub-1 https://engineering.linecorp.com/ko/blog/about-messaging-hub-2 https://engineering.linecorp.com/ko/blog/messaing-hub-troubleshooting

՘ऴΘΓ The End