LINE Messaging: From Active-Standby to Active-Active Multi-DC Architecture

To further enhance business continuity, the LINE Messaging service is embarking on a transition toward a multi-datacenter Active-Active architecture. Implementing this architecture at LINE's immense scale and high-traffic volume presents unique and significant challenges for databases and distributed systems alike.

In this session, we will explore our journey through this architectural evolution. We will first discuss the current status and limitations of our existing Disaster Recovery (DR) framework, followed by the insights and reasoning behind adopting a multi-datacenter Active-Active setup. We will also share the technical hurdles we faced along the way and our vision for future challenges in keeping one of the world's largest messaging platforms resilient.

Additionally, we will introduce YugabyteDB as a key technology enabling this transition, detailing how we selected and integrate it into our messaging backend.

Transcript

1. 2026.06.29 LY Corporation LINE Messaging: From Active-Standby to Active-Active Multi-DC Architecture Javier Luca de Tena | LINE Messaging Dev SBU Tsuruhara Tomu   | LINE Messaging Dev SBU
2. LINE Messaging at a Glance Current Architecture: Active-Standby & Why We Need to Change The Challenge of Going Active-Active and solutions Why do we need a new DB to achieve Active-Active multi-DC? Case Study: DB selection for LINE Messenger Backend 01 03 02 04 05 Agenda 1st Part (Javier Luca de Tena) 2nd Part (Tsuruhara Tomu)
3. One of the largest messaging platforms in Asia LINE Messaging at a Glance Presence & Services Available across Japan, Taiwan, Thailand, Indonesia and more Text, voice, video calls, stickers, ... Mission Criticality Critical communication infrastructure. Reliability and speed for real time communication is paramount. LINE Messaging serves hundred millions of users globally and processes billions of interactions every day.
4. Simplified architecture overview How LINE Messaging Works Client App API Gateway Messaging Backend Valkey API Gateway Gateway managing millions of persistent connections for real-time push delivery. Messaging Backend Core application server ̶ message routing, delivery, synchronization. HBase Persistent NoSQL storage for user data, messages, and operations. Valkey (Redis fork) Caching in-memory layer for frequently accessed data. Apache HBase
5. Simplified architecture overview How LINE Messaging Works Client App API Gateway Messaging Backend API Gateway Gateway managing millions of persistent connections for real-time push delivery. Messaging Backend Core application server ̶ message routing, delivery, synchronization. HBase Persistent NoSQL storage for user data, messages, and operations. Valkey (Redis fork) Caching in-memory layer for frequently accessed data. Valkey Apache HBase
6. Current Architecture: Active-Standby & Why We Need to Change
7. Current Architecture: Active-Standby Disaster Recovery (DR) • One Datacenter handles all production traffic • Second Datacenter is a standby replica ̶ ready for failover • Data replicated asynchronously from Active → Standby Active Region Datacenter ★ Serves 100% traffic Async Replication Standby Region Datacenter ☆ Idle ̶ Waiting for disaster
8. Why Active-Standby Falls Short  Reliability Challenging to test with real traffic/conditions as it is Standby.  Downtime Failover significantly affects UX  Waste Standby infrastructure is mostly idle  Sustainability Maintenance cost grows quadratically. Real-world Case Study During a DR drill, we found a core feature had been silently broken in standby for weeks. Nobody noticed ̶ standby never receives real traffic.
9. The Goal  Regional Resilience All core features must work during any regional outage.  Zero Downtime Minimize service downtime close to zero. The Active-Active Advantage • Continuous Validation: Real production traffic validates all features across regions. • Seamless Failover: Instant transition with close to zero service interruption. But itʼs not easy...
10. What if We Simply "Go Active-Active"? Region A Region B e.g.: 20 ms round trip latency vs 1~2ms round trip latency in the same region
11. Challenge 1: Stale Data Across Regions With asynchronous replication, data across DCs can be inconsistent Region A Data: X User 1 (Write: X) asynchronous replication (not yet completed) Region B Data: Y User 2 (Read: Y) STALE! Write Region A → replication lag → Read Region B = old data.
12. Our Approach: Region-Partitioned Active-Active Region A  Active for - User Profile A - Group Chat X Region B  Active for - User Profile B - Group Chat Y Backup Region Storage ONLY (consensus + durability) Data replicated across regions for resiliency
13. Region B  Active for - User Profile B - Group Chat Y Backup Region Storage ONLY (consensus + durability) Data replicated across regions for resiliency Region A Active for ALL data Our Approach: Region-Partitioned Active-Active
14. Region A  Active for Country A data Region B  Active for Country B data Backup Region Storage ONLY (consensus + durability) Data replicated across regions for resiliency Our Approach: Region-Partitioned Active-Active Partition by communication patterns → fewer cross-region hops in the common case
15. How do we route to the correct region?
16. Region Lookup Manager (RLM) An active-active component deployed in every Region ̶ consulted locally Service Storage Layer Cache Persistence 1) Where is Data X 2) Response: - Primary: Region-A Service Storage Layer Cache Persistence Region A Region B RLM RLM 1) Where is Data X 2) Response: - Primary: Region-A
17. Stores primary/secondary region mapping for every piece of data Active-Active architecture Region Lookup Manager: Internal Architecture Region-A Region-B No service deployed Service Service Region-C Cache Cache Consensus Persistence Layer Consensus based Source of Truth (YugabyteDB) RLM RLM RLM Consensus based Source of Truth (YugabyteDB) Consensus based Source of Truth (YugabyteDB)
18. Challenge 2: Latency Amplification A single API call triggers multiple sequential storage accesses VISUALIZING THE SEQUENTIAL IO PENALTY IO 1 IO 2 IO 3 2ms + 2ms + 2ms = 6ms IO 1 IO 2 IO 3 20ms + 20ms + 20ms = 60ms! LATENCY COMPARISON BY SCENARIO Scenario Local DC Cross-DC 1 storage access ~2 ms ~20 ms 10 sequential accesses ~20 ms ~200 ms 20 sequential accesses ~40 ms ~400 ms 100 sequential accesses ~200 ms ~2000 ms THE IMPACT  Affects SLOs and UX.  Cost multiplies. Local Region Cross Region
19. IO Optimization: Our Approach AS-IS (Sequential) TO-BE (Batched + Parallel) IO 1 IO 2 IO 3 IO 4 IO 5 IO 1 IO 2 IO 3,4 (batch) IO 5 Total: 100ms (5 IO x 20ms) Total: ~40ms (60% reduction) Goal: users do not notice any UX degradation User data Client data Settings data read 1 4 round trips Cross-DC: 4 × 20ms = 80ms Merged Model (combined) read 1 1 round trip Cross-DC: 1 × 20ms = 20ms read 2 read 3 Related data read 4
20. IO Optimization: Our Approach Group chat send-message response time (SLI) over time
21. Valkey Cross-Region replication is not reliable Replication (Persistence only) X Challenge 3: Cold Caches Network latency → Replication Lag → Valkey Full Sync → Network saturation → More Full Syncs (Cascading failure) → Full Sync is resource-intensive → Service requests impacted Region A Data X (Primary) • Cache: WARM Data Y • Cache: COLD Region B Data X • Cache: COLD Data Y (Primary) • Cache: WARM NO Valkey cross-region replication
22. Region A Data X (Primary) • Cache: WARM Data Y • Cache: COLD • Persistence Data (replicated) Replication (Persistence only) X Challenge 3: Cold Caches Region B Data X • Cache: COLD Data Y (Primary) • Cache: WARM NO Valkey cross-region replication
23. Region A Data X (Primary) • Cache: WARM Data Y • Cache: COLD • Persistence Data (replicated) Replication (Persistence only) X Challenge 3: Cold Caches Region B Data X • Cache: COLD Data Y (Primary) • Cache: WARM NO Valkey cross-region replication
24. Region A Data X (Primary) • Cache: WARM Data Y • Cache: WARM Region B Data X (Primary) • Cache: WARM Data Y • Cache: WARM Replication (Persistence only) Challenge 3: Cold Caches Continuously WARM UP caches Nearly instant failover
25. Cross-Region Cache Warmup: How It Works Bidirectional continuous cache warmup Region A Region B Service Warmup Consumer Cache 1) Caching algorithm transaction 2) Cache commit result 3) Consume Persistence Cache Service 4) Write (warmup) Commit TxId > Last TxId Persistence Kafka topic Mutable data, read-through-cache ath (user info, settings, social graph...)
26. Cross-Region Cache Warmup: How It Works Bidirectional continuous cache warmup Region A Region B Service Warmup Consumer Cache 2) Consume Persistence Cache Service 3) Write (warmup) Commit TxId > Last TxId? Persistence Kafka topic 1) Change Data Capture (CDC) Write Ahead Log Immutable data, write-through-cache path (original message records, received receipts, etc...)
27. Caching Algorithm A Distributed locks Storage Layer (Valkey + HBase) Service (Messaging Backend) HA Failover Caching Algorithm B Data filtering Secondary indexes building Challenge 4: Complexity
28. Caching Algorithm A Storage Layer (Valkey + HBase + Consensus Based Store) Service (Messaging Backend) Distributed locks Data filtering Secondary indexes building Data Regional complexity + RLM + Cross-Regional cache warmup Caching Algorithm B HA Failover Challenge 4: Complexity
29. The Solution: A Storage Abstraction Layer Hide complexity behind a simple, unified API Transactions, Locks & Indexes Region Routing (RLM) Storage Abstraction Layer Caching Algorithms (Lease/Invalidation/etc) Cross-Region Cache Warmup Failover & Fallback Storage Layer (Valkey + HBase + Consensus Based Store) Service (Messaging Backend) RLM queries Region Routing (RLM) Transactions & Indexes
30. What is missing? What is missing? Recap: ✓ Region-Partitioned Active-Active Consistency guaranteed ✓ IO Optimization 60% latency reduction ✓ Cross-Region Cache Warmup Instant failover, no cold start ✓ Storage Abstraction Layer One API, all complexity hidden  Valkey and HBase were not designed for native multi-region consistency ̶ and we found their limits. Need a complementary layer: A multi-region reliable source of truth
31. YugabyteDB Selection, Adoption, and Challenges Tsuruhara Tomu LY Corporation
32. Why do we need a new DB to achieve Active-Active multi-DC? • Isn't replication enough? Case Study: DB selection for LINE Messenger Backend • What is YugabyteDB? • Wouldn't TiDB be a better fit? 01 02 Key Takeaways for the Second Half
33. Why New Database?
34. Appears as a single database regardless of access region The latest written data can be read consistently from any region No data loss and minimal performance degradation even if a region fails Highly Available 01 03 02 04 The Ideal Database for Active-Active multi-Region A Multi-Region Reliable Source of Truth
35. HBase Replication is essential to prevent data loss Region1 Region2 HBase Replication
36. Async Replication is not resilient to data loss HBase Region1 Region2 HBase Async Replication • Only the DB in Region 1 has the latest data • Data loss occurs if Region 1 goes down due to Async Replication
37. Sync Replication compromises High Availablity HBase Region1 Region2 HBase Sync Replication • Write failure occurs if the data cannot be written to the DB in Region 2 • Failure of either Region results in a total system outage
38. Automatic Failure Detection for High Availability HBase Region1 Region2 HBase Sync Replication • Automatically stop replication upon failure 👑 Primary
39. Automatic Failure Detection is unfeasible in 2-region deployments HBase Region1 Region2 HBase Sync Replication • Network partitions lead each to assume the other is down • Split Brain X 👑 Primary 👑 Primary
40. Automatic Failure Detection with 3 Regions DB Region1 Region2 DB Sync Replication Region3 Down Detector Down Detector Down Detector e.g. etcd / zookeeper
41. Automatic Failure Detection with 3 Regions HBase Region1 Region2 HBase Sync Replication • No more Split-Brain • However, inter-region network latency between Region 1 and Region 2 continues to impact all write operations • What happens when network latency increases? • The failback process after a failover is complex when we make it Active-Active Region3 Down Detector Down Detector Down Detector e.g. etcd / zookeeper
42. A: Active-Active multi-DC is Hard to achieve with Leader-Follower replication and we need more sophisticated algorithm Q: Why do we need a new DB to achieve Active-Active multi-DC?
43. Cassandraʼs approach, Quorum Write + Quorum Read Node1 Region1 Region２ Node2 Node3 Region3 Coordinator ✉ ✉ ✉
44. Cassandraʼs approach, Quorum Write + Quorum Read Node1 Region1 Region２ Node2 Node3 Region3 Coordinator ✉ ✉ ✉ • Works well when all nodes are healthy • Ultimately, however, it provides only eventual consistency and cannot serve as the source of truth across regions ◦ Example: if a write fails midway, a subsequent READ may return either the new value or the old one
45. How can we achieve highly available and fault tolerant replication?
46. State Machine Replication Raft (multi) Paxos Viewstamped Replication State Machine Replication Consensus Based Approach
47. Spanner / Cloud Spanner (Google) CockroachDB YugabyteDB TiDB 01 03 02 04 Distributed Databases Implementing Paxos / Raft
48. TiDB & YugabyteDB Architecture Overview TiDB YugabyteDB
49. Sharding Primary Key … col N Primary Key … col N Tablet #1 Tablet #2 Tablet #3
50. 3 Replication using Raft Region1 Region3 tablet3 (Follower) tablet2 (Follower) tablet1 (Leader) 3 tablet3 (Follower) tablet2 (Leader) tablet1 (Follower) 3 Region2 tablet3 (Leader) tablet2 (Follower) tablet1 (Follower) ✉ ✉ ✉
51. TiDB & YugabyteDB TiDB & YugabyteDB TiDB YugabyteDB Replication Protocol Raft Raft SQL Compatibility MySQL compatible PostgreSQL compatible Timestamp Allocation Centralized (Timestamp Oracle) Decentralized (Hybrid Logical Clock) Implementation Go C++
52. Timestamp Allocation Transaction 1 Transaction 2 Transaction 3 Time Client1 Client2 Client3 Transaction 1 Transaction 2 Transaction 3 Ordering t = 100 t = 200 t = 300
53. TiDB: Timestamp Oracle Timestamp Oracle (a.k.a. PD Leader) TiDB node1 TiDB node2 TiDB node3
54. YugabyteDB: Hybrid Logical Clock YugabyteDB node1 YugabyteDB node2 Time (1782202002, 0) (1782202002, 1) (1782202002, 2) RPC Send
55. Feature Set Resiliency Performance (Latency + Throughput) 01 03 02 Selection Criteria
56. Essential requirement not to violate the SLO of the Messenger service Two types of evaluation methods • Benchmark tool (YCSB) • Production representative workload replay (Replayer) 01 02 Performance Evaluation
57. Testing Environment
58. READ Query Latency SELECT * FROM sample_table1 WHERE id = ? YugabyteDB TiDB Lower is better
59. WRITE Query Latency UPDATE sample_table1 SET col2 = ?, col3 = ?, col4 = ? .... WHERE id = ? YugabyteDB TiDB Lower is better
60. Performance strengths and weaknesses vary by table TiDB is often faster for write operations in our benchmark The difference is particularly significant for updates on tables with secondary indexes 01 03 02 Performance Differences Between TiDB and YugabyteDB
61. TiDB: Timestamp Allocation is done in the Central Component Cross Region Access
62. TiDB: Access Cost to Central Management Component Latency increased after the PD Leader switchover Lower is better
63. Latency increased after the PD Leader switchover TiDB: Access Cost to Central Management Component Lower is better
64. Performance Metrics Summary Median Latency (Replayer) DB WRITE READ YugabyteDB 40.9 - 144 ms 1.44 - 2.58 ms TiDB 35.2 - 89.6ms 1.56 - 52.5ms Throughput (YCSB) DB Throughput (ops/sec) YugabyteDB 90.6K - 141.8K TiDB 76.9K - 162.7K
65. Resiliency
66. Conculusion TiDB YugabyteDB Feature Set ◎ ⚪ Resiliency ⚪ ◎ Performance Strong on writes Strong on reads
67. YugabyteDB
68. Summary Why do we need a new DB to achieve Active-Active multi-DC? • Isn't replication enough? => No. Simple replication does not help support High Availability and Consistency simultaneously especially in multi-DC setup Case Study: DB selection for LINE Messenger Backend • What is YugabyteDB? => PostgreSQL compatible distributed RDBMS, Raft based replication • Wouldn't TiDB be a better fit? => TiDB's centralized timestamp allocation incurs a significant network round trip penalty in our multi-Region environment 01 02
69. Active-Active Multi-DC Goal HBase covers • Write-heavy workload • Consistency-insensitive data YugabyteDB covers • Consistency-critical data • Region Lookup Manager