LINE's messaging server architecture and engineering underlying large scale and highly reliable service / LINE Campus Talk in UC Berkeley by Yuto Kawamura
LINE's messaging server architecture and engineering underlying large scale and highly reliable service / LINE Campus Talk in UC Berkeley by Yuto Kawamura
19.02.2019 Campus Talk in UC Berkeley
Presented by Yuto Kawamura
centers all over the world • Developed from scratch at LINE for messaging service • Written in Erlang - used widely in Telecom systems • Zero latency code hot swapping w/o closing client connections • Durability thanks to Erlang process and message passing
but volatile • Home-brew clustering • HBase - distributed key-value store based on Hadoop (HDFS) • Slower than Redis, but persistent • Cascading failure handling for consistency • Async write from background task processor • Data correction batch Primary/ Backup talk-server Cache/ Primary Dual write
easier use • Consulting (architecture, usage, requirements) for services that wants to use Kafka • Reliability engineering for Kafka clusters • Troubleshooting when performance violates Service Level Objectives (SLO) • Patch Kafka to fix bug/performance issue • Enhancements for achieving higher availability/performance
either: • 1. Really busy doing lots of work Many requests/ responses to read/write • 2. Blocked by some operations (which should not happen in event loop in general)
transfer topic data to client • sendfile(2) system call might involves reading disk inside • sendfile is system call for copying on-disk data to client socket directly (zero-copy transfer) for efficient data transfer
makes system ultimately reliable in the end • Understanding problem and fixing fundamentally requires understanding of all stacks involved to run the software • Application code - e.g, Kafka • Language runtime - e.g, JVM • Operating system - e.g, Linux
platform needs to be revised • More flexibility for more business integration • Higher availability • Multi iDC deployment • Make more “platforms” as shared building block for services