Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINE's messaging server architecture and engineering underlying large scale and highly reliable service / LINE Campus Talk in UC Berkeley by Yuto Kawamura

LINE's messaging server architecture and engineering underlying large scale and highly reliable service / LINE Campus Talk in UC Berkeley by Yuto Kawamura

19.02.2019 Campus Talk in UC Berkeley
Presented by Yuto Kawamura

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

February 19, 2019
Tweet

Transcript

  1. LINE's messaging server architecture and engineering underlying large scale and

    highly reliable service Yuto Kawamura LINE Corp
  2. About me • Yuto Kawamura • Senior software engineer •

    A LINE server developer • Apache Kafka contributor • Joined Apr, 2015
  3. Requirement for LINE server system • Fast delivery • Message

    must be delivered in nearly realtime • Reliable delivery • No message lost • Do the above for 5 billion messages /day from164 million users!
  4. Messaging System Architecture LINE Apps LEGY JP LEGY DE LEGY

    SG Thrift RPC/HTTP talk-server Distributed Data Store Distributed async task processing
  5. LEGY LINE Apps LEGY JP LEGY DE LEGY SG Thrift

    RPC/HTTP talk-server Distributed Data Store Distributed async task processing
  6. LEGY • API Gateway/Reverse Proxy • Deployed to many data

    centers all over the world • Developed from scratch at LINE for messaging service • Written in Erlang - used widely in Telecom systems • Zero latency code hot swapping w/o closing client connections • Durability thanks to Erlang process and message passing
  7. talk-server LINE Apps LEGY JP LEGY DE LEGY SG Thrift

    RPC/HTTP talk-server Distributed Data Store Distributed async task processing
  8. talk-server • Java based web application server • Implements most

    of business logics • Message delivery • Friendship management • User configuration • Java8 + Spring + Thrift RPC
  9. Data store LINE Apps LEGY JP LEGY DE LEGY SG

    Thrift RPC/HTTP talk-server Distributed Data Store Distributed async task processing
  10. Hybrid data store • Redis - in-memory DB • Fast,

    but volatile • Home-brew clustering • HBase - distributed key-value store based on Hadoop (HDFS) • Slower than Redis, but persistent • Cascading failure handling for consistency • Async write from background task processor • Data correction batch Primary/ Backup talk-server Cache/ Primary Dual write
  11. How realtime message delivery works LEGY LEGY talk-server Storage 1.

    Find nearest LEGY 2. sendMessage(“Bob”, “Hello!”) 3. Proxy request 4. Write to storage talk-server X. fetchOps() 6. Proxy request 7. Read message 8. Return fetchOps() with message 5. Notify message arrival Alice Bob
  12. Communication between internal systems • Communication for querying, transactional updates

    (needs response from peer) • Communication for data synchronization, update notification talk-server Auth Analytics Another Service HTTP/REST/RPC
  13. Apache Kafka • Distributed streaming platform • (narrow sense) Distributed

    persistent message queue which supports Pub-Sub model • Built-in load distribution • Built-in fail-over on both server(broker) and client
  14. How it works Producer Brokers Consumer Topic Topic Consumer Consumer

    Producer Produce Fetch
  15. Consumer GroupA Pub-Sub Brokers Consumer Topic Topic Consumer Consumer GroupB

    Consumer Consumer Records[A, B, C…] Records[A, B, C…] • Multiple consumer “groups” can independently consume a single topic
  16. Example: UserActivityEvent

  17. Traffic scale

  18. So what’s our engineering here? • Develop facilities/client libraries for

    easier use • Consulting (architecture, usage, requirements) for services that wants to use Kafka • Reliability engineering for Kafka clusters • Troubleshooting when performance violates Service Level Objectives (SLO) • Patch Kafka to fix bug/performance issue • Enhancements for achieving higher availability/performance
  19. Example of troubleshooting

  20. Detecting problem • 50x ~ 100x slower response time in

    99th %ile response time • Normal: ~20ms • Observed: 50ms ~ 200ms
  21. Finding clue • Large disk read • High utilization of

    network IO thread
  22. Reading code • Network IO thread multiplexes and handle IO

    with multiple client sockets asynchronously • It’s supposed to never block awaiting IO completion
  23. Guessing • When Network IO thread is busy, it means

    either: • 1. Really busy doing lots of work Many requests/ responses to read/write • 2. Blocked by some operations (which should not happen in event loop in general)
  24. Reading code (again) • Kafka uses sendfile(2) system call to

    transfer topic data to client • sendfile(2) system call might involves reading disk inside • sendfile is system call for copying on-disk data to client socket directly (zero-copy transfer) for efficient data transfer
  25. Guessing (again) • What if sendfile(2) call that involves disk

    read blocks network IO event-loop for long time? • Let’s check call duration of sendfile(2) system calls
  26. Inspecting • Using SystemTap • SystemTap: Dynamic tracing tool •

    Can be use to probe linux kernel behavior by writing few lines of script, with low overhead guarantee
  27. Inspecting

  28. Fixing • Populate page cache from different thread before network

    thread sending response
  29. Confirming • No response time degradation with coincidence of disk

    read
  30. Solving problem fundamentally • Keep making fundamental improvements (!= workaround)

    makes system ultimately reliable in the end • Understanding problem and fixing fundamentally requires understanding of all stacks involved to run the software • Application code - e.g, Kafka • Language runtime - e.g, JVM • Operating system - e.g, Linux
  31. High reliability requirement x High traffic = Very difficult challenge

    What makes our engineering fun?
  32. Unique challenges leads you to • Chances to encounter new

    problems that has never been addressed • If you could solve it, and feedback to community, you will receive a lot of prizes and satisfactions!
  33. So is LINE platform mature? • Definitely NO • LINE

    platform needs to be revised • More flexibility for more business integration • Higher availability • Multi iDC deployment • Make more “platforms” as shared building block for services
  34. Thanks for listening • More detailed explanation about Kafka performance

    issue troubleshooting: • LINE DEVELOPER DAY: https://youtu.be/6xBqTUUe0Sg • Kafka Summit SF 2018: https://www.confluent.io/kafka-summit-sf18/ kafka-multi-tenancy • Our Kafka upstream contribution examples: • https://issues.apache.org/jira/browse/KAFKA-7504 • https://issues.apache.org/jira/browse/KAFKA-4614