IMF introduction booth session @ TW DataCon

Transcript

1. IMF The team behind Kafka clusters at LINE
2. Kafka at Line
3. May I use this cluster...?
4. What is IMF Mission. Aim to provide internal platform at LINE with a high-performance, reliable and easy-to-use data communication platform with Kafka. What we are doing? - Operate Kafka cluster (Monitoring, daily operation, constructing new cluster) - Develop Kafka (Bug fixes, performance patches) and contribute to the OSS community - Develop libraries and frameworks to improve the usability of our Kafka platform - Consulting and support for other teams who uses the platform
5. Scalable Software stack
6. Cluster Scale Truth. - 60 services use our Kafka (e.g., LINE, LINE Bot, LINE Pay, etc.) - 750 topics in our release cluster - 260 billion messages / day - 210 TB incoming data / day - 60 brokers How about availability? 99.98% in 2018
7. Cluster Scale Challenge Accepted. - Tons of dependent services - High traffic - Low latency - Few members In IMF, we have to provide a platform which has the highest availability and reliability in efficient way.
8. The IMF Stack We concentrate on engineering projects which will: - Increase performance - Increase usability - Increase availability - Decrease the cost of operation
9. Performance
10. Those tough problems in our clusters Motivation. In order to provide a high performance platform, scratching the surface is not enough, we have to understand the overall behavior. Achievement. KAFKA-4024 - Removed unnecessary blocking behavior of producer KAFKA-4164 - Long GC pause by disk read in mmap(2) KAFKA-6051 - Reduce significant latency involved during restarting broker
11. We also care about client side Problem. Partition is the only unit of scalability. • 1 thread per partition at maximum • Not enough for IO-intensive workloads • e.g., Kafka stream
12. Decaton - next generation consummer library Main Feature. Flexible parallelism. • Concurrent processing for records from single partition • Key based ordering guarantee • Out of order commit control
13. Decaton - Flexible parallelism Decaton enables concurrent processing of single partition. • Internal queue for each partitions • A dedicated thread for each sub partition
14. Sounds easy, huh? Let's think about offset management.
15. Usability
16. Provisioner - Make our clients happy Objective. Improve the workflow of creating/ altering resource on clusters. Before. • Use email • Topic configurations live in Kafka After. • Use Github PR • Topic configuration is visible to every client • Verbose information for diagnosis • 1-click application
17. ...And more
18. Availability is important, of course Prober Provide tons of metrics for distinguishing whether the problem exists in cluster side or in client side Custom producer • FailureLoggingProducer • Dual cluster setup
19. Future Work Operate Kafka platform with highest reliability and performance enabling all needs. - Bring out 100% potential of Kafka - Contribute to keep service reliability high Provide autonomous Kafka platform operable with constant number of people - More users, same cost
20. JOIN US!