Streamlining Distributed Stream Processing with Superchief

Video: https://www.heavybit.com/library/blog/streamlining-distributed-stream-processing-with-superchief/

Distributed stream processing is all about gaining actionable insight from high volumes of data in near real-time. Stream processing is complex so many people look to frameworks for help. Last October Librato went in the other direction, moving from Apache Storm to an In-House distributed stream processing system named SuperChief.

Ray discusses the experience of moving from Apache Storm to SuperChief. He talks about some of the challenges Librato faced with Storm and the motivation to move away from frameworks. He covers the design and implementation of SuperChief, the benefits of this new approach and the direction SuperChief is going in the future.

Transcript

1. Streamlining Distributed Stream Processing with SuperChief April '16 SF Metrics Meetup Ray Jenkins
2. Hi, I’m Ray @_rayjenkins github.com/rjenkins
4. Metrics as a Service
6. Metrics and Monitoring as a Service
7. Thresholds, Windowed and Absent Detection Alerting
11. Metrics and Monitoring as a Service
12. Metrics and Cloud Monitoring (end of sales pitch)
13. Distributed Stream Processing
14. Stream Processing at Librato • Librato API ~15TB/day • 1 - 2Gb/sec Ingress Streaming Tier • Millions Messages/sec • 20 Kafka Topics / 15 Kafka nodes • 9 SuperChief clusters/workloads • 17 m3.2xls - 24GB Heap / G1 • 12 c3.2xls - 8GB Heap / G1 • Avg CPU Util 15% / Max 30% utilization
15. Philosophy
16. We Favor Composability and Small Components
17. Well Understood Technologies
18. End To End Visibility
19. Start Simple
20. API C*
21. Raw Data 1 Second Granularity
22. Time Series Aggregation 1minute, 15 minute, 1hr Min, Max, Sum, Average, Count
23. Do the easiest thing possible
24. API C* Batch
25. This Works Great!
26. Until It Doesn’t Processing Time > Aggregation Window
28. API API API C*
29. 3 Storm Topologies For Time Series Aggregations
30. ƒ(ƒ(x)) ≡ ƒ(x) Discrete Measurements Per Metric/Source “Rollup Topologies”
31. 60 Discrete One Second Measurements 15 Discrete One Minute Aggregations 4 Discrete 15 Minute Aggregations
32. r60 r3600 r900
33. 3 5 Storm Topologies
34. r60 r_out_60 r900 r_out_900 rwriter60 r3600 C* r60 rwriter r_out_900 r_out_3600 r_out_3600 C* r900 r_out_1 C* r3600 API API API C* raw
35. New Features
36. Service Side Aggregation Non-Discrete Variable Window Aggregations
37. Alerting, Historical Reporting
38. 5 12 Storm Topologies I’m not even going to try to draw it
43. Non-Idiomatic Code
44. Inability to jstack or Heap Dump
45. Unhandled Exception Kills Entire Worker
47. Inability To Isolate, Debug and Reason about Performance Issues
48. Overprovisioning
55. This Works Great! Until It Doesn’t
56. Disparate Tasks In Executors Tweak # of tasks to alter hashing, reship, and hope for a better allocation
57. Rewrote Spout Allocation Logic
58. 1 Worker Per Instance Separate Cluster for each Workload
59. Multiple JVM Hops Serialization and Netty Calls Add Up
60. Queue Contention 4 Threads From Entry to Exit
61. CPU Util Tuple Acking Enabled vs. Disabled
62. Engineering Involves Trade-Offs
63. When You Use A Framework You Trade-Off The Ability To Make Trade-Offs
64. YAGNI Introduce Abstractions When Needed
65. SuperChief Library for Sane Distributed Stream Processing
66. SuperChief > Storm Storm ~45 instances SuperChief ~20 instances ~3x Reduction in Resource Utilization
67. superchief-core
68. Dropwizard Configuration, Metrics, Logging and Managed
71. ZooKeeper Ensemble SC SC SC
72. Work Partitioning and Allocation I Want To Read From Kafka
73. Avoid Coordinating Wherever Possible The real trick of horizontal scalability is independence
74. Leader Partitions and Writes Work No Other Coordination
77. ZooKeeper Ensemble SC SC SC L
78. Semantic Partitioning Write To Kafka Partitions By How You Will Consume Them
80. Kafka Partition x Broker = Work Unit
84. ZooKeeper Ensemble SC SC SC L kafka-reader-config znode “{ partitions: { sc1: [ work … n ] sc2: [ work … n] }}
87. ZooKeeper Ensemble SC SC SC L kafka-reader-config znode “{ partitions: { sc1: [ work … n ] sc2: [ work … n] }}
88. ZooKeeper Ensemble SC SC SC L
90. KafkaReader Thread Per Broker/Topic/Partition
91. KafkaCheckpointer Per Reader, Offsets Pending, Watermark and Checkpoint to Zookeeper
92. Consumer Threads Default: availableProcessors()-1
93. Back Pressure
94. More Back Pressure
95. Offset Tracking / Ack
96. Fail Gracefully Be Partially Available
97. superchief-timeseries
98. C* r60 SC SC L r_out_1 sc-r60 SC SC r_out_60 sc-rwriter60 L
99. New Stuff
102. Thank You & Questions