Reactive Stream Processing with Mantis

Talk at Reactive Summit 2016 on on-demand streaming, job discovery & chaining and auto-scaling aspects of the Mantis Reactive Stream Processing platform at Netflix.

Transcript

1. Reactive Stream Processing With Mantis Neeraj Joshi - Senior Software Engineer, Edge Systems Nick Mahilani - Senior Software Engineer, Edge Systems October 4, Reactive Summit 2016
2. Monitoring a complex distributed service @ scale is hard
3. 81+ Million Subscribers Across the Globe
4. Streaming thousands of Titles
5. Over Millions of Devices
6. Powered by 100s of Micro-services
7. Combinatorial Explosion of Data !!!
8. Complexity and Comprehension
9. So, in order to manage complex environments, need to rethink insights, shift the curve
10. An Insight system that can... Auto-detect anomalies in high volume, high cardinality data
11. An Insight system that can... Auto-detect anomalies in high volume, high cardinality data Identify titles that have an abnormal failure rate and highlight their common characteristics (only on certain devices using certain CDNs etc)
12. An Insight system that can... Aggregate rich data On-demand
13. An Insight system that can... Aggregate rich data On-demand Calculate latency percentiles for PS4 in UK using firmware v1.0 and ui 1.2.1 On Demand
14. An Insight system that can... Find your needle-in-the-haystack in real-time
15. An Insight system that can... Find your needle-in-the-haystack in real-time Find me all requests for customer X with latency > 1 seconds
16. And at the same time be cost effective!
17. Edge servers alone generate 10 Tb/s of operational data!!
18. How can we contain the cost?
19. Reduce Data Two Strategies Optimize Resource Usage
20. What if? We only stream what is needed & when it is needed?
21. Do we really need all the data all the time?
22. Anomaly Detection Use-case Look for abnormal trends in aggregate signal
23. Anomaly Detection Use-case Look for abnormal trends in aggregate signal Deeper analysis on filtered events