The_journey_towards_Pinot.pdf

Transcript

1. Ananth Packkildurai March 27, 2019 1 The Journey Towards bv
2. Me ➢ @ananthdurai ➢ Senior Software Engineer @ Slack ➢ Passionate about all things related to ethical data management
3. Public launch: 2014 1000+ employees across 7 countries worldwide HQ in San Francisco $841M in capital raised Key investors include Softbank, Accel, a16z, Social Capital, Index, Thrive, GV, Kleiner Perkins, GGV, Horizons, Spark, IVP and DST. Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
4. It’s all about `Slog`
5. How reliable our logging? Use case #1: Logging correctness
6. I have well-defined dimensions & metrics, and I want a real-time aggregation and querying. Type 1 (The Analytical pattern) (e.g) ❖ performance analytics ❖ experimentation
7. I want to slice & dice metrics in real-time. If something goes wrong, I would like to see the detail events for the given dimensions. Type 2 (The needle in the haystack pattern) (e.g) ❖ Mysql slow queries ❖ Audit logs ❖ Security logs
8. I want to dump all the events and explore further to understand the pattern. Type 3 (The exploration pattern) (e.g) ❖ Log inspection ❖ Exploratory analysis
9. Druid Druid i3.2X middle manager node 40+ 10+ Pinot T-Digest 17 I3 2x Pinot Servers 0 Ingestion nodes Druid -> Pinot Tranquility ingestion nodes. Approximate Histogram aggregator
10. Log Inspector
11. Log Inspector
12. Did I broke anything while launching an experiment? Is there any performance degrade with my experiments? Use case #2: The experimentation framework
13. Pinot Next Log search support Add support for free text search on the dimensions. https://github.com/apache/incubator-pinot/ issues/2798 Integration with visualization tools Ease the integration with the operational intelligence and business intelligence solutions like grafana, superset etc
14. Thank You! 14 For more information go to: slack.com