Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The_journey_towards_Pinot.pdf

 The_journey_towards_Pinot.pdf

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

March 25, 2019
Tweet

Transcript

  1. Ananth Packkildurai March 27, 2019 1 The Journey Towards bv

  2. Me ➢ @ananthdurai ➢ Senior Software Engineer @ Slack ➢

    Passionate about all things related to ethical data management
  3. Public launch: 2014 1000+ employees across 7 countries worldwide HQ

    in San Francisco $841M in capital raised Key investors include Softbank, Accel, a16z, Social Capital, Index, Thrive, GV, Kleiner Perkins, GGV, Horizons, Spark, IVP and DST. Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
  4. It’s all about `Slog`

  5. How reliable our logging? Use case #1: Logging correctness

  6. I have well-defined dimensions & metrics, and I want a

    real-time aggregation and querying. Type 1 (The Analytical pattern) (e.g) ❖ performance analytics ❖ experimentation
  7. I want to slice & dice metrics in real-time. If

    something goes wrong, I would like to see the detail events for the given dimensions. Type 2 (The needle in the haystack pattern) (e.g) ❖ Mysql slow queries ❖ Audit logs ❖ Security logs
  8. I want to dump all the events and explore further

    to understand the pattern. Type 3 (The exploration pattern) (e.g) ❖ Log inspection ❖ Exploratory analysis
  9. Druid Druid i3.2X middle manager node 40+ 10+ Pinot T-Digest

    17 I3 2x Pinot Servers 0 Ingestion nodes Druid -> Pinot Tranquility ingestion nodes. Approximate Histogram aggregator
  10. Log Inspector

  11. Log Inspector

  12. Did I broke anything while launching an experiment? Is there

    any performance degrade with my experiments? Use case #2: The experimentation framework
  13. Pinot Next Log search support Add support for free text

    search on the dimensions. https://github.com/apache/incubator-pinot/ issues/2798 Integration with visualization tools Ease the integration with the operational intelligence and business intelligence solutions like grafana, superset etc
  14. Thank You! 14 For more information go to: slack.com