The_journey_towards_Pinot.pdf

Transcript

1. Ananth Packkildurai
   March 27, 2019
   1
   The Journey Towards bv
   

   View Slide

2. Me
   ➢ @ananthdurai
   ➢ Senior Software Engineer
   @ Slack
   ➢ Passionate about all things related
   to ethical data management
   

   View Slide

3. Public launch: 2014 1000+ employees across
   7 countries worldwide
   HQ in San Francisco
   $841M in capital raised
   Key investors include Softbank, Accel,
   a16z, Social Capital, Index, Thrive, GV,
   Kleiner Perkins, GGV, Horizons, Spark,
   IVP and DST.
   Diverse set of industries
   including software/technology, retail, media,
   telecom and professional services.
   About Slack
   

   View Slide

4. It’s all about `Slog`
   

   View Slide

5. How reliable our logging?
   Use case #1: Logging correctness
   

   View Slide

6. I have well-defined dimensions & metrics,
   and I want a real-time aggregation and
   querying.
   Type 1 (The Analytical pattern)
   (e.g)
   ❖ performance analytics
   ❖ experimentation
   

   View Slide

7. I want to slice & dice metrics in real-time.
   If something goes wrong, I would like to
   see the detail events for the given
   dimensions.
   Type 2 (The needle in the haystack pattern)
   (e.g)
   ❖ Mysql slow queries
   ❖ Audit logs
   ❖ Security logs
   

   View Slide

8. I want to dump all the events and explore
   further to understand the pattern.
   Type 3 (The exploration pattern)
   (e.g)
   ❖ Log inspection
   ❖ Exploratory analysis
   

   View Slide

9. Druid
   Druid i3.2X middle manager node
   40+
   10+
   Pinot
   T-Digest
   17 I3 2x Pinot
   Servers
   0 Ingestion nodes
   Druid -> Pinot
   Tranquility ingestion nodes.
   Approximate Histogram aggregator
   

   View Slide

10. Log Inspector
    

    View Slide

11. Log Inspector
    

    View Slide

12. Did I broke anything while
    launching an experiment?
    Is there any performance
    degrade with my experiments?
    Use case #2: The experimentation framework
    

    View Slide

13. Pinot Next
    Log search support
    Add support for free text search on the
    dimensions.
    https://github.com/apache/incubator-pinot/
    issues/2798
    Integration with visualization tools
    Ease the integration with the operational
    intelligence and business intelligence
    solutions like grafana, superset etc
    

    View Slide

14. Thank You!
    14
    For more information go to: slack.com
    

    View Slide