Unlocking the Potential of Data

Slides from a talk that I gave at Fifth Elephant, a Big Data Conference in Bangalore, in July 2013.

This talk focusses on how product managers and developers can use simple techniques, and unlock the potential of data in building better products and applications everyday!

Focusses on the potential of real time analytics, and instrumentation of the application, and how that would lead to better insights on the application everyday and every minute it runs!

Transcript

1. Unlocking the Potential of Data ................................................................................................................................
2. Karthik Kastury Software Engineer - PayPal India Engineer / Hacker / Blogger / Apple fanboi and evil minded. *Pun Intended* 24x7 Internet Geek | Unlucky in Cards ................................................................................................................................ ................................................................................................................................ Twitter Email hello@karthik.kastury.in @KarthikDot Internet Addict for 12+ Years...
3. measure everything. http://codeascraft.com/2011/02/15/measure-anything-measure-everything/ ................................................................................................
4. .............................................................................................................. .............................................................................................................. Measure Everything! ‣User actions. (signups, login, other app actions) ‣Application Events (exceptions, deploys, perf. metrics, sql query perf.) ‣Business Metrics (active users, revenue) ‣Internal Metrics (build times, deploy times)
5. .............................................................................................................. .............................................................................................................. Why? ‣Cheap, Effective and Scalable. ‣Power to the Developers/Product Managers, to measure anything/everything they like. ‣As important as a QA team. Know what your app is doing in production.
6. .............................................................................................................. .............................................................................................................. Benefits ‣Total visibility into your app’s behavior in production. ‣No more “Hey it works fine on my machine!” ‣Feedback loops are extremely short, so the time between a change, and things going bad is is super *small* ‣Quickly find the needle in the haystack when things mess up!
7. StatsD https://github.com/etsy/statsd/ ................................................................................................
8. .............................................................................................................. .............................................................................................................. StatsD ‣Open Source Project from the good folks at Etsy. ‣Node.Js Server Implementation, with Support for multiple backends. Server implementation also in C++, Python, Ruby ‣Clients in *almost* every language.
9. .............................................................................................................. .............................................................................................................. What is StatsD? ‣A network daemon that listens for statistics and aggregates them on the fly. ‣Multiple types of metrics, counters, timers, gauges. ‣Default Backend is Graphite, but can send data downstream to all sorts of backends.
10. <namespace>.<section>.<target>.<action> .............................................................................................................. .............................................................................................................. Metrics examples accounts.auth.password.attempted accounts.auth.password.succeeded accounts.auth.password.failed accounts.auth.password.failure.invalid_email accounts.auth.password.failure.incorrect_password metric nesting accounts.auth.password.failure.*
11. .............................................................................................................. .............................................................................................................. Metrics Aggregation ‣StatsD flushes all the metrics in an aggregated fashion at periodic intervals. ‣The backend only receives the aggregated metrics. ‣Most Backends can further aggregate data into batches of time.
12. .............................................................................................................. .............................................................................................................. Backend Configuration ‣Configure your stats backend to capture mean, median, 75th, 90th, 95th and 99th percentile data. ‣Just capturing mean/median is generally not recommended. pro tip! ‣Don’t do DNS resolution while sending stats to StatsD or to the backend.
13. does it scale? ................................................................................................
14. yes ................................................................................................
15. .............................................................................................................. .............................................................................................................. Scalability ‣A Single StatsD Server can receive ten’s of thousands of events per second on a decently powered machine. ‣Horizontally Scalable. You can shard metrics depending on their namespace and send them to different StatsD instances. ‣StatsD receives metrics over UDP rather than TCP.
16. .............................................................................................................. .............................................................................................................. Metric Roll Ups ‣Graphite, and most other backends allow you to roll up data as you go to lower precisions. ‣StatsD by default flushes aggregated metrics to the backend every 10s. ‣Here’s my roll up scheme. 10s, 1min, 5min, 60min, 24hr, 30days, 1year
17. .............................................................................................................. .............................................................................................................. Metrics Backends ‣You can send data to one or multiple backends from StatsD. ‣Graphite is the most popular and the default backend for StatsD. http://graphite.wikidot.com/ ‣Commercial alternatives like InstrumentalApp, Librato Metrics exist. https://instrumentalapp.com/ https://metrics.librato.com/
18. measure everything. ................................................................................................
19. measure everything, everytime. ................................................................................................
20. data visualization Dashboards Alerts ................................................................................................
21. .............................................................................................................. .............................................................................................................. Dashboards? Graphs? Librato Metrics Dashboard - RubyGems Bundler API https://metrics.librato.com/share/dashboards/k4b5bhm8
22. .............................................................................................................. .............................................................................................................. Dashboards (Open Source) ‣Graphite allows to build dashboards fairly easily, combining different metrics in any way you want. ‣But Graphite’s Graphs are kinda boring! ‣Whole lot of open source, dashboard/graphing solutions that work with Graphite exist.
23. Graphite Giraffe Graphiti Tasseo Dashing GDash Graphene graphite.wikidot.com/start giraffe.kenhub.com github.com/paperlesspost/graphiti github.com/obfuscurity/tasseo shopify.github.io/dashing github.com/ripienaar/gdash jondot.github.io/graphene .............................................................................................................. ..............................................................................................................
24. .............................................................................................................. .............................................................................................................. Dashboards (Commercial) ‣StatsD supports various other backends to which it send data to. ‣InstrumentalApp is a wonderful choice, if you quickly want to build dashboards. https://instrumentalapp.com/ ‣Librato Metrics is *even* more awesome. Dashboards + Alerts and Notifications via Email / PagerDuty etc. https://metrics.librato.com/
25. realtime versus delayed ................................................................................................
26. .............................................................................................................. .............................................................................................................. Realtime ‣Extra pressure. You tend to over measure every small change. ‣Never take product decisions with a few hours worth of data. Wait for a few days for patterns to kick in. ‣Realtime makes sense for key business metrics, application metrics, infrastructure. ‣Divorce Product and Operational Metrics.
27. .............................................................................................................. .............................................................................................................. Delayed ‣Product Metrics are best observed over a period of time. ‣Capture them in realtime, but make change decisions only after a significant amount of data has been collected. ‣Realtime makes sense, when things go south!
28. realtime versus delayed Database Metrics Application Metrics Errors and Exceptions Other Operational Metrics Revenue User Conversion Key Business Metrics Other Product Metrics divorce product & operational metrics!
29. success stories ................................................................................................
30. .............................................................................................................. .............................................................................................................. Success Stories ‣ A major part of metric collection at Etsy happens via StatsD. All kinds of operational & product use cases. ‣ Shopify uses StatsD to figure out how their internal caching systems are performing, and to debug issues. ‣ 37Signals the folks behind Basecamp have a custom Golang implementation of StatsD Server to capture app metrics. Doing about 150k metrics/sec.
31. Test Driven Development ................................................................................................
32. Behavior Driven Development ................................................................................................ Test Driven Development
33. Documentation Driven Development ................................................................................................ Behavior Driven Development ................................................................................................ Test Driven Development Documentation Driven Development
34. Documentation Driven Development ................................................................................................ Behavior Driven Development ................................................................................................ Test Driven Development Documentation Driven Development Metrics Driven Development * * I didn’t invent this!
35. measure everything, everytime. ................................................................................................
36. ................................................................................................ thank you! twitter email hello@karthik.kastury.in kkastury@paypal.com @KarthikDot slides https://speakerdeck.com/karthikdot/unlocking-the-potential-of-data http://bit.ly/5elunlock2013