Unlocking the Potential of Data ................................................................................................................................ 

Karthik Kastury Software Engineer - PayPal India Engineer / Hacker / Blogger / Apple fanboi and evil minded. *Pun Intended* 24x7 Internet Geek | Unlucky in Cards ................................................................................................................................ ................................................................................................................................ Twitter Email [email protected] @KarthikDot Internet Addict for 12+ Years... 

measure everything. http://codeascraft.com/2011/02/15/measure-anything-measure-everything/ ................................................................................................ 

.............................................................................................................. .............................................................................................................. Measure Everything! ‣User actions. (signups, login, other app actions) ‣Application Events (exceptions, deploys, perf. metrics, sql query perf.) ‣Business Metrics (active users, revenue) ‣Internal Metrics (build times, deploy times) 

.............................................................................................................. .............................................................................................................. Why? ‣Cheap, Effective and Scalable. ‣Power to the Developers/Product Managers, to measure anything/everything they like. ‣As important as a QA team. Know what your app is doing in production. 

.............................................................................................................. .............................................................................................................. Benefits ‣Total visibility into your app’s behavior in production. ‣No more “Hey it works fine on my machine!” ‣Feedback loops are extremely short, so the time between a change, and things going bad is is super *small* ‣Quickly find the needle in the haystack when things mess up! 

StatsD https://github.com/etsy/statsd/ ................................................................................................ 

.............................................................................................................. .............................................................................................................. StatsD ‣Open Source Project from the good folks at Etsy. ‣Node.Js Server Implementation, with Support for multiple backends. Server implementation also in C++, Python, Ruby ‣Clients in *almost* every language. 

.............................................................................................................. .............................................................................................................. What is StatsD? ‣A network daemon that listens for statistics and aggregates them on the fly. ‣Multiple types of metrics, counters, timers, gauges. ‣Default Backend is Graphite, but can send data downstream to all sorts of backends. 

... .............................................................................................................. .............................................................................................................. Metrics examples accounts.auth.password.attempted accounts.auth.password.succeeded accounts.auth.password.failed accounts.auth.password.failure.invalid_email accounts.auth.password.failure.incorrect_password metric nesting accounts.auth.password.failure.* 

.............................................................................................................. .............................................................................................................. Metrics Aggregation ‣StatsD flushes all the metrics in an aggregated fashion at periodic intervals. ‣The backend only receives the aggregated metrics. ‣Most Backends can further aggregate data into batches of time. 

.............................................................................................................. .............................................................................................................. Backend Configuration ‣Configure your stats backend to capture mean, median, 75th, 90th, 95th and 99th percentile data. ‣Just capturing mean/median is generally not recommended. pro tip! ‣Don’t do DNS resolution while sending stats to StatsD or to the backend. 

does it scale? ................................................................................................ 

yes ................................................................................................ 

.............................................................................................................. .............................................................................................................. Scalability ‣A Single StatsD Server can receive ten’s of thousands of events per second on a decently powered machine. ‣Horizontally Scalable. You can shard metrics depending on their namespace and send them to different StatsD instances. ‣StatsD receives metrics over UDP rather than TCP. 

.............................................................................................................. .............................................................................................................. Metric Roll Ups ‣Graphite, and most other backends allow you to roll up data as you go to lower precisions. ‣StatsD by default flushes aggregated metrics to the backend every 10s. ‣Here’s my roll up scheme. 10s, 1min, 5min, 60min, 24hr, 30days, 1year 

.............................................................................................................. .............................................................................................................. Metrics Backends ‣You can send data to one or multiple backends from StatsD. ‣Graphite is the most popular and the default backend for StatsD. http://graphite.wikidot.com/ ‣Commercial alternatives like InstrumentalApp, Librato Metrics exist. https://instrumentalapp.com/ https://metrics.librato.com/ 

measure everything. ................................................................................................ 

measure everything, everytime. ................................................................................................ 

data visualization Dashboards Alerts ................................................................................................ 

.............................................................................................................. .............................................................................................................. Dashboards? Graphs? Librato Metrics Dashboard - RubyGems Bundler API https://metrics.librato.com/share/dashboards/k4b5bhm8 

.............................................................................................................. .............................................................................................................. Dashboards (Open Source) ‣Graphite allows to build dashboards fairly easily, combining different metrics in any way you want. ‣But Graphite’s Graphs are kinda boring! ‣Whole lot of open source, dashboard/graphing solutions that work with Graphite exist. 

Graphite Giraffe Graphiti Tasseo Dashing GDash Graphene graphite.wikidot.com/start giraffe.kenhub.com github.com/paperlesspost/graphiti github.com/obfuscurity/tasseo shopify.github.io/dashing github.com/ripienaar/gdash jondot.github.io/graphene .............................................................................................................. .............................................................................................................. 

.............................................................................................................. .............................................................................................................. Dashboards (Commercial) ‣StatsD supports various other backends to which it send data to. ‣InstrumentalApp is a wonderful choice, if you quickly want to build dashboards. https://instrumentalapp.com/ ‣Librato Metrics is *even* more awesome. Dashboards + Alerts and Notifications via Email / PagerDuty etc. https://metrics.librato.com/ 

realtime versus delayed ................................................................................................ 

.............................................................................................................. .............................................................................................................. Realtime ‣Extra pressure. You tend to over measure every small change. ‣Never take product decisions with a few hours worth of data. Wait for a few days for patterns to kick in. ‣Realtime makes sense for key business metrics, application metrics, infrastructure. ‣Divorce Product and Operational Metrics. 

.............................................................................................................. .............................................................................................................. Delayed ‣Product Metrics are best observed over a period of time. ‣Capture them in realtime, but make change decisions only after a significant amount of data has been collected. ‣Realtime makes sense, when things go south! 

realtime versus delayed Database Metrics Application Metrics Errors and Exceptions Other Operational Metrics Revenue User Conversion Key Business Metrics Other Product Metrics divorce product & operational metrics! 

success stories ................................................................................................ 

.............................................................................................................. .............................................................................................................. Success Stories ‣ A major part of metric collection at Etsy happens via StatsD. All kinds of operational & product use cases. ‣ Shopify uses StatsD to figure out how their internal caching systems are performing, and to debug issues. ‣ 37Signals the folks behind Basecamp have a custom Golang implementation of StatsD Server to capture app metrics. Doing about 150k metrics/sec. 

Test Driven Development ................................................................................................ 

Behavior Driven Development ................................................................................................ Test Driven Development 

Documentation Driven Development ................................................................................................ Behavior Driven Development ................................................................................................ Test Driven Development Documentation Driven Development 

Documentation Driven Development ................................................................................................ Behavior Driven Development ................................................................................................ Test Driven Development Documentation Driven Development Metrics Driven Development * * I didn’t invent this! 

measure everything, everytime. ................................................................................................ 

................................................................................................ thank you! twitter email [email protected] [email protected] @KarthikDot slides https://speakerdeck.com/karthikdot/unlocking-the-potential-of-data http://bit.ly/5elunlock2013