Collecting Metrics

Collecting metrics in your application with StatsD and librato.

Transcript

1. www.easybib.com @klimpong
   Collecting Metrics
   MEASURE ALL THE THINGS.
   1
   Thursday, November 8, 12
   

   View Slide

2. www.easybib.com @klimpong
   Till Klampäckel
   @klimpong — http://github.com/till
   I love open source.
   I’m addicted to graphs (and effects in
   keynote).
   /Till
   2
   Thursday, November 8, 12
   

   View Slide

3. www.easybib.com @klimpong
   Just in case — we’re (always) looking for
   PHP and frontend developers.
   till@imagineeasy.com
   /jobs
   3
   Thursday, November 8, 12
   

   View Slide

4. www.easybib.com @klimpong
   A unit of measurement.
   An indicator for what is going on.
   /metrics
   4
   Thursday, November 8, 12
   

   View Slide

5. www.easybib.com @klimpong
   Google Analytics/piwik: utm-campaign-tags
   Remember counter.cgi?
   Facebook pages.
   /examples
   5
   Thursday, November 8, 12
   

   View Slide

6. www.easybib.com @klimpong
   Visits per year
   0M
   10M
   20M
   30M
   40M
   50M
   60M
   70M
   80M
   2007 2008 2009 2010 2011
   Visits per year
   /examples/graphs
   6
   Thursday, November 8, 12
   

   View Slide

7. www.easybib.com @klimpong
   ?<br/>PHP to a database, remote service?<br/>StatsD!<br/>/collect<br/>7<br/>Thursday, November 8, 12<br/>

   View Slide

8. www.easybib.com @klimpong
   StatsD is a collection daemon by Etsy.
   StatsD is Node.js.
   StatsD uses UDP.
   It’s asynchronous and painless.
   Backends are pluggable.
   /statsd
   8
   Thursday, November 8, 12
   Etsy’s work was inspired by Flickr.
   standard backends: graphite, console, reapeter
   available backends: librato, cloudwatch, mongodb, ...
   

   View Slide

9. www.easybib.com @klimpong
   Original implementation in Node.JS.
   Erlang, Go, Scala, Java, Python.
   /statsd/pattern
   9
   Thursday, November 8, 12
   Original, original (Cal Henderson) — something in Perl at Flickr.
   StatsD as a pattern to collect metrics.
   

   View Slide

10. www.easybib.com @klimpong
    Counters.
    Gauges.
    Timings.
    /statsd/metrics
    10
    Thursday, November 8, 12
    Really simple, no PhD required.
    Counter: grows absolute over time (number of total downloads)
    Gauge: for current values (e.g. “currently online”)
    Timings: how long something took
    There is more in master.
    

    View Slide

11. www.easybib.com @klimpong
    Default backend.
    http://graphite.wikidot.com/
    /statsd/graphite
    11
    Thursday, November 8, 12
    We wanted a hosted service (less management).
    Read a couple blog posts about scaling Graphite which scared me off.
    Librato has less aggregation and very little transformation but it’s been rock solid.
    

    View Slide

12. www.easybib.com @klimpong
    Download and install Node.js:
    http://nodejs.org/download/
    Clone statsd from Github:
    https://github.com/etsy/statsd
    npm install statsd-librato-backend
    /statsd/install
    12
    Thursday, November 8, 12
    Install Node.js through: homebrew, aptitude, Windows installer
    cd into statsd directory to npm install
    

    View Slide

13. www.easybib.com @klimpong
    {
    port: statsd_port,
    mgmt_address: '127.0.0.1',
    backends: ['statsd-librato-backend'],
    librato: {
    email: 'email_on_librato',
    token: 'librato_token',
    source: 'where_metrics_are_coming_from'
    }
    }
    /statsd/configure
    13
    Thursday, November 8, 12
    Watch me type!
    The minimum is email and token.
    port: port of the statsd process
    librato.email/.token: credentials from librato
    librato.source: e.g. your cluster (application, whatever)
    

    View Slide

14. www.easybib.com @klimpong
    ./bin/statsd
    /statsd/run
    14
    Thursday, November 8, 12
    Maybe add monit for when the node process crashes (0.8.0+ recommended).
    

    View Slide

15. www.easybib.com @klimpong
    Find a PHP client library.
    /statsd/now-what
    15
    Thursday, November 8, 12
    StatsD has examples for half a dozen languages.
    Want something more sophisticated?
    

    View Slide

16. www.easybib.com @klimpong
    // https://github.com/till/php-statsd
    require_once 'StatsD.php';
    StatsD::init(array(
    'enabled' => true,
    'host' => 'statsd1'
    ));
    StatsD::increment('some-value');
    StatsD::increment(array(
    'multiple', 'counters', 'work', 'as', 'well'
    ));
    /collect
    16
    Thursday, November 8, 12
    watch me type again
    forked the PHP sample code
    

    View Slide

17. www.easybib.com @klimpong
    /librato
    17
    Thursday, November 8, 12
    This is an excerpt of stuff we collect. And every time there’s a problem, we add to it.
    https://metrics.librato.com/pricing
    50 metrics every 60 seconds = ~5.00 USD/month
    flush every 30 = ~8.50 USD a month
    

    View Slide

18. www.easybib.com @klimpong
    Graph the ms it takes to pull up different views and _all_docs.
    /librato/demo
    18
    Thursday, November 8, 12
    

    View Slide

19. www.easybib.com @klimpong
    Requests (per second) to our API. And failed requests also (none).
    /librato/demo
    19
    Thursday, November 8, 12
    

    View Slide

20. www.easybib.com @klimpong
    Graph requests (per second) to EasyBib Research
    related APIs.
    /librato/demo
    20
    Thursday, November 8, 12
    

    View Slide

21. www.easybib.com
    Fin
    21
    Thursday, November 8, 12
    

    View Slide