Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Collecting Metrics

Collecting Metrics

Collecting metrics in your application with StatsD and librato.

Till Klampaeckel

November 06, 2012
Tweet

More Decks by Till Klampaeckel

Other Decks in Programming

Transcript

  1. www.easybib.com @klimpong
    Collecting Metrics
    MEASURE ALL THE THINGS.
    1
    Thursday, November 8, 12

    View Slide

  2. www.easybib.com @klimpong
    Till Klampäckel
    @klimpong — http://github.com/till
    I love open source.
    I’m addicted to graphs (and effects in
    keynote).
    /Till
    2
    Thursday, November 8, 12

    View Slide

  3. www.easybib.com @klimpong
    Just in case — we’re (always) looking for
    PHP and frontend developers.
    [email protected]
    /jobs
    3
    Thursday, November 8, 12

    View Slide

  4. www.easybib.com @klimpong
    A unit of measurement.
    An indicator for what is going on.
    /metrics
    4
    Thursday, November 8, 12

    View Slide

  5. www.easybib.com @klimpong
    Google Analytics/piwik: utm-campaign-tags
    Remember counter.cgi?
    Facebook pages.
    /examples
    5
    Thursday, November 8, 12

    View Slide

  6. www.easybib.com @klimpong
    Visits per year
    0M
    10M
    20M
    30M
    40M
    50M
    60M
    70M
    80M
    2007 2008 2009 2010 2011
    Visits per year
    /examples/graphs
    6
    Thursday, November 8, 12

    View Slide

  7. www.easybib.com @klimpong
    ?<br/>PHP to a database, remote service?<br/>StatsD!<br/>/collect<br/>7<br/>Thursday, November 8, 12<br/>

    View Slide

  8. www.easybib.com @klimpong
    StatsD is a collection daemon by Etsy.
    StatsD is Node.js.
    StatsD uses UDP.
    It’s asynchronous and painless.
    Backends are pluggable.
    /statsd
    8
    Thursday, November 8, 12
    Etsy’s work was inspired by Flickr.
    standard backends: graphite, console, reapeter
    available backends: librato, cloudwatch, mongodb, ...

    View Slide

  9. www.easybib.com @klimpong
    Original implementation in Node.JS.
    Erlang, Go, Scala, Java, Python.
    /statsd/pattern
    9
    Thursday, November 8, 12
    Original, original (Cal Henderson) — something in Perl at Flickr.
    StatsD as a pattern to collect metrics.

    View Slide

  10. www.easybib.com @klimpong
    Counters.
    Gauges.
    Timings.
    /statsd/metrics
    10
    Thursday, November 8, 12
    Really simple, no PhD required.
    Counter: grows absolute over time (number of total downloads)
    Gauge: for current values (e.g. “currently online”)
    Timings: how long something took
    There is more in master.

    View Slide

  11. www.easybib.com @klimpong
    Default backend.
    http://graphite.wikidot.com/
    /statsd/graphite
    11
    Thursday, November 8, 12
    We wanted a hosted service (less management).
    Read a couple blog posts about scaling Graphite which scared me off.
    Librato has less aggregation and very little transformation but it’s been rock solid.

    View Slide

  12. www.easybib.com @klimpong
    Download and install Node.js:
    http://nodejs.org/download/
    Clone statsd from Github:
    https://github.com/etsy/statsd
    npm install statsd-librato-backend
    /statsd/install
    12
    Thursday, November 8, 12
    Install Node.js through: homebrew, aptitude, Windows installer
    cd into statsd directory to npm install

    View Slide

  13. www.easybib.com @klimpong
    {
    port: statsd_port,
    mgmt_address: '127.0.0.1',
    backends: ['statsd-librato-backend'],
    librato: {
    email: 'email_on_librato',
    token: 'librato_token',
    source: 'where_metrics_are_coming_from'
    }
    }
    /statsd/configure
    13
    Thursday, November 8, 12
    Watch me type!
    The minimum is email and token.
    port: port of the statsd process
    librato.email/.token: credentials from librato
    librato.source: e.g. your cluster (application, whatever)

    View Slide

  14. www.easybib.com @klimpong
    ./bin/statsd
    /statsd/run
    14
    Thursday, November 8, 12
    Maybe add monit for when the node process crashes (0.8.0+ recommended).

    View Slide

  15. www.easybib.com @klimpong
    Find a PHP client library.
    /statsd/now-what
    15
    Thursday, November 8, 12
    StatsD has examples for half a dozen languages.
    Want something more sophisticated?

    View Slide

  16. www.easybib.com @klimpong
    // https://github.com/till/php-statsd
    require_once 'StatsD.php';
    StatsD::init(array(
    'enabled' => true,
    'host' => 'statsd1'
    ));
    StatsD::increment('some-value');
    StatsD::increment(array(
    'multiple', 'counters', 'work', 'as', 'well'
    ));
    /collect
    16
    Thursday, November 8, 12
    watch me type again
    forked the PHP sample code

    View Slide

  17. www.easybib.com @klimpong
    /librato
    17
    Thursday, November 8, 12
    This is an excerpt of stuff we collect. And every time there’s a problem, we add to it.
    https://metrics.librato.com/pricing
    50 metrics every 60 seconds = ~5.00 USD/month
    flush every 30 = ~8.50 USD a month

    View Slide

  18. www.easybib.com @klimpong
    Graph the ms it takes to pull up different views and _all_docs.
    /librato/demo
    18
    Thursday, November 8, 12

    View Slide

  19. www.easybib.com @klimpong
    Requests (per second) to our API. And failed requests also (none).
    /librato/demo
    19
    Thursday, November 8, 12

    View Slide

  20. www.easybib.com @klimpong
    Graph requests (per second) to EasyBib Research
    related APIs.
    /librato/demo
    20
    Thursday, November 8, 12

    View Slide

  21. www.easybib.com
    Fin
    21
    Thursday, November 8, 12

    View Slide