$30 off During Our Annual Pro Sale. View Details »

Statsd and Graphite

Jeremy Quinton
September 17, 2015

Statsd and Graphite

Application metrics are extremely important but are often hard to gather as our PHP Applications differ significantly. Using StatsD and Graphite we can gather metrics from our applications no matter what their shape or form. In this talk I will discuss how you can use Statsd to send various metrics of your PHP applications to Graphite. StatsD is a simple NodeJS daemon for easy stats aggregation and makes it simple to plot application metrics on a graph in Graphite. Using the metrics that are gathered its possible to get an overview of what is happening with our applications in near realtime which is extremely useful. Graphite additionally allows us to produce easy understandable graphs and dashboards which once analysed can be used to improve our PHP applications. My talk will cover everything from setting up Statsd and Graphite to how you gather the metrics from within your PHP applications. After the talk developers should be confident enough to go away and implement these technologies in their applications.

Jeremy Quinton

September 17, 2015
Tweet

More Decks by Jeremy Quinton

Other Decks in Technology

Transcript

  1. Gathering Metrics
    with StatsD and
    Graphite
    http://www.flickr.com/photos/wwarby/
    Jeremy Quinton - PHP Cape Town 2015
    @jeremyquinton

    View Slide

  2. twitter - @jeremyquinton
    Developer - PHP since 2003
    Open Source Enthusiast
    Devops Evangelist
    Infrastructure and Architecture

    View Slide

  3. Devops
    • Culture
    • Automation
    • Measurement
    • Sharing

    View Slide

  4. Web Application
    Go Live to production
    Majority of our traffic in a
    3 month period with big
    spike at the end
    Load Testing
    Multiple deployments with frequent changes
    Building the app series of sprints - Lead
    traffic
    No Application metrics

    View Slide

  5. You can’t optimise what you can’t measure - Juozas Kaziukenas

    View Slide

  6. Flickr - http://code.flickr.net/2008/10/27/counting-
    timing/
    Etsy - http://codeascraft.com/2011/02/15/measure-
    anything-measure-everything/
    Blogs posts

    View Slide

  7. Measuring at three levels
    • Network
    • Machine
    • Application
    http://codeascraft.com/2011/02/15/measure-anything-measure-everything/

    View Slide

  8. Why measurement is important?
    • Helps us to understand things.
    • Once we understand things we can improve them.

    View Slide

  9. “Measurement is the first step that leads to control
    and eventually to improvement. If you can’t measure
    something, you can’t understand it. If you can’t
    understand it, you can’t control it. If you can’t control
    it, you can’t improve it.”
    ― H. James Harrington

    View Slide

  10. "A system or standard of measurement"
    Metric

    View Slide

  11. System Overview
    Application
    StatsD
    Graphite
    Carbon
    Whisper
    Wep app

    View Slide

  12. Types of Application Metrics
    • Counting
    How many emails did we send over the last 10 minutes?
    How many users logging into the system in the last 5 minutes?
    How many calls have we made to a particular api?

    View Slide

  13. https://packagist.org/search/?q=statsd
    PHP statsD client library

    View Slide

  14. Over the wire
    accounts.authentication.login.attempted:1|c
    $connection = new Domnikl\Statsd\Connection\Socket('localhost', 8125);

    $statsd = new Domnikl\Statsd\Client($connection);
    Example Counting Metric
    Metric name
    Count value
    Metric type
    echo “accounts.authentication.login.attempted:1|c” | nc -w0 -u 127.0.0.1
    // simple counts

    $statsd->increment(“accounts.authentication.login.attempted”);

    View Slide

  15. StatsD
    • Built By Etsy.
    • Network Daemon gathers and flushes metrics to a back-end.
    • Sends them to a back-end at a specific interval. Default 10s.
    • Uses UDP fire and forget.

    View Slide

  16. View Slide

  17. StatsD
    • git clone https://github.com/etsy/statsd.git
    • Create a config file from exampleConfig.js and put it somewhere.
    • Start the Daemon: node stats.js /path/to/config

    View Slide

  18. StatsD
    {
    port: 8125,
    backends: ['./backends/graphite']
    graphitePort: 2003,
    graphiteHost: “127.0.0.1",
    debug: true,
    legacyNamespace: false
    }
    Sample config

    View Slide

  19. StatsD
    flush interval default every 10s
    Half a System
    Application

    $statsd->increment(“accounts.authentication.login.attempted”);
    stats.counters.accounts.authentication.login.attempted.count
    stats.counters.accounts.authentication.login.attempted.rate
    Added by StatsD
    accounts.authentication.login.attempted:1|c
    23
    2.3

    View Slide

  20. Namespacing Metrics
    https://github.com/etsy/statsd/blob/master/docs/namespacing.md
    http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/
    ...
    accounts.authentication.login.attempted

    accounts.authentication.login.succeeded

    accounts.authentication.login.failed
    ...
    emailvision.production.api.sendEmailResponseTime

    View Slide

  21. Graphite
    • Store numeric time-series data.
    • Render graphs of this data on demand.
    • Powerful function library.

    View Slide

  22. Carbon Configured with /opt/graphite/conf/carbon.conf
    Whisper
    Configured with /opt/graphite/conf/storage-schemas.conf
    Configured with /opt/graphite/conf/storage-aggregation.conf
    Graphite

    View Slide

  23. Whisper
    • Fixed-size database, similar in design to Round Robin Database.
    archive n
    archive 2
    metadata archive 1
    1392589140 25
    1392589150 17
    1392589160 34
    1392589170 68
    Example time series data
    https://github.com/graphite-project/graphite-web/blob/master/docs/whisper.rst

    View Slide

  24. storage-schemas.conf
    Whisper
    http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf
    [stats]
    pattern = ^stats.*
    stats.counters.accounts.authentication.login.attempted.count
    metadata archive 1
    10s:6h
    Frequency History
    retentions = 10s:6h
    1m:7d
    archive 2
    ,1m:7d,
    archive n
    10m:5y
    10m:5y
    • Lowest retention must be the same as the default flush interval for StatsD.

    • Whenever metrics are old enough to leave an archive they get aggregated.

    View Slide

  25. Whisper
    storage-aggregation.conf
    http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-aggregation-conf
    https://github.com/etsy/statsd/blob/master/docs/graphite.md
    [count]
    pattern = \.count$
    aggregationMethod = sum
    xFilesFactor = 0
    • By default whisper uses average so for counter metrics we want the

    aggregation method to be sum.
    • Aggregation methods include average,sum,min,max

    View Slide

  26. Application
    StatsD
    flush interval default every 10s
    Graphite
    Carbon
    Whisper
    Wep app
    carbon.conf
    storage-schema.conf
    Full System

    $statsd->increment(“accounts.authentication.login.attempted”);
    storage-aggregation.conf
    accounts.authentication.login.attempted:1|c

    View Slide

  27. public function authenticate(…)
    {
    ……..
    $statsd->increment(“accounts.authentication.login.attempted”);
    if(password_verify($password, $hash) {
    ……..
    $statsd->increment(“accounts.authentication.login.success”);
    } else {
    …….
    $statsd->increment(“accounts.authentication.login.failed”)
    }
    }
    Counting Metrics continued

    View Slide

  28. Graphite
    stats.counters.accounts.authentication.password.attempted.count
    stats.counters.accounts.authentication.password.failed.count
    stats.counters.accounts.authentication.password.succeeded.count
    Data buckets

    View Slide

  29. Graphite

    View Slide

  30. Graphite

    View Slide

  31. Track every release
    The trick to displaying events in Graphite is to apply the drawAsInfinite() function.
    $statsd->increment(“deploys”);
    echo “deploys:1|c” | nc -w0 -u 127.0.0.1

    View Slide

  32. Other Metric Types
    • Timing
    • Gauges
    • Sets
    https://github.com/etsy/statsd/blob/master/docs/metric_types.md

    View Slide

  33. Timing
    $statsd->startTiming(“emailvision.production.api.sendEmailResponseTime");
    // code which connects and sends email
    $statsd->endTiming("emailvision.production.api.sendEmailResponseTime");
    $statsd->timing("emailvision.production.api.sendEmailResponseTime", 320);
    emailvision.production.api.sendEmailResponseTime:320|ms

    View Slide

  34. Timing metrics under the hood
    Mean - 209 = sum(values)/number of values
    Lower - 200
    Upper - 217
    • StatsD does a lot of aggregation for us.
    Set of data for a 10s period - 200,210,208,207,212,217
    • One timing metric produces 9 data buckets.
    stats.timers.emailvision.production.api.sendEmailResponseTime.lower
    stats.timers.emailvision.production.api.sendEmailResponseTime.mean
    stats.timers.emailvision.production.api.sendEmailResponseTime.upper
    stats.timers.emailvision.production.api.sendEmailResponseTime.mean_90
    stats.timers.emailvision.production.api.sendEmailResponseTime.median
    stats.timers.emailvision.production.api.sendEmailResponseTime.std
    stats.timers.emailvision.production.api.sendEmailResponseTime.sum
    stats.timers.emailvision.production.api.sendEmailResponseTime.sum_90
    stats.timers.emailvision.production.api.sendEmailResponseTime.upper_90

    View Slide

  35. Example timing metrics in Graphite

    View Slide

  36. Example timing metrics in Graphite
    What happened?

    View Slide

  37. http://graphiteurl/render?
    from=-2hours&until=now&width=800&height=600&target=stats.counters.accounts.authentication.password.succee
    ded.count&title=Succesful%20logins&format=json
    Graphite

    View Slide

  38. Anomaly Detection
    http://codeascraft.com/2013/06/11/introducing-kale/
    • Skyline developed by Etsy - https://github.com/etsy/skyline
    • Simple Nagios checks - https://github.com/pyr/check-graphite

    View Slide

  39. Metric From your logs
    https://github.com/etsy/logster
    https://github.com/logstash/logstash
    • Parse log files and send data to statsd

    View Slide

  40. input {
    file {
    path => "/var/log/apache/access.log"
    type => "apache-access"
    }
    }
    Sample logstash config
    http://logstash.net/docs/1.4.2/tutorials/metrics-from-logs
    https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
    filter {
    grok {
    type => "apache-access"
    pattern => "%{COMBINEDAPACHELOG}"
    }
    }
    output {
    statsd {
    # Count one hit every event by response
    increment => "apache.response.%{response}"
    }
    }
    COMBINEDAPACHELOG ……. %{NUMBER:response} ……
    Apache log line - 127.0.0.1 - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326

    View Slide

  41. http://logstash.net/docs/1.4.2/tutorials/metrics-from-logs
    http://cookbook.logstash.net/recipes/statsd-metrics/
    Graphite graph of http response codes

    View Slide

  42. http://codeascraft.com/2010/12/08/track-every-release/
    Metric on PHP Warnings

    View Slide

  43. Metric on PHP Warnings correlated to deploys

    View Slide

  44. Metrics from Logfiles
    • Logfiles contain all types of data so their are many possibilities.

    View Slide

  45. Graphite
    http://graphiteurl/render?
    from=-2hours&until=now&width=800&height=600&target=stats.counters.accounts.a
    uthentication.password.succeeded.count&title=Succesful%20logins
    Render Api - outputs a image
    Useful for building custom dashboards

    http://graphite.readthedocs.org/en/latest/render_api.html

    View Slide

  46. Graphite
    Powerful function library
    http://graphite.readthedocs.org/en/latest/functions.html
    • Timeshift (compare today’s quantity of logins vs those from last weeks)
    • asPercent (compare one metric as a percent of another failed/attempted logins)

    View Slide

  47. StatsD
    Pluggable back-end
    amqp-backend
    ganglia-backend
    librato-backend
    socket.io-backend
    statsd-backend
    mongo-backend
    mysql-backend
    datadog-backend
    opentsdb backend
    influxdb backend
    monitis backend
    instrumental backend
    hosted graphite backend
    statsd aggregation backend
    zabbix-backend
    https://github.com/etsy/statsd/wiki/Backends

    View Slide

  48. Wrapping up
    • Measure all the things.
    • Uses StatsD to collect data and graph it with graphite.
    • Better understanding of your applications.
    • Improve your applications.

    View Slide