Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Brubeck - A high-performance statsd-compatible aggregation daemon

Brubeck - A high-performance statsd-compatible aggregation daemon

70bd372389add5e121b7a9a929b2d887?s=128

Scott Sanders

June 16, 2015
Tweet

Transcript

  1. the best way to build and ship software Brubeck a

    high-performance statsd-compatible aggregation daemon
  2. the best way to build and ship software ! It’s

    all about timing… 2
  3. the best way to build and ship software ! 3

    Measure Everything
  4. the best way to build and ship software “ ”

    ! IT’S NOT SHIPPED UNTIL IT’S FAST. - TED NYMAN Measure Everything 4
  5. the best way to build and ship software ! 5

    How did we make Brubeck fast?
  6. the best way to build and ship software “ ”

    ! BRUBECK’S FEATURE IS THAT IT HAS NO FEATURES. IT IS A VERY SIMPLE C DAEMON WHICH USES STATE-OF-THE-ART TECHNOLOGIES SUCH AS "A WHILE LOOP" AND "MORE THAN ONE THREAD". - VICENT MARTÍ How did we make Brubeck fast? 6
  7. the best way to build and ship software ! 7

    What can it do?
  8. the best way to build and ship software Write data

    to Carbon Brubeck consumes metrics via statsd, samples the metrics, and writes the results to Carbon. 8 !
  9. the best way to build and ship software ! 9

    … " PROTOCOLS SHARDING • Plaintext • Pickle • Consistent hashing to multiple backends Carbon Backend
  10. the best way to build and ship software Rich Logging

    Designed to emit detailed logging information as key=value pairs. 10 !
  11. the best way to build and ship software ! Rich

    Logging 11 jssjr@metrics-­‐collector1-­‐cp1-­‐prd:~$  sudo  tail  -­‐F  /var/log/application.log   2015-­‐06-­‐16T15:00:35-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.catchall]  [38047]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.catchall  sampler=statsd  event=bad_key   key='team.resque.payload_size.job.Jobs.DeliverNotification.:1|c'  from=184.73.32.39   2015-­‐06-­‐16T15:00:35-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.catchall]  [38047]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.catchall  sampler=statsd  event=bad_key   key='team.resque.payload_size.job.Jobs.DeliverNotification.:1|c'  from=184.73.32.39   2015-­‐06-­‐16T15:00:38-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_delayed  now=1434492038  timestamp=1434492033  drift=5   2015-­‐06-­‐16T15:00:53-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_delayed  now=1434492053  timestamp=1434492048  drift=5   2015-­‐06-­‐16T15:01:00-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_future  now=1434492060  timestamp=1434492061
  12. the best way to build and ship software Statsd Sampler

    Read data from statsd clients and store it in memory. 12 !
  13. the best way to build and ship software ! 13

    ⚡ $ CONFIGURE TYPES • Number of worker threads • Enable SO_REUSEPORT • Enable recvmsg(2) • Gauges, meters, counters, histograms, and timers Statsd Sampler
  14. the best way to build and ship software Statsd-secure Sampler

    Just like the statsd sampler, but tamper-proof. 14 !
  15. the best way to build and ship software ! 15

    % & SECURITY PREVENTS • Prepends the plaintext message with a timestamp and a nonce • Generates an HMAC-SHA256 signature using a shared key • Prepends the payload with the signature • Tampering • Replay attacks • Impersonation Statsd-secure Sampler
  16. the best way to build and ship software ! Statsd-secure

    Sampler 16 Payload Message https://github.com/github/statsd-ruby
  17. the best way to build and ship software ! 17

    ⚡ ' CONFIGURE CAVEATS • Drift window • Replay length • Doesn’t encrypt messages • Use a tunnel if this is a requirement • Has a 0.1% probability of dropping a valid message because of the bloom filter implementation Statsd-secure Sampler
  18. the best way to build and ship software Metrics Expiration

    Configure the number of seconds before a metric “expires”. If a new data point hasn’t arrived for a metric in the expiration window it will be removed from memory and samples will no longer be flushed to the backend. 18 !
  19. the best way to build and ship software Metrics Dump

    Signal Brubeck with USR2 and it will record all of the metrics in its table to disk, along with each metric’s type. 19 !
  20. the best way to build and ship software HTTP API

    Interact with Brubeck from other tools and services. 20 !
  21. the best way to build and ship software ! 21

    ( ) GET /PING GET /STATS • return a short JSON payload with the current status of the daemon (just to check it's up) • get a large JSON payload with full statistics, including active endpoints and throughputs HTTP API
  22. the best way to build and ship software ! 22

    * + GET /METRIC/{{METRIC_NAME}} POST /EXPIRE/{{METRIC_NAME}} • get the current status of a metric, if it's being aggregated • expire a metric that is no longer being reported to stop it from being aggregated to the backend HTTP API
  23. the best way to build and ship software Proclines Because

    the command-line is a beautiful place. 23 !
  24. the best way to build and ship software ! Proclines

    24 jssjr@metrics-­‐collector1-­‐cp1-­‐prd:~$  ps  auxwww  |  grep  [b]rubeck   brubeck      38043    395    0.0  802596  80092  ?                Ssl    Jun04  71997:07  brubeck  -­‐-­‐   [config.github]  [  ↑  #1  539.5gb,  #2  548.0gb,  #3  531.2gb,  #4  551.0gb,  #5  545.0gb  ]  [  ↓  : 8226  501593/s  ]   brubeck      38045    2.9    0.0  542148  10176  ?                Ssl    Jun04  535:13  brubeck  -­‐-­‐   [config.secure]  [  ↑  #1  67.1gb,  #2  71.0gb,  #3  64.1gb,  #4  69.8gb,  #5  67.1gb  ]  [  ↓  :9126   1131/s  ]   burbeck      38047    112    0.0  793308  94252  ?                Ssl    Jun04  20540:24  brubeck  -­‐-­‐   [config.catchall]  [  ↑  #1  232.0gb,  #2  234.2gb,  #3  232.2gb,  #4  234.2gb,  #5  235.5gb  ]  [  ↓  : 8126  47292/s  ]
  25. the best way to build and ship software Limitations Not

    everything is actually compatible. Sorry. 25 !
  26. the best way to build and ship software ! 26

    x - DOESN’T SUPPORT DIFFERENCES • Multi-line messages • Statsd sample timing (foo:1|ms|@0.1) • The set type • No stats and stats_count namespace split • Histograms are slightly different, let me show you… Limitations
  27. the best way to build and ship software ! Histogram

    Differences 27 Esty Statsd GitHub Brubeck
  28. the best way to build and ship software Thanks! Scott

    Sanders Infrastructure Engineer @ GitHub https://github.com/jssjr https://twitter.com/scott_sanders