$30 off During Our Annual Pro Sale. View Details »

Brubeck - A high-performance statsd-compatible ...

Brubeck - A high-performance statsd-compatible aggregation daemon

Scott Sanders

June 16, 2015
Tweet

More Decks by Scott Sanders

Other Decks in Technology

Transcript

  1. the best way to build and ship software Brubeck a

    high-performance statsd-compatible aggregation daemon
  2. the best way to build and ship software “ ”

    ! IT’S NOT SHIPPED UNTIL IT’S FAST. - TED NYMAN Measure Everything 4
  3. the best way to build and ship software ! 5

    How did we make Brubeck fast?
  4. the best way to build and ship software “ ”

    ! BRUBECK’S FEATURE IS THAT IT HAS NO FEATURES. IT IS A VERY SIMPLE C DAEMON WHICH USES STATE-OF-THE-ART TECHNOLOGIES SUCH AS "A WHILE LOOP" AND "MORE THAN ONE THREAD". - VICENT MARTÍ How did we make Brubeck fast? 6
  5. the best way to build and ship software Write data

    to Carbon Brubeck consumes metrics via statsd, samples the metrics, and writes the results to Carbon. 8 !
  6. the best way to build and ship software ! 9

    … " PROTOCOLS SHARDING • Plaintext • Pickle • Consistent hashing to multiple backends Carbon Backend
  7. the best way to build and ship software Rich Logging

    Designed to emit detailed logging information as key=value pairs. 10 !
  8. the best way to build and ship software ! Rich

    Logging 11 jssjr@metrics-­‐collector1-­‐cp1-­‐prd:~$  sudo  tail  -­‐F  /var/log/application.log   2015-­‐06-­‐16T15:00:35-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.catchall]  [38047]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.catchall  sampler=statsd  event=bad_key   key='team.resque.payload_size.job.Jobs.DeliverNotification.:1|c'  from=184.73.32.39   2015-­‐06-­‐16T15:00:35-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.catchall]  [38047]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.catchall  sampler=statsd  event=bad_key   key='team.resque.payload_size.job.Jobs.DeliverNotification.:1|c'  from=184.73.32.39   2015-­‐06-­‐16T15:00:38-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_delayed  now=1434492038  timestamp=1434492033  drift=5   2015-­‐06-­‐16T15:00:53-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_delayed  now=1434492053  timestamp=1434492048  drift=5   2015-­‐06-­‐16T15:01:00-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_future  now=1434492060  timestamp=1434492061
  9. the best way to build and ship software Statsd Sampler

    Read data from statsd clients and store it in memory. 12 !
  10. the best way to build and ship software ! 13

    ⚡ $ CONFIGURE TYPES • Number of worker threads • Enable SO_REUSEPORT • Enable recvmsg(2) • Gauges, meters, counters, histograms, and timers Statsd Sampler
  11. the best way to build and ship software Statsd-secure Sampler

    Just like the statsd sampler, but tamper-proof. 14 !
  12. the best way to build and ship software ! 15

    % & SECURITY PREVENTS • Prepends the plaintext message with a timestamp and a nonce • Generates an HMAC-SHA256 signature using a shared key • Prepends the payload with the signature • Tampering • Replay attacks • Impersonation Statsd-secure Sampler
  13. the best way to build and ship software ! Statsd-secure

    Sampler 16 Payload Message https://github.com/github/statsd-ruby
  14. the best way to build and ship software ! 17

    ⚡ ' CONFIGURE CAVEATS • Drift window • Replay length • Doesn’t encrypt messages • Use a tunnel if this is a requirement • Has a 0.1% probability of dropping a valid message because of the bloom filter implementation Statsd-secure Sampler
  15. the best way to build and ship software Metrics Expiration

    Configure the number of seconds before a metric “expires”. If a new data point hasn’t arrived for a metric in the expiration window it will be removed from memory and samples will no longer be flushed to the backend. 18 !
  16. the best way to build and ship software Metrics Dump

    Signal Brubeck with USR2 and it will record all of the metrics in its table to disk, along with each metric’s type. 19 !
  17. the best way to build and ship software HTTP API

    Interact with Brubeck from other tools and services. 20 !
  18. the best way to build and ship software ! 21

    ( ) GET /PING GET /STATS • return a short JSON payload with the current status of the daemon (just to check it's up) • get a large JSON payload with full statistics, including active endpoints and throughputs HTTP API
  19. the best way to build and ship software ! 22

    * + GET /METRIC/{{METRIC_NAME}} POST /EXPIRE/{{METRIC_NAME}} • get the current status of a metric, if it's being aggregated • expire a metric that is no longer being reported to stop it from being aggregated to the backend HTTP API
  20. the best way to build and ship software Proclines Because

    the command-line is a beautiful place. 23 !
  21. the best way to build and ship software ! Proclines

    24 jssjr@metrics-­‐collector1-­‐cp1-­‐prd:~$  ps  auxwww  |  grep  [b]rubeck   brubeck      38043    395    0.0  802596  80092  ?                Ssl    Jun04  71997:07  brubeck  -­‐-­‐   [config.github]  [  ↑  #1  539.5gb,  #2  548.0gb,  #3  531.2gb,  #4  551.0gb,  #5  545.0gb  ]  [  ↓  : 8226  501593/s  ]   brubeck      38045    2.9    0.0  542148  10176  ?                Ssl    Jun04  535:13  brubeck  -­‐-­‐   [config.secure]  [  ↑  #1  67.1gb,  #2  71.0gb,  #3  64.1gb,  #4  69.8gb,  #5  67.1gb  ]  [  ↓  :9126   1131/s  ]   burbeck      38047    112    0.0  793308  94252  ?                Ssl    Jun04  20540:24  brubeck  -­‐-­‐   [config.catchall]  [  ↑  #1  232.0gb,  #2  234.2gb,  #3  232.2gb,  #4  234.2gb,  #5  235.5gb  ]  [  ↓  : 8126  47292/s  ]
  22. the best way to build and ship software Limitations Not

    everything is actually compatible. Sorry. 25 !
  23. the best way to build and ship software ! 26

    x - DOESN’T SUPPORT DIFFERENCES • Multi-line messages • Statsd sample timing (foo:1|ms|@0.1) • The set type • No stats and stats_count namespace split • Histograms are slightly different, let me show you… Limitations
  24. the best way to build and ship software ! Histogram

    Differences 27 Esty Statsd GitHub Brubeck
  25. the best way to build and ship software Thanks! Scott

    Sanders Infrastructure Engineer @ GitHub https://github.com/jssjr https://twitter.com/scott_sanders