$30 off During Our Annual Pro Sale. View Details »

Logging Patterns with Logstash and Chef

John Vincent
May 16, 2012
5k

Logging Patterns with Logstash and Chef

Long-form version of my #chefconf 2012 logstash talk.

John Vincent

May 16, 2012
Tweet

Transcript

  1. Logging patterns with Logging patterns with Logstash and Chef Logstash

    and Chef
  2. What is Logstash? What is Logstash? • Log management framework

    • Actually been around a while (2008ish) • Started by Jordan Sissel and Pete Fritchman • Stuff comes in • Bits get twiddled • Stuff comes out
  3. What is Logstash? What is Logstash? • Log management framework

    • Actually been around a while (2008ish) • Started by Jordan Sissel and Pete Fritchman • Stuff comes in • Bits get twiddled • Stuff comes out INPUT FILTER OUTPUT
  4. It takes stuff like this It takes stuff like this

    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
  5. And turns it into this... And turns it into this...

  6. And this.. And this..

  7. Oh and this... Oh and this...

  8. None
  9. Just one tiny seed Just one tiny seed

  10. Log line/JSON message/observed thingie + + The timestamp when it

    happened
  11. Inputs Inputs This is where events enter the pipeline e.g.

    the source (and optionally the format) • STDIN • Twitter search • Socket • ZeroMQ • Heroku • AMQP • Currently 14 inputs in MASTER
  12. Quick note on Input “formats” Quick note on Input “formats”

    • Assumed format for incoming events is defined by the plugin • Most plugins assume plain text • Others assume JSON • Some speak 'json_event'* * I'll get to this in a moment
  13. Example input definition Example input definition input { file {

    type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"] sincedb_path => "/opt/logstash/agent/etc/" } }
  14. input { file { type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"]

    sincedb_path => "/opt/logstash/agent/etc/" } } NOT RUBY! NOT RUBY!
  15. Hash ALL the things Hash ALL the things • Once

    event is received, converted to Ruby hash for the remainder of the pipeline { "@source" => "<format determined by input>", "@type" => "<type from input>", "@tags" => [], "@fields" => {}, "@timestamp" => "<ISO8601 of received event>", "@source_host" => "<localhost or source hostname>", "@source_path" => "<value determined by input>", "@message" => "<the escaped representation of the line>" }
  16. Hey man, nice shot! Hey man, nice shot! • Filters

    are where you do the work • Break the “@message @message” into constituent parts • Identify original timestamp • Add tags • Move parts around • External processing via 0mq • Currently 13 filters in MASTER
  17. Grok and Roll Grok and Roll • DRY and RAD

    for RegEx • Originally a C library • Pure Ruby version in Logstash since 1.1 • Identify patterns, attach identifier
  18. Remember this? Remember this? May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453

    [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
  19. Enhance 224 to 176 Enhance 224 to 176 May 11

    06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12- 15/admin/Job HTTP/1.1" 15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
  20. Center and Stop Center and Stop May 11 06:00:27 localhost

    haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api- be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
  21. Enhance 57 to 19. Stop Enhance 57 to 19. Stop

    May 11 06:00:27 May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api- be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
  22. Give me a hard copy right there Give me a

    hard copy right there May 11 06:00:27 May 11 06:00:27
  23. Let's analyze this Let's analyze this May 11 06:00:27 May

    11 06:00:27 filter { grok { type => "haproxy-log" patterns_dir => ["/opt/logstash/agent/etc/patterns/"] pattern => "%{ESHAPROXYHTTP}" add_tag => ["haproxy-event"] } }
  24. ESHAPROXYHTTP ESHAPROXYHTTP It's REALLY long It's REALLY long

  25. ESHAPROXYCAPTUREDREQUESTHEADERS % {DATA:request_header_x_forwarded_for} ESHAPROXYCAPTUREDRESPONSEHEADERS % {DATA:response_header_content_type}\|% {DATA:response_header_content_encoding}\|% {DATA:response_header_cache_control}\|% {DATA:response_header_last_modified} ESHAPROXYHTTP

    %{SYSLOGTIMESTAMP:syslog_timestamp} % {IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:% {INT:client_port} \[%{HAPROXYDATE:accept_date}\] % {NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/% {NOTSPACE:server_name} %{INT:time_request}/% {INT:time_queue}/%{INT:time_backend_connect}/% {INT:time_backend_response}/\+%{NOTSPACE:time_duration} % {INT:http_status_code} \+%{NOTSPACE:bytes_read} % {DATA:captured_request_cookie} % {DATA:captured_response_cookie} % {NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/% {INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} % {INT:srv_queue}/%{INT:backend_queue} \{% {ESHAPROXYCAPTUREDREQUESTHEADERS}\} "%{WORD:http_verb} % {URIPATHPARAM:http_request} HTTP/%{NUMBER:http_version}"
  26. ESHAPROXYHTTP ESHAPROXYHTTP We're going to focus on the first part

    %{SYSLOGTIMESTAMP:syslog_timestamp}
  27. What's in a SYSLOGTIMESTAMP? What's in a SYSLOGTIMESTAMP?

  28. SYSLOGTIMESTAMP SYSLOGTIMESTAMP May 11 06:00:27 May 11 06:00:27 • %{

    %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} • MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) • TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) • HOUR (?:2[0123]|[01][0-9]) HOUR (?:2[0123]|[01][0-9]) • MINUTE (?:[0-5][0-9]) MINUTE (?:[0-5][0-9]) • SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?) SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
  29. MONTH, MONTHDAY, TIME MONTH, MONTHDAY, TIME May May 11 11

    06:00:27 06:00:27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{ %{MONTH MONTH} +%{ } +%{MONTHDAY MONTHDAY} %{ } %{TIME TIME} } • MONTH MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)? \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| | May| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY MONTHDAY (?:(?:0[1-9])| (?:(?:0[1-9])|(?:[12][0-9]) (?:[12][0-9])|(?:3[01])|[1-9]) |(?:3[01])|[1-9]) • TIME TIME (?!<[0-9]) (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND}) %{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) (?![0-9]) • HOUR (?:2[0123]|[01][0-9]) HOUR (?:2[0123]|[01][0-9]) • MINUTE (?:[0-5][0-9]) MINUTE (?:[0-5][0-9]) • SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?) SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
  30. HOUR, MINUTE, SECOND HOUR, MINUTE, SECOND May 11 May 11

    06 06: :00 00: :27 27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} • MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) • TIME (?!<[0-9])%{ TIME (?!<[0-9])%{HOUR HOUR}:%{ }:%{MINUTE MINUTE}(?::%{ }(?::%{SECOND SECOND})(?![0-9]) })(?![0-9]) • HOUR HOUR (?:2[0123]| (?:2[0123]|[01][0-9] [01][0-9]) ) • MINUTE MINUTE (?: (?:[0-5][0-9] [0-5][0-9]) ) • SECOND SECOND (?:(?: (?:(?:[0-5][0-9] [0-5][0-9]|60)(?:[.,][0-9]+)?) |60)(?:[.,][0-9]+)?)
  31. Name a Thing Name a Thing May 11 06:00:27 May

    11 06:00:27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP: :syslog_timestamp syslog_timestamp} } • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} • MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) • TIME (?!<[0-9])%{ TIME (?!<[0-9])%{HOUR HOUR}:%{ }:%{MINUTE MINUTE}(?::%{ }(?::%{SECOND SECOND})(?![0-9]) })(?![0-9]) • HOUR HOUR (?:2[0123]| (?:2[0123]|[01][0-9] [01][0-9]) ) • MINUTE MINUTE (?: (?:[0-5][0-9] [0-5][0-9]) ) • SECOND SECOND (?:(?: (?:(?:[0-5][0-9] [0-5][0-9]|60)(?:[.,][0-9]+)?) |60)(?:[.,][0-9]+)?)
  32. Named fields Named fields • When you 'identify' something, it

    is added to the hash under the @fields key { "@source" => "<format determined by input>", "@type" => "<type from input>", "@tags" => [“haproxy_event”], "@fields" => {“syslog_timestamp” => “May 11 06:00:27”}, "@timestamp" => "<ISO8601 of received event>", "@source_host" => "<localhost or source hostname>", "@source_path" => "<value determined by input>", "@message" => "<the escaped representation of the line>" }
  33. Rinse/Repeat Rinse/Repeat • Filters are processed in order • Once

    a field is identified, can be used in interpolation later • %{syslog_timestamp} • %{@type} • @ fields are special but not sacred. • date and mutate filters for instance.
  34. Get out(put) Get out(put) • An event can be routed

    to all or a subset of defined outputs based on various criteria • Outputs block (sort of) • Logstash takes a default position that you don't want to lose an event • 1 thread per defined output each • ALWAYS use a stdout output for debugging • This is where it gets REALLY cool • Currently 27 outputs in MASTER
  35. Graphite Example Graphite Example output { graphite { host =>

    "1.1.1.1" metrics => [ "stats.enstratus.% {@source_host}.haproxy.% {server_name}.request_type.% {http_verb}", "1"] tags => ["haproxy-event"] } }
  36. What does it mean? What does it mean? • When

    a message is tagged: “haproxy-event” • I want to write to Graphite: A value of 1 • as 'stats.enstratus.X.request_type.Y' • Where X is the source of the event • And Y is the HTTP verb
  37. End Result End Result

  38. End Result End Result

  39. But we're also storing the events But we're also storing

    the events elasticsearch { cluster => 'logstash' host => 'es-server' port => 9300 }
  40. And publishing them for remote And publishing them for remote

    tailing tailing zeromq { topology => "pubsub" address => "tcp://*:5558" mode => "server" topic => "%{@source_host}.% {level}.%{log}" }
  41. Where does Chef fit in? Where does Chef fit in?

  42. Let's use a familiar example Let's use a familiar example

    • Default Chef Handler JSON files • Parse with Logstash • Apply some filters • Send to some places
  43. Chef-handler JSON logs Chef-handler JSON logs • We're not going

    to be concerned with how you get them INTO logstash • chef-gelf handler works (logstash has a gelf input) • You can write your own (I'm partial to ZeroMQ!) • Set your input type to “json” • Set the “type” to something that flags it as a chef event. • If you send the WHOLE thing, be prepared to cut some stuff (you don't really want the Ohai data in your logs)
  44. What do we care about? What do we care about?

    { “node”:{“name”:”foo”}, “success”: true, “start_time”:”2012-05-14 01:09:31 +0000”, “end_time”:”2012-05-14 01:10:46 +0000”, “elapsed_time”:”1.14”, “updated_resources”:[], “exception”:””, “backtrace”:”” }
  45. Apply a date filter Apply a date filter (set @timestamp

    to start_time) (set @timestamp to start_time) date { start_time => “yyyy-MM-dd HH:mm:ss Z”, type => “chef_handler” }
  46. Apply a mutate filter Apply a mutate filter (set @message)

    (set @message) mutate { replace => [“@message”, % {exception}], type => “chef_handler” }
  47. Apply a mutate filter Apply a mutate filter (convert status)

    (convert status) mutate { convert => [“success”, “string”], type => “chef_handler” }
  48. Send result and timing to Send result and timing to

    Graphite Graphite graphite { metrics => [“stats.% {@source_host}.chef_run.success.% {success}”,”1”, “stats.% {@source_host}.chef_run.duration”, % {elapsed_time}], type => “chef_handler” }
  49. A better approach A better approach • Create a custom

    handler • Build custom JSON from data • Strip the extra stuff • Join stack trace array elements into a single newline-separated value • Send to Logstash and fanout from there
  50. Logstash Cookbook Logstash Cookbook • github.com/lusis-cookbooks/logstash • runit based •

    Agent, Server, Web, PyShipper recipes • Also Kibana (alternate web interface) • No LWRP just yet =(
  51. The Future The Future • Run-time Configuration Changes • No

    more restarting • Push support • MOAR PLUGINS! • Internal Metrics Channel • input { metrics } • Improved AMQP Support • Your imagination
  52. AMA AMA • Questions? • Tips for your peers? •

    Hate mail? • You can ask me later if you'd like
  53. Some links Some links • logstash.net • cookbook.logstash.net • github.com/logstash

    • github.com/lusis/logstash-shipper
  54. Thanks! Thanks! • github.com/lusis • @lusis • #monitoringsucks, #dadops