Logging Patterns with Logstash and Chef

Long-form version of my #chefconf 2012 logstash talk.

1. Logging patterns with Logging patterns with Logstash and Chef Logstash and Chef
2. What is Logstash? What is Logstash? • Log management framework • Actually been around a while (2008ish) • Started by Jordan Sissel and Pete Fritchman • Stuff comes in • Bits get twiddled • Stuff comes out
3. What is Logstash? What is Logstash? • Log management framework • Actually been around a while (2008ish) • Started by Jordan Sissel and Pete Fritchman • Stuff comes in • Bits get twiddled • Stuff comes out INPUT FILTER OUTPUT
4. It takes stuff like this It takes stuff like this May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
5. And turns it into this... And turns it into this...
6. And this.. And this..
7. Oh and this... Oh and this...
9. Just one tiny seed Just one tiny seed
10. Log line/JSON message/observed thingie + + The timestamp when it happened
11. Inputs Inputs This is where events enter the pipeline e.g. the source (and optionally the format) • STDIN • Twitter search • Socket • ZeroMQ • Heroku • AMQP • Currently 14 inputs in MASTER
12. Quick note on Input “formats” Quick note on Input “formats” • Assumed format for incoming events is defined by the plugin • Most plugins assume plain text • Others assume JSON • Some speak 'json_event'* * I'll get to this in a moment
13. Example input definition Example input definition input { file { type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"] sincedb_path => "/opt/logstash/agent/etc/" } }
14. input { file { type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"] sincedb_path => "/opt/logstash/agent/etc/" } } NOT RUBY! NOT RUBY!
15. Hash ALL the things Hash ALL the things • Once event is received, converted to Ruby hash for the remainder of the pipeline { "@source" => "<format determined by input>", "@type" => "<type from input>", "@tags" => [], "@fields" => {}, "@timestamp" => "<ISO8601 of received event>", "@source_host" => "<localhost or source hostname>", "@source_path" => "<value determined by input>", "@message" => "<the escaped representation of the line>" }
16. Hey man, nice shot! Hey man, nice shot! • Filters are where you do the work • Break the “@message @message” into constituent parts • Identify original timestamp • Add tags • Move parts around • External processing via 0mq • Currently 13 filters in MASTER
17. Grok and Roll Grok and Roll • DRY and RAD for RegEx • Originally a C library • Pure Ruby version in Logstash since 1.1 • Identify patterns, attach identifier
18. Remember this? Remember this? May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
19. Enhance 224 to 176 Enhance 224 to 176 May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12- 15/admin/Job HTTP/1.1" 15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
20. Center and Stop Center and Stop May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api- be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
21. Enhance 57 to 19. Stop Enhance 57 to 19. Stop May 11 06:00:27 May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api- be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
22. Give me a hard copy right there Give me a hard copy right there May 11 06:00:27 May 11 06:00:27
23. Let's analyze this Let's analyze this May 11 06:00:27 May 11 06:00:27 filter { grok { type => "haproxy-log" patterns_dir => ["/opt/logstash/agent/etc/patterns/"] pattern => "%{ESHAPROXYHTTP}" add_tag => ["haproxy-event"] } }
24. ESHAPROXYHTTP ESHAPROXYHTTP It's REALLY long It's REALLY long
25. ESHAPROXYCAPTUREDREQUESTHEADERS % {DATA:request_header_x_forwarded_for} ESHAPROXYCAPTUREDRESPONSEHEADERS % {DATA:response_header_content_type}\|% {DATA:response_header_content_encoding}\|% {DATA:response_header_cache_control}\|% {DATA:response_header_last_modified} ESHAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} % {IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:% {INT:client_port} \[%{HAPROXYDATE:accept_date}\] % {NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/% {NOTSPACE:server_name} %{INT:time_request}/% {INT:time_queue}/%{INT:time_backend_connect}/% {INT:time_backend_response}/\+%{NOTSPACE:time_duration} % {INT:http_status_code} \+%{NOTSPACE:bytes_read} % {DATA:captured_request_cookie} % {DATA:captured_response_cookie} % {NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/% {INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} % {INT:srv_queue}/%{INT:backend_queue} \{% {ESHAPROXYCAPTUREDREQUESTHEADERS}\} "%{WORD:http_verb} % {URIPATHPARAM:http_request} HTTP/%{NUMBER:http_version}"
26. ESHAPROXYHTTP ESHAPROXYHTTP We're going to focus on the first part %{SYSLOGTIMESTAMP:syslog_timestamp}
27. What's in a SYSLOGTIMESTAMP? What's in a SYSLOGTIMESTAMP?
28. SYSLOGTIMESTAMP SYSLOGTIMESTAMP May 11 06:00:27 May 11 06:00:27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} • MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) • TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) • HOUR (?:2[0123]|[01][0-9]) HOUR (?:2[0123]|[01][0-9]) • MINUTE (?:[0-5][0-9]) MINUTE (?:[0-5][0-9]) • SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?) SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
29. MONTH, MONTHDAY, TIME MONTH, MONTHDAY, TIME May May 11 11 06:00:27 06:00:27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{ %{MONTH MONTH} +%{ } +%{MONTHDAY MONTHDAY} %{ } %{TIME TIME} } • MONTH MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)? \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| | May| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY MONTHDAY (?:(?:0[1-9])| (?:(?:0[1-9])|(?:[12][0-9]) (?:[12][0-9])|(?:3[01])|[1-9]) |(?:3[01])|[1-9]) • TIME TIME (?!<[0-9]) (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND}) %{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) (?![0-9]) • HOUR (?:2[0123]|[01][0-9]) HOUR (?:2[0123]|[01][0-9]) • MINUTE (?:[0-5][0-9]) MINUTE (?:[0-5][0-9]) • SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?) SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
30. HOUR, MINUTE, SECOND HOUR, MINUTE, SECOND May 11 May 11 06 06: :00 00: :27 27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} • MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) • TIME (?!<[0-9])%{ TIME (?!<[0-9])%{HOUR HOUR}:%{ }:%{MINUTE MINUTE}(?::%{ }(?::%{SECOND SECOND})(?![0-9]) })(?![0-9]) • HOUR HOUR (?:2[0123]| (?:2[0123]|[01][0-9] [01][0-9]) ) • MINUTE MINUTE (?: (?:[0-5][0-9] [0-5][0-9]) ) • SECOND SECOND (?:(?: (?:(?:[0-5][0-9] [0-5][0-9]|60)(?:[.,][0-9]+)?) |60)(?:[.,][0-9]+)?)
31. Name a Thing Name a Thing May 11 06:00:27 May 11 06:00:27 • %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP: :syslog_timestamp syslog_timestamp} } • SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} • MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b • MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) • TIME (?!<[0-9])%{ TIME (?!<[0-9])%{HOUR HOUR}:%{ }:%{MINUTE MINUTE}(?::%{ }(?::%{SECOND SECOND})(?![0-9]) })(?![0-9]) • HOUR HOUR (?:2[0123]| (?:2[0123]|[01][0-9] [01][0-9]) ) • MINUTE MINUTE (?: (?:[0-5][0-9] [0-5][0-9]) ) • SECOND SECOND (?:(?: (?:(?:[0-5][0-9] [0-5][0-9]|60)(?:[.,][0-9]+)?) |60)(?:[.,][0-9]+)?)
32. Named fields Named fields • When you 'identify' something, it is added to the hash under the @fields key { "@source" => "<format determined by input>", "@type" => "<type from input>", "@tags" => [“haproxy_event”], "@fields" => {“syslog_timestamp” => “May 11 06:00:27”}, "@timestamp" => "<ISO8601 of received event>", "@source_host" => "<localhost or source hostname>", "@source_path" => "<value determined by input>", "@message" => "<the escaped representation of the line>" }
33. Rinse/Repeat Rinse/Repeat • Filters are processed in order • Once a field is identified, can be used in interpolation later • %{syslog_timestamp} • %{@type} • @ fields are special but not sacred. • date and mutate filters for instance.
34. Get out(put) Get out(put) • An event can be routed to all or a subset of defined outputs based on various criteria • Outputs block (sort of) • Logstash takes a default position that you don't want to lose an event • 1 thread per defined output each • ALWAYS use a stdout output for debugging • This is where it gets REALLY cool • Currently 27 outputs in MASTER
35. Graphite Example Graphite Example output { graphite { host => "1.1.1.1" metrics => [ "stats.enstratus.% {@source_host}.haproxy.% {server_name}.request_type.% {http_verb}", "1"] tags => ["haproxy-event"] } }
36. What does it mean? What does it mean? • When a message is tagged: “haproxy-event” • I want to write to Graphite: A value of 1 • as 'stats.enstratus.X.request_type.Y' • Where X is the source of the event • And Y is the HTTP verb
37. End Result End Result
38. End Result End Result
39. But we're also storing the events But we're also storing the events elasticsearch { cluster => 'logstash' host => 'es-server' port => 9300 }
40. And publishing them for remote And publishing them for remote tailing tailing zeromq { topology => "pubsub" address => "tcp://*:5558" mode => "server" topic => "%{@source_host}.% {level}.%{log}" }
41. Where does Chef fit in? Where does Chef fit in?
42. Let's use a familiar example Let's use a familiar example • Default Chef Handler JSON files • Parse with Logstash • Apply some filters • Send to some places
43. Chef-handler JSON logs Chef-handler JSON logs • We're not going to be concerned with how you get them INTO logstash • chef-gelf handler works (logstash has a gelf input) • You can write your own (I'm partial to ZeroMQ!) • Set your input type to “json” • Set the “type” to something that flags it as a chef event. • If you send the WHOLE thing, be prepared to cut some stuff (you don't really want the Ohai data in your logs)
44. What do we care about? What do we care about? { “node”:{“name”:”foo”}, “success”: true, “start_time”:”2012-05-14 01:09:31 +0000”, “end_time”:”2012-05-14 01:10:46 +0000”, “elapsed_time”:”1.14”, “updated_resources”:[], “exception”:””, “backtrace”:”” }
45. Apply a date filter Apply a date filter (set @timestamp to start_time) (set @timestamp to start_time) date { start_time => “yyyy-MM-dd HH:mm:ss Z”, type => “chef_handler” }
46. Apply a mutate filter Apply a mutate filter (set @message) (set @message) mutate { replace => [“@message”, % {exception}], type => “chef_handler” }
47. Apply a mutate filter Apply a mutate filter (convert status) (convert status) mutate { convert => [“success”, “string”], type => “chef_handler” }
48. Send result and timing to Send result and timing to Graphite Graphite graphite { metrics => [“stats.% {@source_host}.chef_run.success.% {success}”,”1”, “stats.% {@source_host}.chef_run.duration”, % {elapsed_time}], type => “chef_handler” }
49. A better approach A better approach • Create a custom handler • Build custom JSON from data • Strip the extra stuff • Join stack trace array elements into a single newline-separated value • Send to Logstash and fanout from there
50. Logstash Cookbook Logstash Cookbook • github.com/lusis-cookbooks/logstash • runit based • Agent, Server, Web, PyShipper recipes • Also Kibana (alternate web interface) • No LWRP just yet =(
51. The Future The Future • Run-time Configuration Changes • No more restarting • Push support • MOAR PLUGINS! • Internal Metrics Channel • input { metrics } • Improved AMQP Support • Your imagination
52. AMA AMA • Questions? • Tips for your peers? • Hate mail? • You can ask me later if you'd like
53. Some links Some links • logstash.net • cookbook.logstash.net • github.com/logstash • github.com/lusis/logstash-shipper
54. Thanks! Thanks! • github.com/lusis • @lusis • #monitoringsucks, #dadops