Tweet
Tweet

Slide 1

Slide 1 text

Logging patterns with Logging patterns with Logstash and Chef Logstash and Chef

Slide 2

Slide 2 text

What is Logstash? What is Logstash? ● Log management framework ● Actually been around a while (2008ish) ● Started by Jordan Sissel and Pete Fritchman ● Stuff comes in ● Bits get twiddled ● Stuff comes out

Slide 3

Slide 3 text

What is Logstash? What is Logstash? ● Log management framework ● Actually been around a while (2008ish) ● Started by Jordan Sissel and Pete Fritchman ● Stuff comes in ● Bits get twiddled ● Stuff comes out INPUT FILTER OUTPUT

Slide 4

Slide 4 text

It takes stuff like this It takes stuff like this May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

Slide 5

Slide 5 text

And turns it into this... And turns it into this...

Slide 6

Slide 6 text

And this.. And this..

Slide 7

Slide 7 text

Oh and this... Oh and this...

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Just one tiny seed Just one tiny seed

Slide 10

Slide 10 text

Log line/JSON message/observed thingie + + The timestamp when it happened

Slide 11

Slide 11 text

Inputs Inputs This is where events enter the pipeline e.g. the source (and optionally the format) ● STDIN ● Twitter search ● Socket ● ZeroMQ ● Heroku ● AMQP ● Currently 14 inputs in MASTER

Slide 12

Slide 12 text

Quick note on Input “formats” Quick note on Input “formats” ● Assumed format for incoming events is defined by the plugin ● Most plugins assume plain text ● Others assume JSON ● Some speak 'json_event'* * I'll get to this in a moment

Slide 13

Slide 13 text

Example input definition Example input definition input { file { type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"] sincedb_path => "/opt/logstash/agent/etc/" } }

Slide 14

Slide 14 text

input { file { type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"] sincedb_path => "/opt/logstash/agent/etc/" } } NOT RUBY! NOT RUBY!

Slide 15

Slide 15 text

Hash ALL the things Hash ALL the things ● Once event is received, converted to Ruby hash for the remainder of the pipeline { "@source" => "", "@type" => "", "@tags" => [], "@fields" => {}, "@timestamp" => "", "@source_host" => "", "@source_path" => "", "@message" => "" }

Slide 16

Slide 16 text

Hey man, nice shot! Hey man, nice shot! ● Filters are where you do the work ● Break the “@message @message” into constituent parts ● Identify original timestamp ● Add tags ● Move parts around ● External processing via 0mq ● Currently 13 filters in MASTER

Slide 17

Slide 17 text

Grok and Roll Grok and Roll ● DRY and RAD for RegEx ● Originally a C library ● Pure Ruby version in Logstash since 1.1 ● Identify patterns, attach identifier

Slide 18

Slide 18 text

Remember this? Remember this? May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

Slide 19

Slide 19 text

Enhance 224 to 176 Enhance 224 to 176 May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282] api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282] api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282] api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282] api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282] api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12- 15/admin/Job HTTP/1.1" 15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034] api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013] api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160] api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1" May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074] api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

Slide 20

Slide 20 text

Center and Stop Center and Stop May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api- be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

Slide 21

Slide 21 text

Enhance 57 to 19. Stop Enhance 57 to 19. Stop May 11 06:00:27 May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe api- be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

Slide 22

Slide 22 text

Give me a hard copy right there Give me a hard copy right there May 11 06:00:27 May 11 06:00:27

Slide 23

Slide 23 text

Let's analyze this Let's analyze this May 11 06:00:27 May 11 06:00:27 filter { grok { type => "haproxy-log" patterns_dir => ["/opt/logstash/agent/etc/patterns/"] pattern => "%{ESHAPROXYHTTP}" add_tag => ["haproxy-event"] } }

Slide 24

Slide 24 text

ESHAPROXYHTTP ESHAPROXYHTTP It's REALLY long It's REALLY long

Slide 25

Slide 25 text

ESHAPROXYCAPTUREDREQUESTHEADERS % {DATA:request_header_x_forwarded_for} ESHAPROXYCAPTUREDRESPONSEHEADERS % {DATA:response_header_content_type}\|% {DATA:response_header_content_encoding}\|% {DATA:response_header_cache_control}\|% {DATA:response_header_last_modified} ESHAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} % {IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:% {INT:client_port} \[%{HAPROXYDATE:accept_date}\] % {NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/% {NOTSPACE:server_name} %{INT:time_request}/% {INT:time_queue}/%{INT:time_backend_connect}/% {INT:time_backend_response}/\+%{NOTSPACE:time_duration} % {INT:http_status_code} \+%{NOTSPACE:bytes_read} % {DATA:captured_request_cookie} % {DATA:captured_response_cookie} % {NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/% {INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} % {INT:srv_queue}/%{INT:backend_queue} \{% {ESHAPROXYCAPTUREDREQUESTHEADERS}\} "%{WORD:http_verb} % {URIPATHPARAM:http_request} HTTP/%{NUMBER:http_version}"

Slide 26

Slide 26 text

ESHAPROXYHTTP ESHAPROXYHTTP We're going to focus on the first part %{SYSLOGTIMESTAMP:syslog_timestamp}

Slide 27

Slide 27 text

What's in a SYSLOGTIMESTAMP? What's in a SYSLOGTIMESTAMP?

Slide 28

Slide 28 text

SYSLOGTIMESTAMP SYSLOGTIMESTAMP May 11 06:00:27 May 11 06:00:27 ● %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} ● SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} ● MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b ● MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) ● TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) ● HOUR (?:2[0123]|[01][0-9]) HOUR (?:2[0123]|[01][0-9]) ● MINUTE (?:[0-5][0-9]) MINUTE (?:[0-5][0-9]) ● SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?) SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)

Slide 29

Slide 29 text

MONTH, MONTHDAY, TIME MONTH, MONTHDAY, TIME May May 11 11 06:00:27 06:00:27 ● %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} ● SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{ %{MONTH MONTH} +%{ } +%{MONTHDAY MONTHDAY} %{ } %{TIME TIME} } ● MONTH MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)? \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| | May| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b ● MONTHDAY MONTHDAY (?:(?:0[1-9])| (?:(?:0[1-9])|(?:[12][0-9]) (?:[12][0-9])|(?:3[01])|[1-9]) |(?:3[01])|[1-9]) ● TIME TIME (?!<[0-9]) (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND}) %{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) (?![0-9]) ● HOUR (?:2[0123]|[01][0-9]) HOUR (?:2[0123]|[01][0-9]) ● MINUTE (?:[0-5][0-9]) MINUTE (?:[0-5][0-9]) ● SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?) SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)

Slide 30

Slide 30 text

HOUR, MINUTE, SECOND HOUR, MINUTE, SECOND May 11 May 11 06 06: :00 00: :27 27 ● %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP:syslog_timestamp} :syslog_timestamp} ● SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} ● MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b ● MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) ● TIME (?!<[0-9])%{ TIME (?!<[0-9])%{HOUR HOUR}:%{ }:%{MINUTE MINUTE}(?::%{ }(?::%{SECOND SECOND})(?![0-9]) })(?![0-9]) ● HOUR HOUR (?:2[0123]| (?:2[0123]|[01][0-9] [01][0-9]) ) ● MINUTE MINUTE (?: (?:[0-5][0-9] [0-5][0-9]) ) ● SECOND SECOND (?:(?: (?:(?:[0-5][0-9] [0-5][0-9]|60)(?:[.,][0-9]+)?) |60)(?:[.,][0-9]+)?)

Slide 31

Slide 31 text

Name a Thing Name a Thing May 11 06:00:27 May 11 06:00:27 ● %{ %{SYSLOGTIMESTAMP SYSLOGTIMESTAMP: :syslog_timestamp syslog_timestamp} } ● SYSLOGTIMESTAMP SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} %{MONTH} +%{MONTHDAY} %{TIME} ● MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b ● MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) ● TIME (?!<[0-9])%{ TIME (?!<[0-9])%{HOUR HOUR}:%{ }:%{MINUTE MINUTE}(?::%{ }(?::%{SECOND SECOND})(?![0-9]) })(?![0-9]) ● HOUR HOUR (?:2[0123]| (?:2[0123]|[01][0-9] [01][0-9]) ) ● MINUTE MINUTE (?: (?:[0-5][0-9] [0-5][0-9]) ) ● SECOND SECOND (?:(?: (?:(?:[0-5][0-9] [0-5][0-9]|60)(?:[.,][0-9]+)?) |60)(?:[.,][0-9]+)?)

Slide 32

Slide 32 text

Named fields Named fields ● When you 'identify' something, it is added to the hash under the @fields key { "@source" => "", "@type" => "", "@tags" => [“haproxy_event”], "@fields" => {“syslog_timestamp” => “May 11 06:00:27”}, "@timestamp" => "", "@source_host" => "", "@source_path" => "", "@message" => "" }

Slide 33

Slide 33 text

Rinse/Repeat Rinse/Repeat ● Filters are processed in order ● Once a field is identified, can be used in interpolation later ● %{syslog_timestamp} ● %{@type} ● @ fields are special but not sacred. ● date and mutate filters for instance.

Slide 34

Slide 34 text

Get out(put) Get out(put) ● An event can be routed to all or a subset of defined outputs based on various criteria ● Outputs block (sort of) ● Logstash takes a default position that you don't want to lose an event ● 1 thread per defined output each ● ALWAYS use a stdout output for debugging ● This is where it gets REALLY cool ● Currently 27 outputs in MASTER

Slide 35

Slide 35 text

Graphite Example Graphite Example output { graphite { host => "1.1.1.1" metrics => [ "stats.enstratus.% {@source_host}.haproxy.% {server_name}.request_type.% {http_verb}", "1"] tags => ["haproxy-event"] } }

Slide 36

Slide 36 text

What does it mean? What does it mean? ● When a message is tagged: “haproxy-event” ● I want to write to Graphite: A value of 1 ● as 'stats.enstratus.X.request_type.Y' ● Where X is the source of the event ● And Y is the HTTP verb

Slide 37

Slide 37 text

End Result End Result

Slide 38

Slide 38 text

End Result End Result

Slide 39

Slide 39 text

But we're also storing the events But we're also storing the events elasticsearch { cluster => 'logstash' host => 'es-server' port => 9300 }

Slide 40

Slide 40 text

And publishing them for remote And publishing them for remote tailing tailing zeromq { topology => "pubsub" address => "tcp://*:5558" mode => "server" topic => "%{@source_host}.% {level}.%{log}" }

Slide 41

Slide 41 text

Where does Chef fit in? Where does Chef fit in?

Slide 42

Slide 42 text

Let's use a familiar example Let's use a familiar example ● Default Chef Handler JSON files ● Parse with Logstash ● Apply some filters ● Send to some places

Slide 43

Slide 43 text

Chef-handler JSON logs Chef-handler JSON logs ● We're not going to be concerned with how you get them INTO logstash ● chef-gelf handler works (logstash has a gelf input) ● You can write your own (I'm partial to ZeroMQ!) ● Set your input type to “json” ● Set the “type” to something that flags it as a chef event. ● If you send the WHOLE thing, be prepared to cut some stuff (you don't really want the Ohai data in your logs)

Slide 44

Slide 44 text

What do we care about? What do we care about? { “node”:{“name”:”foo”}, “success”: true, “start_time”:”2012-05-14 01:09:31 +0000”, “end_time”:”2012-05-14 01:10:46 +0000”, “elapsed_time”:”1.14”, “updated_resources”:[], “exception”:””, “backtrace”:”” }

Slide 45

Slide 45 text

Apply a date filter Apply a date filter (set @timestamp to start_time) (set @timestamp to start_time) date { start_time => “yyyy-MM-dd HH:mm:ss Z”, type => “chef_handler” }

Slide 46

Slide 46 text

Apply a mutate filter Apply a mutate filter (set @message) (set @message) mutate { replace => [“@message”, % {exception}], type => “chef_handler” }

Slide 47

Slide 47 text

Apply a mutate filter Apply a mutate filter (convert status) (convert status) mutate { convert => [“success”, “string”], type => “chef_handler” }

Slide 48

Slide 48 text

Send result and timing to Send result and timing to Graphite Graphite graphite { metrics => [“stats.% {@source_host}.chef_run.success.% {success}”,”1”, “stats.% {@source_host}.chef_run.duration”, % {elapsed_time}], type => “chef_handler” }

Slide 49

Slide 49 text

A better approach A better approach ● Create a custom handler ● Build custom JSON from data ● Strip the extra stuff ● Join stack trace array elements into a single newline-separated value ● Send to Logstash and fanout from there

Slide 50

Slide 50 text

Logstash Cookbook Logstash Cookbook ● github.com/lusis-cookbooks/logstash ● runit based ● Agent, Server, Web, PyShipper recipes ● Also Kibana (alternate web interface) ● No LWRP just yet =(

Slide 51

Slide 51 text

The Future The Future ● Run-time Configuration Changes ● No more restarting ● Push support ● MOAR PLUGINS! ● Internal Metrics Channel ● input { metrics } ● Improved AMQP Support ● Your imagination

Slide 52

Slide 52 text

AMA AMA ● Questions? ● Tips for your peers? ● Hate mail? ● You can ask me later if you'd like

Slide 53

Slide 53 text

Some links Some links ● logstash.net ● cookbook.logstash.net ● github.com/logstash ● github.com/lusis/logstash-shipper

Slide 54

Slide 54 text

Thanks! Thanks! ● github.com/lusis ● @lusis ● #monitoringsucks, #dadops