Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging Patterns with Logstash and Chef

John Vincent
May 16, 2012
5.1k

Logging Patterns with Logstash and Chef

Long-form version of my #chefconf 2012 logstash talk.

John Vincent

May 16, 2012
Tweet

Transcript

  1. Logging patterns with
    Logging patterns with
    Logstash and Chef
    Logstash and Chef

    View full-size slide

  2. What is Logstash?
    What is Logstash?

    Log management framework

    Actually been around a while (2008ish)

    Started by Jordan Sissel and Pete
    Fritchman

    Stuff comes in

    Bits get twiddled

    Stuff comes out

    View full-size slide

  3. What is Logstash?
    What is Logstash?

    Log management framework

    Actually been around a while (2008ish)

    Started by Jordan Sissel and Pete
    Fritchman

    Stuff comes in

    Bits get twiddled

    Stuff comes out
    INPUT
    FILTER
    OUTPUT

    View full-size slide

  4. It takes stuff like this
    It takes stuff like this
    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282]
    api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920]
    api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034]
    api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013]
    api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160]
    api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074]
    api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

    View full-size slide

  5. And turns it into this...
    And turns it into this...

    View full-size slide

  6. And this..
    And this..

    View full-size slide

  7. Oh and this...
    Oh and this...

    View full-size slide

  8. Just one tiny seed
    Just one tiny seed

    View full-size slide

  9. Log line/JSON
    message/observed thingie
    +
    +
    The timestamp when it
    happened

    View full-size slide

  10. Inputs
    Inputs
    This is where events enter the pipeline
    e.g. the source (and optionally the format)

    STDIN

    Twitter search

    Socket

    ZeroMQ

    Heroku

    AMQP

    Currently 14 inputs in MASTER

    View full-size slide

  11. Quick note on Input “formats”
    Quick note on Input “formats”

    Assumed format for incoming events is
    defined by the plugin

    Most plugins assume plain text

    Others assume JSON

    Some speak 'json_event'*
    * I'll get to this in a moment

    View full-size slide

  12. Example input definition
    Example input definition
    input {
    file {
    type => "haproxy-log"
    path =>
    ["/var/log/haproxy/haproxy.log"]
    sincedb_path =>
    "/opt/logstash/agent/etc/"
    }
    }

    View full-size slide

  13. input {
    file {
    type => "haproxy-log"
    path =>
    ["/var/log/haproxy/haproxy.log"]
    sincedb_path =>
    "/opt/logstash/agent/etc/"
    }
    }
    NOT RUBY!
    NOT RUBY!

    View full-size slide

  14. Hash ALL the things
    Hash ALL the things

    Once event is received, converted to Ruby hash for
    the remainder of the pipeline
    {
    "@source" => "",
    "@type" => "",
    "@tags" => [],
    "@fields" => {},
    "@timestamp" => "",
    "@source_host" => "",
    "@source_path" => "",
    "@message" => ""
    }

    View full-size slide

  15. Hey man, nice shot!
    Hey man, nice shot!

    Filters are where you do the work

    Break the “@message
    @message” into constituent
    parts

    Identify original timestamp

    Add tags

    Move parts around

    External processing via 0mq

    Currently 13 filters in MASTER

    View full-size slide

  16. Grok and Roll
    Grok and Roll

    DRY and RAD for RegEx

    Originally a C library

    Pure Ruby version in Logstash since 1.1

    Identify patterns, attach identifier

    View full-size slide

  17. Remember this?
    Remember this?
    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282]
    api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920]
    api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034]
    api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013]
    api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160]
    api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074]
    api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

    View full-size slide

  18. Enhance 224 to 176
    Enhance 224 to 176
    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282]
    api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]:
    May 11 06:00:27 localhost haproxy[7829]:
    50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe
    50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe
    api-be/api-b 0/0/0/390/+390 200 +181 - - ----
    api-be/api-b 0/0/0/390/+390 200 +181 - - ----
    10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-
    10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-
    15/admin/Job HTTP/1.1"
    15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034]
    api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013]
    api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160]
    api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074]
    api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

    View full-size slide

  19. Center and Stop
    Center and Stop
    May 11 06:00:27 localhost haproxy[7829]:
    50.18.111.113:43565
    [11/May/2012:06:00:26.920] api-fe api-
    be/api-b 0/0/0/390/+390 200 +181 - - ----
    10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job
    HTTP/1.1"

    View full-size slide

  20. Enhance 57 to 19. Stop
    Enhance 57 to 19. Stop
    May 11 06:00:27
    May 11 06:00:27 localhost
    haproxy[7829]: 50.18.111.113:43565
    [11/May/2012:06:00:26.920] api-fe api-
    be/api-b 0/0/0/390/+390 200 +181 - - ----
    10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job
    HTTP/1.1"

    View full-size slide

  21. Give me a hard copy right there
    Give me a hard copy right there
    May 11 06:00:27
    May 11 06:00:27

    View full-size slide

  22. Let's analyze this
    Let's analyze this
    May 11 06:00:27
    May 11 06:00:27
    filter {
    grok {
    type => "haproxy-log"
    patterns_dir =>
    ["/opt/logstash/agent/etc/patterns/"]
    pattern => "%{ESHAPROXYHTTP}"
    add_tag => ["haproxy-event"]
    }
    }

    View full-size slide

  23. ESHAPROXYHTTP
    ESHAPROXYHTTP
    It's REALLY long
    It's REALLY long

    View full-size slide

  24. ESHAPROXYCAPTUREDREQUESTHEADERS %
    {DATA:request_header_x_forwarded_for}
    ESHAPROXYCAPTUREDRESPONSEHEADERS %
    {DATA:response_header_content_type}\|%
    {DATA:response_header_content_encoding}\|%
    {DATA:response_header_cache_control}\|%
    {DATA:response_header_last_modified}
    ESHAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} %
    {IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%
    {INT:client_port} \[%{HAPROXYDATE:accept_date}\] %
    {NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%
    {NOTSPACE:server_name} %{INT:time_request}/%
    {INT:time_queue}/%{INT:time_backend_connect}/%
    {INT:time_backend_response}/\+%{NOTSPACE:time_duration} %
    {INT:http_status_code} \+%{NOTSPACE:bytes_read} %
    {DATA:captured_request_cookie} %
    {DATA:captured_response_cookie} %
    {NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%
    {INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %
    {INT:srv_queue}/%{INT:backend_queue} \{%
    {ESHAPROXYCAPTUREDREQUESTHEADERS}\} "%{WORD:http_verb} %
    {URIPATHPARAM:http_request} HTTP/%{NUMBER:http_version}"

    View full-size slide

  25. ESHAPROXYHTTP
    ESHAPROXYHTTP
    We're going to focus on the first part
    %{SYSLOGTIMESTAMP:syslog_timestamp}

    View full-size slide

  26. What's in a SYSLOGTIMESTAMP?
    What's in a SYSLOGTIMESTAMP?

    View full-size slide

  27. SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP
    May 11 06:00:27
    May 11 06:00:27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:syslog_timestamp}
    :syslog_timestamp}

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
    %{MONTH} +%{MONTHDAY} %{TIME}

    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

    TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
    TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])

    HOUR (?:2[0123]|[01][0-9])
    HOUR (?:2[0123]|[01][0-9])

    MINUTE (?:[0-5][0-9])
    MINUTE (?:[0-5][0-9])

    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)

    View full-size slide

  28. MONTH, MONTHDAY, TIME
    MONTH, MONTHDAY, TIME
    May
    May 11
    11 06:00:27
    06:00:27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:syslog_timestamp}
    :syslog_timestamp}

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{
    %{MONTH
    MONTH} +%{
    } +%{MONTHDAY
    MONTHDAY} %{
    } %{TIME
    TIME}
    }

    MONTH
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?
    \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    |
    May|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY
    MONTHDAY (?:(?:0[1-9])|
    (?:(?:0[1-9])|(?:[12][0-9])
    (?:[12][0-9])|(?:3[01])|[1-9])
    |(?:3[01])|[1-9])

    TIME
    TIME (?!<[0-9])
    (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})
    %{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
    (?![0-9])

    HOUR (?:2[0123]|[01][0-9])
    HOUR (?:2[0123]|[01][0-9])

    MINUTE (?:[0-5][0-9])
    MINUTE (?:[0-5][0-9])

    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)

    View full-size slide

  29. HOUR, MINUTE, SECOND
    HOUR, MINUTE, SECOND
    May 11
    May 11 06
    06:
    :00
    00:
    :27
    27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:syslog_timestamp}
    :syslog_timestamp}

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
    %{MONTH} +%{MONTHDAY} %{TIME}

    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

    TIME (?!<[0-9])%{
    TIME (?!<[0-9])%{HOUR
    HOUR}:%{
    }:%{MINUTE
    MINUTE}(?::%{
    }(?::%{SECOND
    SECOND})(?![0-9])
    })(?![0-9])

    HOUR
    HOUR (?:2[0123]|
    (?:2[0123]|[01][0-9]
    [01][0-9])
    )

    MINUTE
    MINUTE (?:
    (?:[0-5][0-9]
    [0-5][0-9])
    )

    SECOND
    SECOND (?:(?:
    (?:(?:[0-5][0-9]
    [0-5][0-9]|60)(?:[.,][0-9]+)?)
    |60)(?:[.,][0-9]+)?)

    View full-size slide

  30. Name a Thing
    Name a Thing
    May 11 06:00:27
    May 11 06:00:27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:
    :syslog_timestamp
    syslog_timestamp}
    }

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
    %{MONTH} +%{MONTHDAY} %{TIME}

    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

    TIME (?!<[0-9])%{
    TIME (?!<[0-9])%{HOUR
    HOUR}:%{
    }:%{MINUTE
    MINUTE}(?::%{
    }(?::%{SECOND
    SECOND})(?![0-9])
    })(?![0-9])

    HOUR
    HOUR (?:2[0123]|
    (?:2[0123]|[01][0-9]
    [01][0-9])
    )

    MINUTE
    MINUTE (?:
    (?:[0-5][0-9]
    [0-5][0-9])
    )

    SECOND
    SECOND (?:(?:
    (?:(?:[0-5][0-9]
    [0-5][0-9]|60)(?:[.,][0-9]+)?)
    |60)(?:[.,][0-9]+)?)

    View full-size slide

  31. Named fields
    Named fields

    When you 'identify' something, it is added to the hash
    under the @fields key
    {
    "@source" => "",
    "@type" => "",
    "@tags" => [“haproxy_event”],
    "@fields" => {“syslog_timestamp” => “May 11 06:00:27”},
    "@timestamp" => "",
    "@source_host" => "",
    "@source_path" => "",
    "@message" => ""
    }

    View full-size slide

  32. Rinse/Repeat
    Rinse/Repeat

    Filters are processed in order

    Once a field is identified, can be used in
    interpolation later

    %{syslog_timestamp}

    %{@type}

    @ fields are special but not sacred.

    date and mutate filters for instance.

    View full-size slide

  33. Get out(put)
    Get out(put)

    An event can be routed to all or a subset of
    defined outputs based on various criteria

    Outputs block (sort of)

    Logstash takes a default position that you
    don't want to lose an event

    1 thread per defined output each

    ALWAYS use a stdout output for debugging

    This is where it gets REALLY cool

    Currently 27 outputs in MASTER

    View full-size slide

  34. Graphite Example
    Graphite Example
    output {
    graphite {
    host => "1.1.1.1"
    metrics => [ "stats.enstratus.%
    {@source_host}.haproxy.%
    {server_name}.request_type.%
    {http_verb}", "1"]
    tags => ["haproxy-event"]
    }
    }

    View full-size slide

  35. What does it mean?
    What does it mean?

    When a message is tagged:
    “haproxy-event”

    I want to write to Graphite:
    A value of 1

    as 'stats.enstratus.X.request_type.Y'

    Where X is the source of the event

    And Y is the HTTP verb

    View full-size slide

  36. End Result
    End Result

    View full-size slide

  37. End Result
    End Result

    View full-size slide

  38. But we're also storing the events
    But we're also storing the events
    elasticsearch {
    cluster => 'logstash'
    host => 'es-server'
    port => 9300
    }

    View full-size slide

  39. And publishing them for remote
    And publishing them for remote
    tailing
    tailing
    zeromq {
    topology => "pubsub"
    address => "tcp://*:5558"
    mode => "server"
    topic => "%{@source_host}.%
    {level}.%{log}"
    }

    View full-size slide

  40. Where does Chef fit in?
    Where does Chef fit in?

    View full-size slide

  41. Let's use a familiar example
    Let's use a familiar example

    Default Chef Handler JSON files

    Parse with Logstash

    Apply some filters

    Send to some places

    View full-size slide

  42. Chef-handler JSON logs
    Chef-handler JSON logs

    We're not going to be concerned with how you get
    them INTO logstash

    chef-gelf handler works (logstash has a gelf input)

    You can write your own (I'm partial to ZeroMQ!)

    Set your input type to “json”

    Set the “type” to something that flags it as a chef
    event.

    If you send the WHOLE thing, be prepared to cut some
    stuff (you don't really want the Ohai data in your logs)

    View full-size slide

  43. What do we care about?
    What do we care about?
    {
    “node”:{“name”:”foo”},
    “success”: true,
    “start_time”:”2012-05-14 01:09:31
    +0000”,
    “end_time”:”2012-05-14 01:10:46
    +0000”,
    “elapsed_time”:”1.14”,
    “updated_resources”:[],
    “exception”:””,
    “backtrace”:””
    }

    View full-size slide

  44. Apply a date filter
    Apply a date filter
    (set @timestamp to start_time)
    (set @timestamp to start_time)
    date {
    start_time => “yyyy-MM-dd
    HH:mm:ss Z”,
    type => “chef_handler”
    }

    View full-size slide

  45. Apply a mutate filter
    Apply a mutate filter
    (set @message)
    (set @message)
    mutate {
    replace => [“@message”, %
    {exception}],
    type => “chef_handler”
    }

    View full-size slide

  46. Apply a mutate filter
    Apply a mutate filter
    (convert status)
    (convert status)
    mutate {
    convert => [“success”,
    “string”],
    type => “chef_handler”
    }

    View full-size slide

  47. Send result and timing to
    Send result and timing to
    Graphite
    Graphite
    graphite {
    metrics => [“stats.%
    {@source_host}.chef_run.success.%
    {success}”,”1”, “stats.%
    {@source_host}.chef_run.duration”, %
    {elapsed_time}],
    type => “chef_handler”
    }

    View full-size slide

  48. A better approach
    A better approach

    Create a custom handler

    Build custom JSON from data

    Strip the extra stuff

    Join stack trace array elements into a
    single newline-separated value

    Send to Logstash and fanout from
    there

    View full-size slide

  49. Logstash Cookbook
    Logstash Cookbook

    github.com/lusis-cookbooks/logstash

    runit based

    Agent, Server, Web, PyShipper recipes

    Also Kibana (alternate web interface)

    No LWRP just yet =(

    View full-size slide

  50. The Future
    The Future

    Run-time Configuration Changes

    No more restarting

    Push support

    MOAR PLUGINS!

    Internal Metrics Channel

    input { metrics }

    Improved AMQP Support

    Your imagination

    View full-size slide

  51. AMA
    AMA

    Questions?

    Tips for your peers?

    Hate mail?

    You can ask me later if you'd like

    View full-size slide

  52. Some links
    Some links

    logstash.net

    cookbook.logstash.net

    github.com/logstash

    github.com/lusis/logstash-shipper

    View full-size slide

  53. Thanks!
    Thanks!

    github.com/lusis

    @lusis

    #monitoringsucks, #dadops

    View full-size slide