Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging Patterns with Logstash and Chef

John Vincent
May 16, 2012
5.1k

Logging Patterns with Logstash and Chef

Long-form version of my #chefconf 2012 logstash talk.

John Vincent

May 16, 2012
Tweet

Transcript

  1. Logging patterns with
    Logging patterns with
    Logstash and Chef
    Logstash and Chef

    View Slide

  2. What is Logstash?
    What is Logstash?

    Log management framework

    Actually been around a while (2008ish)

    Started by Jordan Sissel and Pete
    Fritchman

    Stuff comes in

    Bits get twiddled

    Stuff comes out

    View Slide

  3. What is Logstash?
    What is Logstash?

    Log management framework

    Actually been around a while (2008ish)

    Started by Jordan Sissel and Pete
    Fritchman

    Stuff comes in

    Bits get twiddled

    Stuff comes out
    INPUT
    FILTER
    OUTPUT

    View Slide

  4. It takes stuff like this
    It takes stuff like this
    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282]
    api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920]
    api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034]
    api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013]
    api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160]
    api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074]
    api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

    View Slide

  5. And turns it into this...
    And turns it into this...

    View Slide

  6. And this..
    And this..

    View Slide

  7. Oh and this...
    Oh and this...

    View Slide

  8. View Slide

  9. Just one tiny seed
    Just one tiny seed

    View Slide

  10. Log line/JSON
    message/observed thingie
    +
    +
    The timestamp when it
    happened

    View Slide

  11. Inputs
    Inputs
    This is where events enter the pipeline
    e.g. the source (and optionally the format)

    STDIN

    Twitter search

    Socket

    ZeroMQ

    Heroku

    AMQP

    Currently 14 inputs in MASTER

    View Slide

  12. Quick note on Input “formats”
    Quick note on Input “formats”

    Assumed format for incoming events is
    defined by the plugin

    Most plugins assume plain text

    Others assume JSON

    Some speak 'json_event'*
    * I'll get to this in a moment

    View Slide

  13. Example input definition
    Example input definition
    input {
    file {
    type => "haproxy-log"
    path =>
    ["/var/log/haproxy/haproxy.log"]
    sincedb_path =>
    "/opt/logstash/agent/etc/"
    }
    }

    View Slide

  14. input {
    file {
    type => "haproxy-log"
    path =>
    ["/var/log/haproxy/haproxy.log"]
    sincedb_path =>
    "/opt/logstash/agent/etc/"
    }
    }
    NOT RUBY!
    NOT RUBY!

    View Slide

  15. Hash ALL the things
    Hash ALL the things

    Once event is received, converted to Ruby hash for
    the remainder of the pipeline
    {
    "@source" => "",
    "@type" => "",
    "@tags" => [],
    "@fields" => {},
    "@timestamp" => "",
    "@source_host" => "",
    "@source_path" => "",
    "@message" => ""
    }

    View Slide

  16. Hey man, nice shot!
    Hey man, nice shot!

    Filters are where you do the work

    Break the “@message
    @message” into constituent
    parts

    Identify original timestamp

    Add tags

    Move parts around

    External processing via 0mq

    Currently 13 filters in MASTER

    View Slide

  17. Grok and Roll
    Grok and Roll

    DRY and RAD for RegEx

    Originally a C library

    Pure Ruby version in Logstash since 1.1

    Identify patterns, attach identifier

    View Slide

  18. Remember this?
    Remember this?
    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282]
    api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43565 [11/May/2012:06:00:26.920]
    api-fe api-be/api-b 0/0/0/390/+390 200 +181 - - ---- 10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034]
    api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013]
    api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160]
    api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074]
    api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

    View Slide

  19. Enhance 224 to 176
    Enhance 224 to 176
    May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453 [11/May/2012:06:00:06.282]
    api-fe api-be/api-c 32/0/1/770/+803 202 +153 - - ---- 15/15/5/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43441 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 32/0/1/3374/+3407 202 +153 - - ---- 4/4/4/4/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43444 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 36/0/0/3394/+3430 202 +153 - - ---- 4/4/3/3/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43448 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 37/0/0/3430/+3467 202 +153 - - ---- 4/4/2/2/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:09 localhost haproxy[7829]: 50.18.111.113:43443 [11/May/2012:06:00:06.282]
    api-fe api-be/api-b 38/0/0/3429/+3467 202 +153 - - ---- 4/4/1/1/0 0/0 {} "POST
    /api/enstratus/2011-12-15/infrastructure/Snapshot HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]:
    May 11 06:00:27 localhost haproxy[7829]:
    50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe
    50.18.111.113:43565 [11/May/2012:06:00:26.920] api-fe
    api-be/api-b 0/0/0/390/+390 200 +181 - - ----
    api-be/api-b 0/0/0/390/+390 200 +181 - - ----
    10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-
    10/10/10/5/0 0/0 {} "GET /api/enstratus/2011-12-
    15/admin/Job HTTP/1.1"
    15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43567 [11/May/2012:06:00:27.034]
    api-fe api-be/api-b 0/0/1/366/+367 200 +181 - - ---- 10/10/9/4/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43566 [11/May/2012:06:00:27.013]
    api-fe api-be/api-c 0/0/0/477/+477 200 +181 - - ---- 10/10/8/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43571 [11/May/2012:06:00:27.160]
    api-fe api-be/api-b 0/0/1/379/+380 200 +181 - - ---- 10/10/7/3/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"
    May 11 06:00:27 localhost haproxy[7829]: 50.18.111.113:43569 [11/May/2012:06:00:27.074]
    api-fe api-be/api-b 0/0/2/469/+471 200 +181 - - ---- 10/10/6/2/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job HTTP/1.1"

    View Slide

  20. Center and Stop
    Center and Stop
    May 11 06:00:27 localhost haproxy[7829]:
    50.18.111.113:43565
    [11/May/2012:06:00:26.920] api-fe api-
    be/api-b 0/0/0/390/+390 200 +181 - - ----
    10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job
    HTTP/1.1"

    View Slide

  21. Enhance 57 to 19. Stop
    Enhance 57 to 19. Stop
    May 11 06:00:27
    May 11 06:00:27 localhost
    haproxy[7829]: 50.18.111.113:43565
    [11/May/2012:06:00:26.920] api-fe api-
    be/api-b 0/0/0/390/+390 200 +181 - - ----
    10/10/10/5/0 0/0 {} "GET
    /api/enstratus/2011-12-15/admin/Job
    HTTP/1.1"

    View Slide

  22. Give me a hard copy right there
    Give me a hard copy right there
    May 11 06:00:27
    May 11 06:00:27

    View Slide

  23. Let's analyze this
    Let's analyze this
    May 11 06:00:27
    May 11 06:00:27
    filter {
    grok {
    type => "haproxy-log"
    patterns_dir =>
    ["/opt/logstash/agent/etc/patterns/"]
    pattern => "%{ESHAPROXYHTTP}"
    add_tag => ["haproxy-event"]
    }
    }

    View Slide

  24. ESHAPROXYHTTP
    ESHAPROXYHTTP
    It's REALLY long
    It's REALLY long

    View Slide

  25. ESHAPROXYCAPTUREDREQUESTHEADERS %
    {DATA:request_header_x_forwarded_for}
    ESHAPROXYCAPTUREDRESPONSEHEADERS %
    {DATA:response_header_content_type}\|%
    {DATA:response_header_content_encoding}\|%
    {DATA:response_header_cache_control}\|%
    {DATA:response_header_last_modified}
    ESHAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} %
    {IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%
    {INT:client_port} \[%{HAPROXYDATE:accept_date}\] %
    {NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%
    {NOTSPACE:server_name} %{INT:time_request}/%
    {INT:time_queue}/%{INT:time_backend_connect}/%
    {INT:time_backend_response}/\+%{NOTSPACE:time_duration} %
    {INT:http_status_code} \+%{NOTSPACE:bytes_read} %
    {DATA:captured_request_cookie} %
    {DATA:captured_response_cookie} %
    {NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%
    {INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %
    {INT:srv_queue}/%{INT:backend_queue} \{%
    {ESHAPROXYCAPTUREDREQUESTHEADERS}\} "%{WORD:http_verb} %
    {URIPATHPARAM:http_request} HTTP/%{NUMBER:http_version}"

    View Slide

  26. ESHAPROXYHTTP
    ESHAPROXYHTTP
    We're going to focus on the first part
    %{SYSLOGTIMESTAMP:syslog_timestamp}

    View Slide

  27. What's in a SYSLOGTIMESTAMP?
    What's in a SYSLOGTIMESTAMP?

    View Slide

  28. SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP
    May 11 06:00:27
    May 11 06:00:27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:syslog_timestamp}
    :syslog_timestamp}

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
    %{MONTH} +%{MONTHDAY} %{TIME}

    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

    TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
    TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])

    HOUR (?:2[0123]|[01][0-9])
    HOUR (?:2[0123]|[01][0-9])

    MINUTE (?:[0-5][0-9])
    MINUTE (?:[0-5][0-9])

    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)

    View Slide

  29. MONTH, MONTHDAY, TIME
    MONTH, MONTHDAY, TIME
    May
    May 11
    11 06:00:27
    06:00:27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:syslog_timestamp}
    :syslog_timestamp}

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{
    %{MONTH
    MONTH} +%{
    } +%{MONTHDAY
    MONTHDAY} %{
    } %{TIME
    TIME}
    }

    MONTH
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?
    \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    |
    May|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY
    MONTHDAY (?:(?:0[1-9])|
    (?:(?:0[1-9])|(?:[12][0-9])
    (?:[12][0-9])|(?:3[01])|[1-9])
    |(?:3[01])|[1-9])

    TIME
    TIME (?!<[0-9])
    (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})
    %{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
    (?![0-9])

    HOUR (?:2[0123]|[01][0-9])
    HOUR (?:2[0123]|[01][0-9])

    MINUTE (?:[0-5][0-9])
    MINUTE (?:[0-5][0-9])

    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)
    SECOND (?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)

    View Slide

  30. HOUR, MINUTE, SECOND
    HOUR, MINUTE, SECOND
    May 11
    May 11 06
    06:
    :00
    00:
    :27
    27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:syslog_timestamp}
    :syslog_timestamp}

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
    %{MONTH} +%{MONTHDAY} %{TIME}

    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

    TIME (?!<[0-9])%{
    TIME (?!<[0-9])%{HOUR
    HOUR}:%{
    }:%{MINUTE
    MINUTE}(?::%{
    }(?::%{SECOND
    SECOND})(?![0-9])
    })(?![0-9])

    HOUR
    HOUR (?:2[0123]|
    (?:2[0123]|[01][0-9]
    [01][0-9])
    )

    MINUTE
    MINUTE (?:
    (?:[0-5][0-9]
    [0-5][0-9])
    )

    SECOND
    SECOND (?:(?:
    (?:(?:[0-5][0-9]
    [0-5][0-9]|60)(?:[.,][0-9]+)?)
    |60)(?:[.,][0-9]+)?)

    View Slide

  31. Name a Thing
    Name a Thing
    May 11 06:00:27
    May 11 06:00:27

    %{
    %{SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP:
    :syslog_timestamp
    syslog_timestamp}
    }

    SYSLOGTIMESTAMP
    SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
    %{MONTH} +%{MONTHDAY} %{TIME}

    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b
    Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b

    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
    MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])

    TIME (?!<[0-9])%{
    TIME (?!<[0-9])%{HOUR
    HOUR}:%{
    }:%{MINUTE
    MINUTE}(?::%{
    }(?::%{SECOND
    SECOND})(?![0-9])
    })(?![0-9])

    HOUR
    HOUR (?:2[0123]|
    (?:2[0123]|[01][0-9]
    [01][0-9])
    )

    MINUTE
    MINUTE (?:
    (?:[0-5][0-9]
    [0-5][0-9])
    )

    SECOND
    SECOND (?:(?:
    (?:(?:[0-5][0-9]
    [0-5][0-9]|60)(?:[.,][0-9]+)?)
    |60)(?:[.,][0-9]+)?)

    View Slide

  32. Named fields
    Named fields

    When you 'identify' something, it is added to the hash
    under the @fields key
    {
    "@source" => "",
    "@type" => "",
    "@tags" => [“haproxy_event”],
    "@fields" => {“syslog_timestamp” => “May 11 06:00:27”},
    "@timestamp" => "",
    "@source_host" => "",
    "@source_path" => "",
    "@message" => ""
    }

    View Slide

  33. Rinse/Repeat
    Rinse/Repeat

    Filters are processed in order

    Once a field is identified, can be used in
    interpolation later

    %{syslog_timestamp}

    %{@type}

    @ fields are special but not sacred.

    date and mutate filters for instance.

    View Slide

  34. Get out(put)
    Get out(put)

    An event can be routed to all or a subset of
    defined outputs based on various criteria

    Outputs block (sort of)

    Logstash takes a default position that you
    don't want to lose an event

    1 thread per defined output each

    ALWAYS use a stdout output for debugging

    This is where it gets REALLY cool

    Currently 27 outputs in MASTER

    View Slide

  35. Graphite Example
    Graphite Example
    output {
    graphite {
    host => "1.1.1.1"
    metrics => [ "stats.enstratus.%
    {@source_host}.haproxy.%
    {server_name}.request_type.%
    {http_verb}", "1"]
    tags => ["haproxy-event"]
    }
    }

    View Slide

  36. What does it mean?
    What does it mean?

    When a message is tagged:
    “haproxy-event”

    I want to write to Graphite:
    A value of 1

    as 'stats.enstratus.X.request_type.Y'

    Where X is the source of the event

    And Y is the HTTP verb

    View Slide

  37. End Result
    End Result

    View Slide

  38. End Result
    End Result

    View Slide

  39. But we're also storing the events
    But we're also storing the events
    elasticsearch {
    cluster => 'logstash'
    host => 'es-server'
    port => 9300
    }

    View Slide

  40. And publishing them for remote
    And publishing them for remote
    tailing
    tailing
    zeromq {
    topology => "pubsub"
    address => "tcp://*:5558"
    mode => "server"
    topic => "%{@source_host}.%
    {level}.%{log}"
    }

    View Slide

  41. Where does Chef fit in?
    Where does Chef fit in?

    View Slide

  42. Let's use a familiar example
    Let's use a familiar example

    Default Chef Handler JSON files

    Parse with Logstash

    Apply some filters

    Send to some places

    View Slide

  43. Chef-handler JSON logs
    Chef-handler JSON logs

    We're not going to be concerned with how you get
    them INTO logstash

    chef-gelf handler works (logstash has a gelf input)

    You can write your own (I'm partial to ZeroMQ!)

    Set your input type to “json”

    Set the “type” to something that flags it as a chef
    event.

    If you send the WHOLE thing, be prepared to cut some
    stuff (you don't really want the Ohai data in your logs)

    View Slide

  44. What do we care about?
    What do we care about?
    {
    “node”:{“name”:”foo”},
    “success”: true,
    “start_time”:”2012-05-14 01:09:31
    +0000”,
    “end_time”:”2012-05-14 01:10:46
    +0000”,
    “elapsed_time”:”1.14”,
    “updated_resources”:[],
    “exception”:””,
    “backtrace”:””
    }

    View Slide

  45. Apply a date filter
    Apply a date filter
    (set @timestamp to start_time)
    (set @timestamp to start_time)
    date {
    start_time => “yyyy-MM-dd
    HH:mm:ss Z”,
    type => “chef_handler”
    }

    View Slide

  46. Apply a mutate filter
    Apply a mutate filter
    (set @message)
    (set @message)
    mutate {
    replace => [“@message”, %
    {exception}],
    type => “chef_handler”
    }

    View Slide

  47. Apply a mutate filter
    Apply a mutate filter
    (convert status)
    (convert status)
    mutate {
    convert => [“success”,
    “string”],
    type => “chef_handler”
    }

    View Slide

  48. Send result and timing to
    Send result and timing to
    Graphite
    Graphite
    graphite {
    metrics => [“stats.%
    {@source_host}.chef_run.success.%
    {success}”,”1”, “stats.%
    {@source_host}.chef_run.duration”, %
    {elapsed_time}],
    type => “chef_handler”
    }

    View Slide

  49. A better approach
    A better approach

    Create a custom handler

    Build custom JSON from data

    Strip the extra stuff

    Join stack trace array elements into a
    single newline-separated value

    Send to Logstash and fanout from
    there

    View Slide

  50. Logstash Cookbook
    Logstash Cookbook

    github.com/lusis-cookbooks/logstash

    runit based

    Agent, Server, Web, PyShipper recipes

    Also Kibana (alternate web interface)

    No LWRP just yet =(

    View Slide

  51. The Future
    The Future

    Run-time Configuration Changes

    No more restarting

    Push support

    MOAR PLUGINS!

    Internal Metrics Channel

    input { metrics }

    Improved AMQP Support

    Your imagination

    View Slide

  52. AMA
    AMA

    Questions?

    Tips for your peers?

    Hate mail?

    You can ask me later if you'd like

    View Slide

  53. Some links
    Some links

    logstash.net

    cookbook.logstash.net

    github.com/logstash

    github.com/lusis/logstash-shipper

    View Slide

  54. Thanks!
    Thanks!

    github.com/lusis

    @lusis

    #monitoringsucks, #dadops

    View Slide