• Actually been around a while (2008ish) • Started by Jordan Sissel and Pete Fritchman • Stuff comes in • Bits get twiddled • Stuff comes out INPUT FILTER OUTPUT
• Assumed format for incoming events is defined by the plugin • Most plugins assume plain text • Others assume JSON • Some speak 'json_event'* * I'll get to this in a moment
event is received, converted to Ruby hash for the remainder of the pipeline { "@source" => "<format determined by input>", "@type" => "<type from input>", "@tags" => [], "@fields" => {}, "@timestamp" => "<ISO8601 of received event>", "@source_host" => "<localhost or source hostname>", "@source_path" => "<value determined by input>", "@message" => "<the escaped representation of the line>" }
are where you do the work • Break the “@message @message” into constituent parts • Identify original timestamp • Add tags • Move parts around • External processing via 0mq • Currently 13 filters in MASTER
is added to the hash under the @fields key { "@source" => "<format determined by input>", "@type" => "<type from input>", "@tags" => [“haproxy_event”], "@fields" => {“syslog_timestamp” => “May 11 06:00:27”}, "@timestamp" => "<ISO8601 of received event>", "@source_host" => "<localhost or source hostname>", "@source_path" => "<value determined by input>", "@message" => "<the escaped representation of the line>" }
a field is identified, can be used in interpolation later • %{syslog_timestamp} • %{@type} • @ fields are special but not sacred. • date and mutate filters for instance.
to all or a subset of defined outputs based on various criteria • Outputs block (sort of) • Logstash takes a default position that you don't want to lose an event • 1 thread per defined output each • ALWAYS use a stdout output for debugging • This is where it gets REALLY cool • Currently 27 outputs in MASTER
a message is tagged: “haproxy-event” • I want to write to Graphite: A value of 1 • as 'stats.enstratus.X.request_type.Y' • Where X is the source of the event • And Y is the HTTP verb
to be concerned with how you get them INTO logstash • chef-gelf handler works (logstash has a gelf input) • You can write your own (I'm partial to ZeroMQ!) • Set your input type to “json” • Set the “type” to something that flags it as a chef event. • If you send the WHOLE thing, be prepared to cut some stuff (you don't really want the Ohai data in your logs)
handler • Build custom JSON from data • Strip the extra stuff • Join stack trace array elements into a single newline-separated value • Send to Logstash and fanout from there
lusis
paulrobertlloyd
matthewcrist
vanstee
colly
malarkey
edds
kevinliebholz
maltzj
orderedlist
dougneiner
smashingmag
geeforr
![input { file { type => "haproxy-log" path => ["/var/log/haproxy/haproxy.log"]](https://files.speakerdeck.com/presentations/4fb4285f2cf6f5002100c142/slide_13.jpg){kind=link}
![Remember this? Remember this? May 11 06:00:07 localhost haproxy[7829]: 50.18.111.113:43453](https://files.speakerdeck.com/presentations/4fb4285f2cf6f5002100c142/slide_17.jpg){kind=link}
