logstash - devopsfinland

Transcript

1. Logstash A Real Life Design
2. Ramez Hanna Husband and Father Sys Admin Get It Done @informatiq
3. Disclaimer I am not affiliated with Logstash in any way I am a happy user
4. Stages of a log system
5. Collection Bring order to chaos
6. Frontend Backend Admin Database
7. Frontend PY Nginx SYS RSYSLOG
8. Transport Get it somewhere
9. servers Redis RSYSLOG LOGSTASH LOGSTASH
10. Process Make sense of your logs
11. input { redis { host => "10.0.1.189" data_type => "list" key => "logstash" message_format => "json_event" } } filter { ## drop unneeded logs # DHCP client if [program] =~ 'DHCP' or [program] =~ 'dhclient' { drop{ } } # start tagging logs # ansible if [program] =~ "ansible" { mutate { add_tag => "ansible" } }
12. ## start parsing the actual log for information # access­log if 'access­log' in [tags] { grok { match => ["message", '%{IPORHOST:clientIP} %{USER:ident} %{USER:auth} \[% {HTTPDATE:nginxTimeStamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/% {NUMBER:httpVersion}" %{NUMBER:responseCode} (?:%{NUMBER:bytes}|­) (?:"(?:% {URI:referrer}|­)"|%{QS:referrer}) %{QS:agent} %{BASE10NUM:logTime} % {BASE10NUM:requestDuration} "%{GREEDYDATA:sslClientDn}"' ] } } #parse vpn if 'vpn' in [tags] { grok { match => ["message", '%{IPORHOST:clientIP}:%{POSINT} \[%{WORD:user}\] Peer Connection Initiated with \[AF_INET\]'] add_tag => 'vpnlogin' } } #geoip all clientIP fields geoip { add_tag => 'geoip' source => 'clientIP' } }
13. Store
14. output { #elasticsearch { # embedded => false # bind_host => "10.0.1.189" # max_inflight_requests => "2000" # port => "9300" #} elasticsearch_http { host => "10.0.1.189" }
15. visualize logs with Kibana
16. Action Time
17. Output { if "access­log" in [tags] { if [request] =~ "login" { statsd { host => "10.0.1.196" port => 8125 namespace => "holvi_com" sender => "%{logsource}" increment => "login" } } statsd { host => "10.0.1.196" port => 8125 namespace => "holvi_com" sender => "%{logsource}" increment => "response.%{responseCode}" timing => [ "ResponseTime", "%{requestDuration}" ] count => [ "bytes", "%{bytes}" ] } }