Ingesting and Visualizing Data using Logstash and Kibana

Transcript

1. 1 Aravind Putrevu Developer | Evangelist @aravindputrevu | aravindputrevu.in Elastic Stack Data Ingestion & Visualizations
2. 2 2 Agenda The Ingest Story 1 Configuring and Managing Logstash 3 Visualizing Data Ingested in Kibana 4 Demo 5 Logstash: Only for Logs? 2
3. 3 3 Agenda The Ingest Story 1 Configuring and Managing Logstash 3 Visualizing Data Ingested in Kibana 4 Demo 5 Logstash: Only for Logs? 2
4. 4 4 Agenda The Ingest Story 1 Configuring and Managing Logstash 3 Visualizing Data Ingested in Kibana 4 Demo 5 Logstash: Only for Logs? 2
5. 5 5 Agenda The Ingest Story 1 Configuring and Managing Logstash 3 Visualizing Data Ingested in Kibana 4 Demo 5 Logstash: Only for Logs? 2
6. 6 6 Agenda The Ingest Story 1 Configuring and Managing Logstash 3 Visualizing Data Ingested in Kibana 4 Demo 5 Logstash: Only for Logs? 2
7. 7 Elastic Stack No enterprise edition All new versions with 6.2 X-Pack Security Alerting Monitoring Reporting Machine Learning Graph
8. 8 \ LOG ANALYTICS METRICS ANALYTICS BUSINESS ANALYTICS SEARCH SECURITY ANALYTICS Monitor your Elastic Stack Find links in your data Be alerted on changes Protect your data Share your insights Detect anomalies APM
9. 9 Logstash Data processing pipeline Ingest data of all shapes, sizes, and sources Parse and dynamically transform data Transport data to any output Secure and encrypt data inputs Build your own pipeline More than 200+ plugins
10. 10 Beats Log Files Metrics Wire Data Datastore Web APIs Social Sensors Kafka Redis Messaging Queue ES-Hadoop Elasticsearch Kibana Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Notes – Warm (X) Instances (X) your{beat} X-Pack X-Pack Custom UI LDAP Authentication AD Notification SSO Hadoop Ecosystem Logstash Nodes (X)
11. 11 Beats Log Files Metrics Wire Data Datastore Web APIs Social Sensors Kafka Redis Messaging Queue ES-Hadoop Elasticsearch Kibana Master Nodes (3) Ingest Nodes (X) Data Nodes – Hot (X) Data Notes – Warm (X) Instances (X) your{beat} X-Pack X-Pack Custom UI LDAP Authentication AD Notification SSO Hadoop Ecosystem Logstash Nodes (X)
12. 12 Popular Data Sources Analysis Archiving Monitoring Alerting MQs Web Apps IoT DBs Logstash Elasticsearch
13. 13 How it works?
14. Configuring Logstash 14
15. Configuring Logstash 15 127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0" client ip timestamp request agent
16. Configuring Logstash 16
17. Configuring Logstash 17
18. Grok 18 https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
19. Best Practices for Grok • Grok may not perform well when a match fails • Monitor the occurrence of _grokparsefailures and then benchmark their cost • Use anchors such as ^ and $ to remove ambiguity and aid the regex engine • Tiered matching increases performance if you don’t use anchors, otherwise don’t bother. When in doubt, measure! • Use Monitoring or Metrics API. 19 https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
20. Grok Debugger 20 https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
21. 21 Pipelines in 1, 2, 3 Start Fast, Think Big • Configure your pipelines, not code them • Stash your first event in minutes • Grow iteratively, scale horizontally
22. Multiple Pipelines 22
23. Centralized Pipeline Management 23
24. Logstash Modules 24 • Elastic Cloud • Arcsight • Netflow
26. 26 Data Resiliency • Persistent Queues • Dead Letter Queues
27. Persistent Queue • Handle bursts of events • At-least once delivery 27
28. Dead Letter Queue 28
29. Scaling Logstash 29
32. 32 Measure...Measure...Measure...
33. Scaling Logstash 33
34. Scaling Logstash 34
35. Logstash Plugins • Input • Output • Filter 35 https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
36. Parse/Extract
37. The Grok Filter filter { grok { match => {“message” => “%{TIMESTAMP_8601:ts}%{SPACE}%{GREEDYDATA:message}”} } } The go-to-tool for parsing fields https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
38. The Date Filter filter { date { match => ["timestamp_string", "ISO8601"] } } Use data strings to set @timestamp https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
39. The KV Filter filter { kv { source => “message” target => “parsed” value_split => “:” } } The easy way to parse data in key/value pairs https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
40. Core operations
41. The Mutate Filter • Convert field types (from strings to integers etc.) • Add/rename/replace/copy fields • Upper/lowercase transformation • Join arrays together (useful for Array=>String operations) • Merge hashes • Split fields into Arrays • Strip whitespace The go-to-tool for field manipulation in Logstash https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
42. Core Transformation Filters • Mutate - Modify / Add Individual Fields • Split - Divide a single event into multiple events • Drop - Delete an event https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
43. Core Operations Example filter { mutate { lowercase => “account” } if [type] == “batch” { split { field => actions target => action } } if { “action” =~ /special/ } { drop {} } } https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
44. Search Supportive
45. GeoIP Filter filter { geoip { fields => “my_geoip_field” } } Enrich IP address information https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
46. User Agent Filter filter { useragent { source => “useragent” } } Enrich browser user agent information https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
47. Translate Filter filter { translate { dictionary => [ "100", "Continue", "101", "Switching Protocols", "merci", "thank you", "old version", "new version" ] } } Use local data to map / enrich events https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
48. Elasticsearch Filter elasticsearch { hosts => ["es-server"] query => "type:start AND operation:%{[opid]}" fields => { "@timestamp" => "started" } } Use Elasticsearch Data to Enrich Events https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
49. JDBC Streaming Filter filter { jdbc_streaming { jdbc_driver_library => "/path/to/mysql-connector-java-5.1.34-bin.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_connection_string => ""jdbc:mysql://localhost:3306/mydatabase" jdbc_user => "me" jdbc_password => "secret" statement => "select * from WORLD.COUNTRY WHERE Code = :code" parameters => { "code" => "country_code"} target => "country_details" } } Use a database to enrich events https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html
51. Resources • https://www.elastic.co/learn • https://www.elastic.co/blog/category/engineering • https://discuss.elastic.co/ • https://fb.com/groups/ElasticIndiaUserGroup • https://elastic.co/community 51
52. 52 Fin! discuss.elastic.co | aravind@elastic.co | @aravindputrevu