Why I love Logstash and you should too.

What is Logstash? Do you use it? What is it for? Why Ruby? How does it work?
This talk also covers what the newly released Logstash 1.5 brings and what's planned for 2.x
Presented at DevoxxPL 2015 by João Duarte on the 24th of June, 2015

1. @jsvd #DevoxxPL Platinum Sponsors: Why I love Logstash and you should too. João Duarte elastic
2. @jsvd #DevoxxPL • João Duarte • Portugal! • Software Engineer at • (Mostly) Ruby • Lover of all things Event-Driven Who Am I?
3. @jsvd #DevoxxPL Why are we here?
4. @jsvd #DevoxxPL Why are we here?
5. @jsvd #DevoxxPL Show of hands
6. @jsvd #DevoxxPL 1. What is Logstash? 2. What is it for? 3. Latest developments 4. The future!! Schedule
7. @jsvd #DevoxxPL •Event processing pipeline •Plugin based •Written in Ruby •Open Source What is Logstash?
8. @jsvd #DevoxxPL Plugin Based Event Processing Pipeline?? PBEPP?!
9. @jsvd #DevoxxPL Plugin Based Event Processing Pipeline!! inputs filters outputs file syslog tcp websockets grok date split geoip anonymize elasticsearch pagerduty file
10. @jsvd #DevoxxPL Logstash Use Cases
11. @jsvd #DevoxxPL Logstash - Use cases Apache DB elasticsearch
12. @jsvd #DevoxxPL Logstash - Use cases Apache Apache DB
13. @jsvd #DevoxxPL Logstash - Use cases Apache Apache DB elasticsearch Kibana
14. @jsvd #DevoxxPL Logstash - Use cases
15. @jsvd #DevoxxPL Logstash - Use cases zaakceptować te informacje
16. @jsvd #DevoxxPL Logstash - Use cases zaakceptować te informacje What?!
17. @jsvd #DevoxxPL Logstash - Use cases … …
18. @jsvd #DevoxxPL Logstash - Use cases
19. @jsvd #DevoxxPL •Options available for communicating data are huge •Most software logging is targeted at humans •Single line, multi line •Plain text, json, xml •Log4j, Log files, Syslog, TCP, UDP •Don’t get me started on date formats Heterogeneity
20. @jsvd #DevoxxPL Logstash - Use cases
21. @jsvd #DevoxxPL Logstash - Use cases
22. @jsvd #DevoxxPL Logstash - Use cases zaakceptować te informacje
23. @jsvd #DevoxxPL Logstash - Use cases accept this information
24. @jsvd #DevoxxPL Logstash - Use cases TCP(csv)
25. @jsvd #DevoxxPL Logstash - Use cases TCP(csv) HTTP(json)
26. @jsvd #DevoxxPL Logstash - Use cases
27. @jsvd #DevoxxPL Tell me more..
28. @jsvd #DevoxxPL Logstash - Open Source opensource http://github.com/elastic/logstash
29. @jsvd #DevoxxPL Logstash - Ruby
30. @jsvd #DevoxxPL Logstash - JRuby
31. @jsvd #DevoxxPL Logstash - JRuby • Leverages JVM • Battle-proven GC • Opens interop with Java, Scala, Clojure • Mature libraries for EVERYTHING • VisualVM <3 • Future: 9.0.0.0, Truffle
32. @jsvd #DevoxxPL Logstash - JRuby + Truffle • Project started at Oracle Labs in 2013 • Implementation of the Ruby language using • Graal dynamic compiler • Truffle AST interpreter framework • Simpler code, evolving AST • control over the compiler with Graal https://github.com/jruby/jruby/wiki/Truffle
33. @jsvd #DevoxxPL Logstash - JRuby + Truffle $ rbenv install jruby-9.0.0.0+graal-dev $ rbenv shell jruby-9.0.0.0+graal-dev $ ruby -X+T -e 'puts Truffle.graal?' true
34. @jsvd #DevoxxPL Logstash - Concepts • Events • Plugins • Input: capture an occurrence in outside world into an event; • Filter: transform, drop and validate events; • Output: send an event to the outside world.
35. @jsvd #DevoxxPL Logstash Config • Text file driven configuration • Describe the three stages of the pipeline • Configuration changes require restart (for now..) $ bin/logstash -f my.conf Logstash startup completed ...
36. @jsvd #DevoxxPL Logstash Config input { } filter { } output { }
37. @jsvd #DevoxxPL Logstash Config input { file { path => "/var/log/apache/*.log" } udp { port => 3333 } } filter { } output { }
38. @jsvd #DevoxxPL Logstash Config input { file { path => "/var/log/apache/*.log" } udp { port => 3333 } } filter { checksum { keys => ["message", "@timestamp"] } } output { }
39. @jsvd #DevoxxPL Logstash Config input { file { path => "/var/log/apache/*.log" } udp { port => 3333 } } filter { checksum { keys => ["message", "@timestamp"] } } output { elasticsearch { protocol => http } stdout { } }
40. @jsvd #DevoxxPL • Input Plugins: - file, rabbitmq, redis, snmptrap, syslog, TCP, UDP, twitter, S3, etc. • Filter Plugins: - grok, date, mutate, split, multiline, ruby, etc. • Output Plugins: - elasticsearch, rabbitmq, redis, file, pagerduty, email, nagios, etc. Logstash Plugins
41. @jsvd #DevoxxPL mysql-slow.log ..... # Time: 120819 5:51:50 # Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000 SET timestamp=1345373510; SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; .....
42. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50", "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000", "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SET timestamp=1345373510;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" }
43. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50", "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000", "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SET timestamp=1345373510;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" }
44. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern => "^# Time:" negate => true what => "previous" } }
45. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T12:23:31.241Z", "path" => "/var/lib/mysql/mysql-slow.log" "tags" => [ [0] "multiline" ] }
46. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T12:23:31.241Z", "path" => "/var/lib/mysql/mysql-slow.log" "tags" => [ [0] "multiline" ] }
47. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;
48. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern => "^# Time:" negate => true what => "previous" } }
49. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern => "^# Time:" negate => true what => "previous" } ? }
50. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;
51. @jsvd #DevoxxPL Logstash - Grok (?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)? [ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)? [ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\ [\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?: [^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)? [ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\ ["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?: (?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+| \Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)? [ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(? =[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:( # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;
52. @jsvd #DevoxxPL Logstash - Grok Patterns
53. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
54. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
55. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
56. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
57. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
58. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
59. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern => "^# Time:" negate => true what => "previous" } grok { ... } }
60. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern => "^# Time:" negate => true what => "previous" } grok { match => ["message", "# Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: % {NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp}; \n%{GREEDYDATA:query}" ] } }
61. @jsvd #DevoxxPL { "message" => "# Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T13:32:16.328Z", "path" => "/var/lib/mysql/mysql-slow.log", "tags" => [ [0] "multiline" ], "time" => "120819 5:51:50", "query_time" => 27.115751, "lock_time" => 7.0e-05, "rows_sent" => 55996, "rows_examined" => 56000, "query_timestamp" => "1345373510", "query" => "SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;" } Added structure to the event
62. @jsvd #DevoxxPL Logstash Plugins
63. @jsvd #DevoxxPL Logstash Plugins • 53 Inputs • 47 Filters • 64 Outputs And counting..
64. @jsvd #DevoxxPL Logstash 1.5
65. @jsvd #DevoxxPL Logstash - 1.4.x and before • Core code and plugins in the same github repository • A bug in a plugin would only ship in a new Logstash
66. @jsvd #DevoxxPL Core/Plugin separation
67. @jsvd #DevoxxPL Logstash - Plugins • Plugins are Rubygems • Core and plugins can have separate release cycles • Install/Uninstall/Update Plugins • from rubygems.org, local .gem file, local path • A plugin's spec suite can be executed in its repo
68. @jsvd #DevoxxPL Logstash Plugins https://github.com/logstash-plugins
69. @jsvd #DevoxxPL Creating new plugins is easy!
70. @jsvd #DevoxxPL Logstash - New Plugins
71. @jsvd #DevoxxPL Logstash - New Input Plugin
72. @jsvd #DevoxxPL Logstash - New Filter Plugin
73. @jsvd #DevoxxPL Logstash - New Output Plugin
74. @jsvd #DevoxxPL Logstash - Then you only need to $ gem build logstash-filter-mything.gemspec Successfully built RubyGem Name: logstash-filter-mything Version: 0.0.1 File: logstash-filter-mything-0.0.1.gem $ cd ../logstash-1.5.1 $ bin/plugin install ../logstash-filter-mything-0.0.1.gem
75. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
76. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
77. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
78. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
79. @jsvd #DevoxxPL /tmp % git clone https://github.com/logstash-plugins/logstash-filter-grok Cloning into 'logstash-filter-grok'... [...] /tmp % cd logstash-filter-grok /tmp/logstash-filter-grok [master] % bundle install [...] Bundle complete! 2 Gemfile dependencies, 29 gems now installed. Use `bundle show [gemname]` to see where a bundled gem is installed. /tmp/logstash-filter-grok [master] % bundle exec rspec [...] ............................................................................ .............. Finished in 3.74 seconds 90 examples, 0 failures /tmp/logstash-filter-grok [master] % Logstash - Plugin Manager
80. @jsvd #DevoxxPL Logstash - Plugin Manager The good: +Leverages rubygems.org and Bundler +Delegates dependency management +Less code we have to write <3 +It’s the Ruby Way!
81. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Multiple usage scenarios => multiple gemsets • run logstash from a clean git clone • run core tests from a clean git clone • package a release from a clean git clone • run logstash from a release • run plugin tests from a release The fix: • patching Bundler to reduce side-effects
82. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Coding against Bundler’s API proved challenging • sticky options • side effects (bundle/.config) • "Bundler.reset!"..didn't The fix: • patch bundler
83. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Bundled versions of jar-dependencies were buggy The fix: • vendoring jars inside gems :| • makes gems a lot bigger • not a permanent solution?
84. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Hard to test - The plugin manager does a lot of disk operations - downloading and unpacking gems - updating manifest files The fix: • acceptance tests
85. @jsvd #DevoxxPL Logstash 2.x
86. @jsvd #DevoxxPL Resilience
87. @jsvd #DevoxxPL Logstash - Lack of Persistence input filter output
88. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence
89. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence
90. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence
91. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence
92. @jsvd #DevoxxPL Logstash - Adding Persistence [WIP] feature/persistent_queue https://github.com/elastic/logstash/pull/1939
93. @jsvd #DevoxxPL Logstash - Adding Persistence input filter output
94. @jsvd #DevoxxPL Logstash - Adding Persistence input filter output
95. @jsvd #DevoxxPL input filter output Logstash - Dealing with Failure
96. @jsvd #DevoxxPL input filter output ❌ Logstash - Dealing with Failure
97. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2. Discard Logstash - Dealing with Failure
98. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2. Discard 3. ?? Logstash - Dealing with Failure
99. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2. Discard 3. DLQ Logstash - Dead-letter Queue
100. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2. Discard 3. DLQ Logstash - Dead-letter Queue
101. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2. Discard 3. DLQ Logstash - Dead-letter Queue
102. @jsvd #DevoxxPL Clustering
103. @jsvd #DevoxxPL Logstash - Clustering • Provide API for controlling a logstash instance • Allow a set of logstash instances to fetch configurations from a common location • Allow dynamic configuration updates • Coordinate multiple instances to reach a cluster-level entity Add support for clustering Logstash instances https://github.com/elastic/logstash/issues/2632
104. @jsvd #DevoxxPL Introspection
105. @jsvd #DevoxxPL Logstash - Introspection • Increase visibility into a running instance • Use an (REST?) API to do runtime status queries of • health • throughput • queue sizes • latencies • Minimize impact of extraction (make it togglable?) Provide APIs to monitor pipeline https://github.com/elastic/logstash/issues/2611
106. @jsvd #DevoxxPL What else?
107. @jsvd #DevoxxPL Logstash - Other concerns a) Ensure quality of plugins • Must not alienate contributions • Testing needs to be easier • How to easily communicate the quality status b) Improve integration testing • Lot of experimentation with containers c) Better performance d) Predictable behaviour
108. @jsvd #DevoxxPL Logstash - And now? • Go play with it • Create a plugin • File an issue • Write a test • Experiment with the ELK stack • Logstash → Elasticsearch → Kibana
109. @jsvd #DevoxxPL Logstash - And now? • Go play with it • Create a plugin • File an issue • Write a test • Experiment with the ELK stack • Logstash → Elasticsearch → Kibana • Complain on IRC..
110. @jsvd #DevoxxPL Logstash - And now? • Go play with it • Create a plugin • File an issue • Write a test • Experiment with the ELK stack • Logstash → Elasticsearch → Kibana • Complain on IRC.. I’m "jsvd" on freenode#logstash
111. @jsvd #DevoxxPL End Questions?