Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why I love Logstash and you should too.

Why I love Logstash and you should too.

What is Logstash? Do you use it? What is it for? Why Ruby? How does it work?
This talk also covers what the newly released Logstash 1.5 brings and what's planned for 2.x
Presented at DevoxxPL 2015 by João Duarte on the 24th of June, 2015

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

June 24, 2015
Tweet

More Decks by Elastic Co

Other Decks in Programming

Transcript

  1. @jsvd #DevoxxPL Platinum Sponsors: Why I love Logstash and you

    should too. João Duarte elastic
  2. @jsvd #DevoxxPL • João Duarte • Portugal! • Software Engineer

    at • (Mostly) Ruby • Lover of all things Event-Driven Who Am I?
  3. @jsvd #DevoxxPL Why are we here?

  4. @jsvd #DevoxxPL Why are we here?

  5. @jsvd #DevoxxPL Show of hands

  6. @jsvd #DevoxxPL 1. What is Logstash? 2. What is it

    for? 3. Latest developments 4. The future!! Schedule
  7. @jsvd #DevoxxPL •Event processing pipeline •Plugin based •Written in Ruby

    •Open Source What is Logstash?
  8. @jsvd #DevoxxPL Plugin Based Event Processing Pipeline?? PBEPP?!

  9. @jsvd #DevoxxPL Plugin Based Event Processing Pipeline!! inputs filters outputs

    file syslog tcp websockets grok date split geoip anonymize elasticsearch pagerduty file
  10. @jsvd #DevoxxPL Logstash Use Cases

  11. @jsvd #DevoxxPL Logstash - Use cases Apache DB elasticsearch

  12. @jsvd #DevoxxPL Logstash - Use cases Apache Apache DB

  13. @jsvd #DevoxxPL Logstash - Use cases Apache Apache DB elasticsearch

    Kibana
  14. @jsvd #DevoxxPL Logstash - Use cases

  15. @jsvd #DevoxxPL Logstash - Use cases zaakceptować te informacje

  16. @jsvd #DevoxxPL Logstash - Use cases zaakceptować te informacje What?!

  17. @jsvd #DevoxxPL Logstash - Use cases … …

  18. @jsvd #DevoxxPL Logstash - Use cases

  19. @jsvd #DevoxxPL •Options available for communicating data are huge •Most

    software logging is targeted at humans •Single line, multi line •Plain text, json, xml •Log4j, Log files, Syslog, TCP, UDP •Don’t get me started on date formats Heterogeneity
  20. @jsvd #DevoxxPL Logstash - Use cases

  21. @jsvd #DevoxxPL Logstash - Use cases

  22. @jsvd #DevoxxPL Logstash - Use cases zaakceptować te informacje

  23. @jsvd #DevoxxPL Logstash - Use cases accept this information

  24. @jsvd #DevoxxPL Logstash - Use cases TCP(csv)

  25. @jsvd #DevoxxPL Logstash - Use cases TCP(csv) HTTP(json)

  26. @jsvd #DevoxxPL Logstash - Use cases

  27. @jsvd #DevoxxPL Tell me more..

  28. @jsvd #DevoxxPL Logstash - Open Source opensource http://github.com/elastic/logstash

  29. @jsvd #DevoxxPL Logstash - Ruby

  30. @jsvd #DevoxxPL Logstash - JRuby

  31. @jsvd #DevoxxPL Logstash - JRuby • Leverages JVM • Battle-proven

    GC • Opens interop with Java, Scala, Clojure • Mature libraries for EVERYTHING • VisualVM <3 • Future: 9.0.0.0, Truffle
  32. @jsvd #DevoxxPL Logstash - JRuby + Truffle • Project started

    at Oracle Labs in 2013 • Implementation of the Ruby language using • Graal dynamic compiler • Truffle AST interpreter framework • Simpler code, evolving AST • control over the compiler with Graal https://github.com/jruby/jruby/wiki/Truffle
  33. @jsvd #DevoxxPL Logstash - JRuby + Truffle $ rbenv install

    jruby-9.0.0.0+graal-dev $ rbenv shell jruby-9.0.0.0+graal-dev $ ruby -X+T -e 'puts Truffle.graal?' true
  34. @jsvd #DevoxxPL Logstash - Concepts • Events • Plugins •

    Input: capture an occurrence in outside world into an event; • Filter: transform, drop and validate events; • Output: send an event to the outside world.
  35. @jsvd #DevoxxPL Logstash Config • Text file driven configuration •

    Describe the three stages of the pipeline • Configuration changes require restart (for now..) $ bin/logstash -f my.conf Logstash startup completed ...
  36. @jsvd #DevoxxPL Logstash Config input { } filter { }

    output { }
  37. @jsvd #DevoxxPL Logstash Config input { file { path =>

    "/var/log/apache/*.log" } udp { port => 3333 } } filter { } output { }
  38. @jsvd #DevoxxPL Logstash Config input { file { path =>

    "/var/log/apache/*.log" } udp { port => 3333 } } filter { checksum { keys => ["message", "@timestamp"] } } output { }
  39. @jsvd #DevoxxPL Logstash Config input { file { path =>

    "/var/log/apache/*.log" } udp { port => 3333 } } filter { checksum { keys => ["message", "@timestamp"] } } output { elasticsearch { protocol => http } stdout { } }
  40. @jsvd #DevoxxPL • Input Plugins: - file, rabbitmq, redis, snmptrap,

    syslog, TCP, UDP, twitter, S3, etc. • Filter Plugins: - grok, date, mutate, split, multiline, ruby, etc. • Output Plugins: - elasticsearch, rabbitmq, redis, file, pagerduty, email, nagios, etc. Logstash Plugins
  41. @jsvd #DevoxxPL mysql-slow.log ..... # Time: 120819 5:51:50 # Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000 SET timestamp=1345373510; SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; .....
  42. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50",

    "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000", "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SET timestamp=1345373510;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" }
  43. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50",

    "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000", "@timestamp" => "2015-05-11T11:49:45.653Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SET timestamp=1345373510;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" } { "message" => "SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T11:49:45.654Z", "path" => "/var/lib/mysql/mysql-slow.log" }
  44. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern

    => "^# Time:" negate => true what => "previous" } }
  45. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50\n#

    Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T12:23:31.241Z", "path" => "/var/lib/mysql/mysql-slow.log" "tags" => [ [0] "multiline" ] }
  46. @jsvd #DevoxxPL mysql-slow.log { "message" => "# Time: 120819 5:51:50\n#

    Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T12:23:31.241Z", "path" => "/var/lib/mysql/mysql-slow.log" "tags" => [ [0] "multiline" ] }
  47. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;
  48. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern

    => "^# Time:" negate => true what => "previous" } }
  49. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern

    => "^# Time:" negate => true what => "previous" } ? }
  50. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;
  51. @jsvd #DevoxxPL Logstash - Grok (?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)? [ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?

    [ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\ [\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?: [^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)? [ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\ ["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?: (?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+| \Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)? [ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(? =[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:( # Time: 120819 5:51:50\n# Query_time: 27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;
  52. @jsvd #DevoxxPL Logstash - Grok Patterns

  53. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
  54. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
  55. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
  56. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
  57. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
  58. @jsvd #DevoxxPL Logstash - Grok # Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC; # Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: %{NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp};\n%{GREEDYDATA:query}
  59. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern

    => "^# Time:" negate => true what => "previous" } grok { ... } }
  60. @jsvd #DevoxxPL Logstash - Filters filter { multiline { pattern

    => "^# Time:" negate => true what => "previous" } grok { match => ["message", "# Time: %{GREEDYDATA:time}\n# Query_time: %{NUMBER:query_time:float} Lock_time: %{NUMBER:lock_time:float} Rows_sent: %{NUMBER:rows_sent:int} Rows_examined: % {NUMBER:rows_examined:int}\nSET timestamp= %{NUMBER:query_timestamp}; \n%{GREEDYDATA:query}" ] } }
  61. @jsvd #DevoxxPL { "message" => "# Time: 120819 5:51:50\n# Query_time:

    27.115751 Lock_time: 0.000070 Rows_sent: 55996 Rows_examined: 56000\nSET timestamp=1345373510;\nSELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;", "@timestamp" => "2015-05-11T13:32:16.328Z", "path" => "/var/lib/mysql/mysql-slow.log", "tags" => [ [0] "multiline" ], "time" => "120819 5:51:50", "query_time" => 27.115751, "lock_time" => 7.0e-05, "rows_sent" => 55996, "rows_examined" => 56000, "query_timestamp" => "1345373510", "query" => "SELECT ID FROM wp_posts WHERE post_parent = 17 AND post_status IN ( 'publish', 'closed' ) AND post_type = 'topic' ORDER BY ID DESC;" } Added structure to the event
  62. @jsvd #DevoxxPL Logstash Plugins

  63. @jsvd #DevoxxPL Logstash Plugins • 53 Inputs • 47 Filters

    • 64 Outputs And counting..
  64. @jsvd #DevoxxPL Logstash 1.5

  65. @jsvd #DevoxxPL Logstash - 1.4.x and before • Core code

    and plugins in the same github repository • A bug in a plugin would only ship in a new Logstash
  66. @jsvd #DevoxxPL Core/Plugin separation

  67. @jsvd #DevoxxPL Logstash - Plugins • Plugins are Rubygems •

    Core and plugins can have separate release cycles • Install/Uninstall/Update Plugins • from rubygems.org, local .gem file, local path • A plugin's spec suite can be executed in its repo
  68. @jsvd #DevoxxPL Logstash Plugins https://github.com/logstash-plugins

  69. @jsvd #DevoxxPL Creating new plugins is easy!

  70. @jsvd #DevoxxPL Logstash - New Plugins

  71. @jsvd #DevoxxPL Logstash - New Input Plugin

  72. @jsvd #DevoxxPL Logstash - New Filter Plugin

  73. @jsvd #DevoxxPL Logstash - New Output Plugin

  74. @jsvd #DevoxxPL Logstash - Then you only need to $

    gem build logstash-filter-mything.gemspec Successfully built RubyGem Name: logstash-filter-mything Version: 0.0.1 File: logstash-filter-mything-0.0.1.gem $ cd ../logstash-1.5.1 $ bin/plugin install ../logstash-filter-mything-0.0.1.gem
  75. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing

    logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
  76. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing

    logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
  77. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing

    logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
  78. @jsvd #DevoxxPL /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing

    logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash - Plugin Manager
  79. @jsvd #DevoxxPL /tmp % git clone https://github.com/logstash-plugins/logstash-filter-grok Cloning into 'logstash-filter-grok'...

    [...] /tmp % cd logstash-filter-grok /tmp/logstash-filter-grok [master] % bundle install [...] Bundle complete! 2 Gemfile dependencies, 29 gems now installed. Use `bundle show [gemname]` to see where a bundled gem is installed. /tmp/logstash-filter-grok [master] % bundle exec rspec [...] ............................................................................ .............. Finished in 3.74 seconds 90 examples, 0 failures /tmp/logstash-filter-grok [master] % Logstash - Plugin Manager
  80. @jsvd #DevoxxPL Logstash - Plugin Manager The good: +Leverages rubygems.org

    and Bundler +Delegates dependency management +Less code we have to write <3 +It’s the Ruby Way!
  81. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Multiple

    usage scenarios => multiple gemsets • run logstash from a clean git clone • run core tests from a clean git clone • package a release from a clean git clone • run logstash from a release • run plugin tests from a release The fix: • patching Bundler to reduce side-effects
  82. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Coding

    against Bundler’s API proved challenging • sticky options • side effects (bundle/.config) • "Bundler.reset!"..didn't The fix: • patch bundler
  83. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Bundled

    versions of jar-dependencies were buggy The fix: • vendoring jars inside gems :| • makes gems a lot bigger • not a permanent solution?
  84. @jsvd #DevoxxPL Logstash - Plugin Manager The bad: - Hard

    to test - The plugin manager does a lot of disk operations - downloading and unpacking gems - updating manifest files The fix: • acceptance tests
  85. @jsvd #DevoxxPL Logstash 2.x

  86. @jsvd #DevoxxPL Resilience

  87. @jsvd #DevoxxPL Logstash - Lack of Persistence input filter output

  88. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence

  89. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence

  90. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence

  91. @jsvd #DevoxxPL input filter output Logstash - Lack of Persistence

  92. @jsvd #DevoxxPL Logstash - Adding Persistence [WIP] feature/persistent_queue https://github.com/elastic/logstash/pull/1939

  93. @jsvd #DevoxxPL Logstash - Adding Persistence input filter output

  94. @jsvd #DevoxxPL Logstash - Adding Persistence input filter output

  95. @jsvd #DevoxxPL input filter output Logstash - Dealing with Failure

  96. @jsvd #DevoxxPL input filter output ❌ Logstash - Dealing with

    Failure
  97. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2.

    Discard Logstash - Dealing with Failure
  98. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2.

    Discard 3. ?? Logstash - Dealing with Failure
  99. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2.

    Discard 3. DLQ Logstash - Dead-letter Queue
  100. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2.

    Discard 3. DLQ Logstash - Dead-letter Queue
  101. @jsvd #DevoxxPL input filter output ❌ Options: 1. Retry 2.

    Discard 3. DLQ Logstash - Dead-letter Queue
  102. @jsvd #DevoxxPL Clustering

  103. @jsvd #DevoxxPL Logstash - Clustering • Provide API for controlling

    a logstash instance • Allow a set of logstash instances to fetch configurations from a common location • Allow dynamic configuration updates • Coordinate multiple instances to reach a cluster-level entity Add support for clustering Logstash instances https://github.com/elastic/logstash/issues/2632
  104. @jsvd #DevoxxPL Introspection

  105. @jsvd #DevoxxPL Logstash - Introspection • Increase visibility into a

    running instance • Use an (REST?) API to do runtime status queries of • health • throughput • queue sizes • latencies • Minimize impact of extraction (make it togglable?) Provide APIs to monitor pipeline https://github.com/elastic/logstash/issues/2611
  106. @jsvd #DevoxxPL What else?

  107. @jsvd #DevoxxPL Logstash - Other concerns a) Ensure quality of

    plugins • Must not alienate contributions • Testing needs to be easier • How to easily communicate the quality status b) Improve integration testing • Lot of experimentation with containers c) Better performance d) Predictable behaviour
  108. @jsvd #DevoxxPL Logstash - And now? • Go play with

    it • Create a plugin • File an issue • Write a test • Experiment with the ELK stack • Logstash → Elasticsearch → Kibana
  109. @jsvd #DevoxxPL Logstash - And now? • Go play with

    it • Create a plugin • File an issue • Write a test • Experiment with the ELK stack • Logstash → Elasticsearch → Kibana • Complain on IRC..
  110. @jsvd #DevoxxPL Logstash - And now? • Go play with

    it • Create a plugin • File an issue • Write a test • Experiment with the ELK stack • Logstash → Elasticsearch → Kibana • Complain on IRC.. I’m "jsvd" on freenode#logstash
  111. @jsvd #DevoxxPL End Questions?