Writing Logstash Plugins in the 5.X Era

Learn how to write plugins for Logstash and what goodies the 5.X version line will bring to the plugin developer. João will cover the basics, as well as how to write a Java-based plugin, how to instrument your plugin with metrics, and more!

João Duarte l Software Engineer l Elastic

Transcript

1. Elastic March 9th 2017 @jsvd Writing Logstash Plugins in the 5.x Era João Duarte, Log Whisperer
2. 2 Logstash < 1.5 Logstash >= 1.5 Logstash 5.x • All plugins in logstash repo • Plugin fix => wait for release • Use custom plugins with -- plugin-path=path • One plugin => One Repository • Each plugin is a Ruby gem • Introduced Plugin Manager • Install/Update/Remove • Logstash Event in Java • Plugin Generator • Metric Subsystem A Brief History of Logstash Plugins
3. Why would I create a plugin?
4. Reasons to create a Logstash Plugin 4 Read from an uncommon source Parse events with a new, complex data format Send events to a new destination (e.g. new message broker) Custom data transformation with a Ruby (or Java) library Improve performance with a custom fit transformation process
5. Writing a Logstash Plugin What do I need?
6. 6 • Logstash Release https://www.elastic.co/downloads/logstash • Java - JDK8 • JRuby 1.7.24 or higher (but not 9.x.x.x) What do you need?
7. $ bin/logstash-plugin generate -h Usage: bin/logstash-plugin generate [OPTIONS] Options: --type TYPE Type of the plugin {input, filter, codec, output}s --name PLUGIN Name of the new plugin --path PATH Location where the plugin skeleton will be created (default: "/tmp/logstash-5.2.1") -h, --help print help Create an empty skeleton of a plugin The Plugin Generator 7
8. $ bin/logstash-plugin generate --type filter --name geocoder --path /tmp Create an empty skeleton of a plugin The Plugin Generator 8
9. $ bin/logstash-plugin generate --type filter --name geocoder --path /tmp Creating /tmp/logstash-filter-geocoder create logstash-filter-geocoder/CHANGELOG.md create logstash-filter-geocoder/CONTRIBUTORS create logstash-filter-geocoder/DEVELOPER.md create logstash-filter-geocoder/Gemfile create logstash-filter-geocoder/lib/logstash/filters/geocoder.rb create logstash-filter-geocoder/LICENSE create logstash-filter-geocoder/logstash-filter-geocoder.gemspec create logstash-filter-geocoder/Rakefile create logstash-filter-geocoder/README.md create logstash-filter-geocoder/spec/filters/geocoder_spec.rb create logstash-filter-geocoder/spec/spec_helper.rb Create an empty skeleton of a plugin The Plugin Generator 9
10. Writing a Logstash Plugin Setting things up
11. /tmp/logstash-filter-geocoder $ vim logstash-filter-geocoder.gemspec /tmp/logstash-filter-geocoder $ 1. Edit the gemspec file (e.g. define author, description) Setting Up 11
12. /tmp/logstash-filter-geocoder $ vim logstash-filter-geocoder.gemspec /tmp/logstash-filter-geocoder $ bundle install Fetching gem metadata from https://rubygems.org/......... # stuff .. Bundle complete! 2 Gemfile dependencies, 49 gems now installed. /tmp/logstash-filter-geocoder $ 2. Install all the plugin dependencies Setting Up 12
13. /tmp/logstash-filter-geocoder $ vim logstash-filter-geocoder.gemspec /tmp/logstash-filter-geocoder $ bundle install Fetching gem metadata from https://rubygems.org/......... # stuff .. Bundle complete! 2 Gemfile dependencies, 49 gems now installed. /tmp/logstash-filter-geocoder $ bundle exec rspec Randomized with seed 52631 . Finished in 0.098 seconds (files took 4.53 seconds to load) 1 example, 0 failures /tmp/logstash-filter-geocoder $ 3. Run the specs Setting Up 13
14. /tmp/logstash-5.2.1 % grep geocoder Gemfile gem "logstash-filter-geocoder", :path => "/tmp/logstash-filter-geocoder" /tmp/logstash-5.2.1 % bin/logstash-plugin install --no-verify Installing... Installation successful /tmp/logstash-5.2.1 % bin/logstash -e "filter { geocoder { ... } }" 4. Experiment with the plugin in Logstash Setting Up 14
15. Writing a Logstash Plugin Common plugin mechanics
16. class LogStash::Filters::Myfilter < LogStash::Filters::Base # config parameters become instance variables # in this case: @message config :message, :validate => :string, :default => "Hello World!" config :port, :validate => :number config :mode, :validate => ["server", "client"], :default => "server" config :password, :validate => :password config :ssl_cert, :validate => :path end Plugin Configuration Parameters DSL Common plugin mechanics 16
17. class LogStash::Inputs::MyInput < LogStash::Inputs::Base config_name "myinput" public def register @client = create_socket( .. ) @buffer = [] # other initializations.. end end The "register" method Common plugin mechanics 17
18. Writing a Logstash Plugin Plugin APIs
19. Plugin execution sequence Writing a Logstash Plugin 19 Input#run I C F C O Codec#decode Filter#multi_filter Codec#multi_encode Output# multi_receive_encoded
20. Plugin execution sequence Writing a Logstash Plugin 20 Input#run I C F C O Codec#decode Filter#multi_filter Codec#multi_encode Output# multi_receive_encoded Byte Stream -> Logstash Event
21. Plugin execution sequence Writing a Logstash Plugin 21 Input#run I C F C O Codec#decode Filter#multi_filter Codec#multi_encode Output# multi_receive_encoded Batches of events
22. Plugin execution sequence Writing a Logstash Plugin 22 Input#run I C F C O Codec#decode Filter#filter Codec#encode Output# receive Batches of events
23. Metrics, Metrics, Metrics Instrumenting the execution of your plugin
24. def filter(event) matched = # .... perform grok matching .... if matched metric.increment(:matches) filter_matched(event) else metric.increment(:failures) @tag_on_failure.each {|tag| event.tag(tag)} end rescue ::LogStash::Filters::Grok::TimeoutException => e @logger.warn(e.message) metric.increment(:timeouts) event.tag(@tag_on_timeout) end Example from grok filter (extract from the filter method) Collecting metrics in a plugin 24
25. % curl -s -XGET http://localhost:9600/_node/stats | jq ".pipeline.plugins.filters[0]" { "id": "d7bfd0fe52eb920880b2cf45c8910aeec4dff3d0-1", "events": { "duration_in_millis": 4, "in": 2, "out": 2 }, "matches": 1, "failures": 1, "patterns_per_field": { "message": 1 }, "name": "grok" } Plugin metrics appear in /_node/stats and /_node/stats/pipeline Collecting metrics in a plugin 25
26. % curl -s -XGET http://localhost:9600/_node/stats | jq ".pipeline.plugins.filters[0]" { "id": "d7bfd0fe52eb920880b2cf45c8910aeec4dff3d0-1", "events": { "duration_in_millis": 4, "in": 2, "out": 2 }, "matches": 1, "failures": 1, "patterns_per_field": { "message": 1 }, "name": "grok" } Plugin metrics appear in /_node/stats and /_node/stats/pipeline Collecting metrics in a plugin 26 Built-in metrics for each plugin
27. % curl -s -XGET http://localhost:9600/_node/stats | jq ".pipeline.plugins.filters[0]" { "id": "d7bfd0fe52eb920880b2cf45c8910aeec4dff3d0-1", "events": { "duration_in_millis": 4, "in": 2, "out": 2 }, "matches": 1, "failures": 1, "patterns_per_field": { "message": 1 }, "name": "grok" } Plugin metrics appear in /_node/stats and /_node/stats/pipeline Collecting metrics in a plugin 27 Plugin (Grok) specific metrics
28. # All plugins provide a "metric" object that you can interact with # to generate metrics # metrics are initialized when used the first time metric.increment(:matches) # increments by 1 (starts at 0) metric.decrement(:failures, 100) # decrements by 100 metric.gauge(:last_status, "success") # sets metric value to "success" # elapsed time in milliseconds metric.time(:execution) { perform_computation(event) } # or, alternatively time = metric.time(:execution) perform_computation(event) time.stop # calling stop will report the elapsed time Different kinds of metrics Collecting metrics in a plugin 28
29. Java, Java, Java Because sometimes Ruby just won't do
30. Reasons to use Java 3 0 Not familiar with Ruby More familiar with Java Use a Java library Performance
31. Filter Using a Java Library 3 1 Pipeline Logstash Event Java Library
32. Filter Using a Java Library 3 2 Pipeline Logstash Event Java Library JRuby Proxy
33. require 'commons-codec/commons-codec/1.10/commons-codec-1.10.jar' java_import 'org.apache.commons.codec.digest.DigestUtils' class LogStash::Filters::Shajruby < LogStash::Filters::Base config_name "shajruby" config :source, :validate => :string, :default => "message" def register; end public def multi_filter(events) events.each do |event| sha256hex = DigestUtils.sha256Hex(event.get(@source)); event.set("hexdigest", sha256hex) filter_matched(event) end end end Method 1: call Java libraries from JRuby Using a Java Library 33
34. Filter Using Java 3 4 Pipeline Logstash Event Java Library
35. Filter Using Java 35 Pipeline Java Library JRuby Ext Logstash Event
36. Filter Using Java + Logstash Event in Java 36 Pipeline Java Library JRuby Ext Logstash Event Java Filter Code
37. class LogStash::Filters::Shajava < LogStash::Filters::Base config_name "shajava" config :source, :validate => :string, :default => "message" public def register @digester = org.logstash.filters.ShaJava.new(@source) end public def multi_filter(events) @digester.receive(events) end end Method 2: call your Java code from JRuby Ruby side Using Java 37
38. public class ShaJava { private static Logger logger = LogManager.getLogger(); private String sourceField; public ShaJava(String sourceField) { this.sourceField = sourceField; } public List<RubyEvent> receive(List<RubyEvent> rubyEvents) { for (RubyEvent rubyEvent : rubyEvents) { Event event = rubyEvent.getEvent(); String text = (String) event.getField(sourceField); event.setField("digest", digest(text)); } return rubyEvents; } private String digest(String text) { return DigestUtils.sha256Hex(text); } } Method 2: call your Java code from JRuby Java side Java, Java, Java 38
39. public class ShaJava { private static Logger logger = LogManager.getLogger(); private String sourceField; public ShaJava(String sourceField) { this.sourceField = sourceField; } public List<RubyEvent> receive(List<RubyEvent> rubyEvents) { for (RubyEvent rubyEvent : rubyEvents) { Event event = rubyEvent.getEvent(); String text = (String) event.getField(sourceField); event.setField("digest", digest(text)); } return rubyEvents; } private String digest(String text) { return DigestUtils.sha256Hex(text); } } Method 2: call your Java code from JRuby Java side Java, Java, Java 39 For a complete example of this strategy, see the date filter: https://github.com/logstash-plugins/logstash-filter-date
40. Filter Using Java + Logstash Event in Java 40 Pipeline Java Library JRuby Ext Logstash Event Java Filter Code
41. Filter Using Java + Logstash Event in Java 41 Pipeline Java Library JRuby Ext Logstash Event Java Filter Code JRuby Proxy
42. Filter Using Java + Logstash Event in Java 42 Pipeline Java Library JRuby Ext Logstash Event Java Filter Code JRuby Extension
43. public class ShaJava { } Method 3: create a JRuby Extension Java Side Java, Java, Java 43 For a complete example of this strategy, see the dissect filter: https://github.com/logstash-plugins/logstash-filter-dissect
44. Logstash Plugin Development 4 4 Improve OOTB Java Plugin creation experience Next tasks and goals Continue improving tooling and documentation for plugin development Convert other core classes to Java: plugin base classes, pipeline class
47. Thank you!
48. 48 Questions? Visit us at the AMA
49. www.elastic.c o