Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logstash 1.5 and the three silly plugins

Logstash 1.5 and the three silly plugins

In this 20 minute presentation I show how the separation between logstash core and the plugins in 1.5 allows you to easily create plugins and use them to suit your needs…even if the needs are..well..silly. Talk given at Elastic FR meetup group.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

May 26, 2015
Tweet

More Decks by Elastic Co

Other Decks in Programming

Transcript

  1. Logstash 1.5 and the three silly plugins João Duarte -

    @jsvd
  2. Agenda • Logstash 1.5 • Three silly plugins • input

    • filter • output • Demo!
  3. show of hands

  4. 1.5

  5. Logstash 1.5 • Separation between core and plugins • New

    plugins: Kafka (input and output), JMX input, RSS input, Heartbeat input • Performance improvements: grok filter, json (de)serialization • Windows love • documentation moved to https://elastic.co/guide/en/ logstash
  6. • Plugins are Rubygems • Core and plugins can have

    separate release cycles • Install/Uninstall/Update Plugins • from rubygems.org, local .gem file, local path • A plugin's spec suite can be executed in its repo Plugin system
  7. Logstash 1.5 +180 plugins | 1 plugin => 1 github

    repo
  8. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation

    successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
  9. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation

    successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
  10. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation

    successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
  11. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation

    successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
  12. /tmp % git clone https://github.com/logstash-plugins/logstash-filter- grok Cloning into 'logstash-filter-grok'... [...]

    /tmp % cd logstash-filter-grok /tmp/logstash-filter-grok [master] % bundle install [...] Bundle complete! 2 Gemfile dependencies, 29 gems now installed. Use `bundle show [gemname]` to see where a bundled gem is installed. /tmp/logstash-filter-grok [master] % bundle exec rspec [...] ....................................................................... ................... Finished in 3.74 seconds 90 examples, 0 failures /tmp/logstash-filter-grok [master] % Logstash 1.5 bin/plugin
  13. The story of the 3 silly plugins

  14. Logstash not just for logs!

  15. Logstash not just for logs!

  16. Logstash burglar alert system!!

  17. Logstash burglar alert system!! !!!!!!!

  18. None
  19. logstash-input-imagesnap

  20. imagesnap

  21. logstash-input

  22. logstash-input-imagesnap

  23. logstash-filter-imagedetect !!!!!! .jpg .jpg

  24. logstash-filter

  25. imagedetect http://mindmeat.blogspot.pt/2008/07/java-image-comparison.html

  26. logstash-filter-imagecompare

  27. "Nous ne comprenons pas tout, mais nous n’insultons rien." logstash-output-say

  28. say

  29. logstash-output

  30. logstash-output-say

  31. Now what?

  32. imagesnap imagecompare say if different { }

  33. imagesnap imagecompare say if different { }

  34. imagesnap imagecompare say if different { }

  35. imagesnap imagecompare say if different { }

  36. imagesnap imagecompare say if different { }

  37. imagesnap imagecompare say if different { } different

  38. imagesnap imagecompare say if different { } different

  39. imagesnap imagecompare say if different { } different

  40. Alright let’s build it

  41. ~/experiments/meetup_fr_20150526 % tree -L 1 . !"" logstash-1.5.0/ !"" logstash-filter-imagecompare/

    !"" logstash-input-imagesnap/ !"" logstash-output-say/
  42. % tail logstash-1.5.0/Gemfile gem "logstash-input-imagesnap", :path => "~/experiments/meetup_fr_20150526/logstash- input-imagesnap" gem

    "logstash-filter-imagecompare", :path => "~/experiments/experiments/meetup_fr_20150526/ logstash-filter-imagecompare" gem "logstash-output-say", :path => "~/experiments/meetup_fr_20150526/logstash- output-say" logstash-1.5.0/Gemfile
  43. input { imagesnap {} } filter { imagecompare { add_tag

    => ['different'] } } output { if 'different' in [tags] { say { voice => Thomas text => "Intrus! Intrus ! Allez-vous en! Je vais appeler la police!" } } } demo.conf
  44. Demo time!