Logstash 1.5 and the three silly plugins

In this 20 minute presentation I show how the separation between logstash core and the plugins in 1.5 allows you to easily create plugins and use them to suit your needs…even if the needs are..well..silly. Talk given at Elastic FR meetup group.

1. Logstash 1.5 and the three silly plugins João Duarte - @jsvd
2. Agenda • Logstash 1.5 • Three silly plugins • input • filter • output • Demo!
3. show of hands
4. 1.5
5. Logstash 1.5 • Separation between core and plugins • New plugins: Kafka (input and output), JMX input, RSS input, Heartbeat input • Performance improvements: grok filter, json (de)serialization • Windows love • documentation moved to https://elastic.co/guide/en/ logstash
6. • Plugins are Rubygems • Core and plugins can have separate release cycles • Install/Uninstall/Update Plugins • from rubygems.org, local .gem file, local path • A plugin's spec suite can be executed in its repo Plugin system
7. Logstash 1.5 +180 plugins | 1 plugin => 1 github repo
8. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
9. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
10. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
11. /tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr Validating logstash-filter-cidr Installing logstash-filter-cidr Installation successful /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr Uninstalling logstash-filter-cidr /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin list tcp logstash-input-tcp logstash-output-tcp /tmp/logstash-1.5.0 % /tmp/logstash-1.5.0 % bin/plugin update ........ Updated logstash-filter-geoip 0.1.9 to 0.1.10 Updated logstash-input-kafka 0.1.14 to 0.1.15 Updated logstash-output-elasticsearch 0.2.4 to 0.2.5 /tmp/logstash-1.5.0 % Logstash 1.5 bin/plugin
12. /tmp % git clone https://github.com/logstash-plugins/logstash-filter- grok Cloning into 'logstash-filter-grok'... [...] /tmp % cd logstash-filter-grok /tmp/logstash-filter-grok [master] % bundle install [...] Bundle complete! 2 Gemfile dependencies, 29 gems now installed. Use `bundle show [gemname]` to see where a bundled gem is installed. /tmp/logstash-filter-grok [master] % bundle exec rspec [...] ....................................................................... ................... Finished in 3.74 seconds 90 examples, 0 failures /tmp/logstash-filter-grok [master] % Logstash 1.5 bin/plugin
13. The story of the 3 silly plugins
14. Logstash not just for logs!
15. Logstash not just for logs!
16. Logstash burglar alert system!!
17. Logstash burglar alert system!! !!!!!!!
19. logstash-input-imagesnap
20. imagesnap
21. logstash-input
22. logstash-input-imagesnap
23. logstash-filter-imagedetect !!!!!! .jpg .jpg
24. logstash-filter
25. imagedetect http://mindmeat.blogspot.pt/2008/07/java-image-comparison.html
26. logstash-filter-imagecompare
27. "Nous ne comprenons pas tout, mais nous n’insultons rien." logstash-output-say
28. say
29. logstash-output
30. logstash-output-say
31. Now what?
32. imagesnap imagecompare say if different { }
33. imagesnap imagecompare say if different { }
34. imagesnap imagecompare say if different { }
35. imagesnap imagecompare say if different { }
36. imagesnap imagecompare say if different { }
37. imagesnap imagecompare say if different { } different
38. imagesnap imagecompare say if different { } different
39. imagesnap imagecompare say if different { } different
40. Alright let’s build it
41. ~/experiments/meetup_fr_20150526 % tree -L 1 . !"" logstash-1.5.0/ !"" logstash-filter-imagecompare/ !"" logstash-input-imagesnap/ !"" logstash-output-say/
42. % tail logstash-1.5.0/Gemfile gem "logstash-input-imagesnap", :path => "~/experiments/meetup_fr_20150526/logstash- input-imagesnap" gem "logstash-filter-imagecompare", :path => "~/experiments/experiments/meetup_fr_20150526/ logstash-filter-imagecompare" gem "logstash-output-say", :path => "~/experiments/meetup_fr_20150526/logstash- output-say" logstash-1.5.0/Gemfile
43. input { imagesnap {} } filter { imagecompare { add_tag => ['different'] } } output { if 'different' in [tags] { say { voice => Thomas text => "Intrus! Intrus ! Allez-vous en! Je vais appeler la police!" } } } demo.conf
44. Demo time!