Slide 1
Slide 1 text
Logstash 1.5
and the three silly plugins
João Duarte - @jsvd
Slide 2
Slide 2 text
Agenda
• Logstash 1.5
• Three silly plugins
• input
• filter
• output
• Demo!
Slide 3
Slide 3 text
show of hands
Slide 4
Slide 4 text
1.5
Slide 5
Slide 5 text
Logstash 1.5
• Separation between core and plugins
• New plugins: Kafka (input and output), JMX input, RSS
input, Heartbeat input
• Performance improvements: grok filter, json
(de)serialization
• Windows love
• documentation moved to https://elastic.co/guide/en/
logstash
Slide 6
Slide 6 text
• Plugins are Rubygems
• Core and plugins can have separate release cycles
• Install/Uninstall/Update Plugins
• from rubygems.org, local .gem file, local path
• A plugin's spec suite can be executed in its repo
Plugin system
Slide 7
Slide 7 text
Logstash 1.5
+180 plugins | 1 plugin => 1 github repo
Slide 8
Slide 8 text
/tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr
Validating logstash-filter-cidr
Installing logstash-filter-cidr
Installation successful
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr
Uninstalling logstash-filter-cidr
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin update
........
Updated logstash-filter-geoip 0.1.9 to 0.1.10
Updated logstash-input-kafka 0.1.14 to 0.1.15
Updated logstash-output-elasticsearch 0.2.4 to 0.2.5
/tmp/logstash-1.5.0 %
Logstash 1.5
bin/plugin
Slide 9
Slide 9 text
/tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr
Validating logstash-filter-cidr
Installing logstash-filter-cidr
Installation successful
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr
Uninstalling logstash-filter-cidr
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin update
........
Updated logstash-filter-geoip 0.1.9 to 0.1.10
Updated logstash-input-kafka 0.1.14 to 0.1.15
Updated logstash-output-elasticsearch 0.2.4 to 0.2.5
/tmp/logstash-1.5.0 %
Logstash 1.5
bin/plugin
Slide 10
Slide 10 text
/tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr
Validating logstash-filter-cidr
Installing logstash-filter-cidr
Installation successful
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr
Uninstalling logstash-filter-cidr
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin update
........
Updated logstash-filter-geoip 0.1.9 to 0.1.10
Updated logstash-input-kafka 0.1.14 to 0.1.15
Updated logstash-output-elasticsearch 0.2.4 to 0.2.5
/tmp/logstash-1.5.0 %
Logstash 1.5
bin/plugin
Slide 11
Slide 11 text
/tmp/logstash-1.5.0 % bin/plugin install logstash-filter-cidr
Validating logstash-filter-cidr
Installing logstash-filter-cidr
Installation successful
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin uninstall logstash-filter-cidr
Uninstalling logstash-filter-cidr
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
/tmp/logstash-1.5.0 %
/tmp/logstash-1.5.0 % bin/plugin update
........
Updated logstash-filter-geoip 0.1.9 to 0.1.10
Updated logstash-input-kafka 0.1.14 to 0.1.15
Updated logstash-output-elasticsearch 0.2.4 to 0.2.5
/tmp/logstash-1.5.0 %
Logstash 1.5
bin/plugin
Slide 12
Slide 12 text
/tmp % git clone https://github.com/logstash-plugins/logstash-filter-
grok
Cloning into 'logstash-filter-grok'...
[...]
/tmp % cd logstash-filter-grok
/tmp/logstash-filter-grok [master] % bundle install
[...]
Bundle complete! 2 Gemfile dependencies, 29 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
/tmp/logstash-filter-grok [master] % bundle exec rspec
[...]
.......................................................................
...................
Finished in 3.74 seconds
90 examples, 0 failures
/tmp/logstash-filter-grok [master] %
Logstash 1.5
bin/plugin
Slide 13
Slide 13 text
The story of the 3 silly plugins
Slide 14
Slide 14 text
Logstash
not just for logs!
Slide 15
Slide 15 text
Logstash
not just for logs!
Slide 16
Slide 16 text
Logstash
burglar alert system!!
Slide 17
Slide 17 text
Logstash
burglar alert system!!
!!!!!!!
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
logstash-input-imagesnap
Slide 20
Slide 20 text
imagesnap
Slide 21
Slide 21 text
logstash-input
Slide 22
Slide 22 text
logstash-input-imagesnap
Slide 23
Slide 23 text
logstash-filter-imagedetect
!!!!!!
.jpg
.jpg
Slide 24
Slide 24 text
logstash-filter
Slide 25
Slide 25 text
imagedetect
http://mindmeat.blogspot.pt/2008/07/java-image-comparison.html
Slide 26
Slide 26 text
logstash-filter-imagecompare
Slide 27
Slide 27 text
"Nous ne comprenons
pas tout, mais nous
n’insultons rien."
logstash-output-say
Slide 28
Slide 28 text
say
Slide 29
Slide 29 text
logstash-output
Slide 30
Slide 30 text
logstash-output-say
Slide 31
Slide 31 text
Now what?
Slide 32
Slide 32 text
imagesnap
imagecompare
say
if different {
}
Slide 33
Slide 33 text
imagesnap
imagecompare
say
if different {
}
Slide 34
Slide 34 text
imagesnap
imagecompare
say
if different {
}
Slide 35
Slide 35 text
imagesnap
imagecompare
say
if different {
}
Slide 36
Slide 36 text
imagesnap
imagecompare
say
if different {
}
Slide 37
Slide 37 text
imagesnap
imagecompare
say
if different {
}
different
Slide 38
Slide 38 text
imagesnap
imagecompare
say
if different {
}
different
Slide 39
Slide 39 text
imagesnap
imagecompare
say
if different {
} different
Slide 40
Slide 40 text
Alright let’s build it
Slide 41
Slide 41 text
~/experiments/meetup_fr_20150526 % tree -L 1
.
!"" logstash-1.5.0/
!"" logstash-filter-imagecompare/
!"" logstash-input-imagesnap/
!"" logstash-output-say/
Slide 42
Slide 42 text
% tail logstash-1.5.0/Gemfile
gem "logstash-input-imagesnap",
:path => "~/experiments/meetup_fr_20150526/logstash-
input-imagesnap"
gem "logstash-filter-imagecompare",
:path => "~/experiments/experiments/meetup_fr_20150526/
logstash-filter-imagecompare"
gem "logstash-output-say",
:path => "~/experiments/meetup_fr_20150526/logstash-
output-say"
logstash-1.5.0/Gemfile
Slide 43
Slide 43 text
input {
imagesnap {}
}
filter {
imagecompare {
add_tag => ['different']
}
}
output {
if 'different' in [tags] {
say {
voice => Thomas
text => "Intrus! Intrus ! Allez-vous en! Je
vais appeler la police!"
}
}
}
demo.conf
Slide 44
Slide 44 text
Demo time!