Ingesting logs with style

Slide 1

Slide 1 text

‹#› What has been cooking lately in Logstash world. Ingesting Logs with style

Slide 2

Slide 2 text

$whoami Pere Urbon-Bayes (Software Engineer since ever) Have always worked with databases, data and analytics. GraphDevRoom@FOSDEM When not coding I enjoy my time with my wife and kid. I also enjoy movies and tv series, and used to like running, too. Basically, I’m doing everything to enjoy live. 2

Slide 3

Slide 3 text

3 Life demo?

Slide 4

Slide 4 text

4 Ingredients

Slide 5

Slide 5 text

5 Metrics

Slide 6

Slide 6 text

What is the current Logstash throughput? 6 Logstash is blocked?, ask for a hot threads dump. Worried about OOM? ask for memory consumption. Broken by plugin instance. Per instance and node. to be continued……

Slide 7

Slide 7 text

Web API resources • Current web api resources: • http://localhost:9600/_node/hot_threads • http://localhost:9600/_node/stats/ • http://localhost:9600/_node/stats/events • http://localhost:9600/_stats/jvm • http://localhost:9600/_plugins/ • ….. 7

Slide 8

Slide 8 text

Slide 9

Slide 9 text

9 Next: Monitoring UI

Slide 10

Slide 10 text

10 Configuration Management

Slide 11

Slide 11 text

Step 1: Config Reloading Previously: Any config change made to file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 11 File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart Why? How?

Slide 12

Slide 12 text

12 role: frontend-logs 1 Create a role, upload config

Slide 13

Slide 13 text

13 The Java Event

Slide 14

Slide 14 text

The java reincarnation • Logstash is based on JRuby, Ruby on top of the JVM • Some core components are being rewritten in plain java. • First step: the event representation. • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. • Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. • Will enable not just ruby plugins, but using any JVM languages • And not only about this, but performance ….. 14

Slide 15

Slide 15 text

15 logstash-benchmarks.elastic.co

Slide 16

Slide 16 text

16 The other changes

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Native offline plugins management support • Build your plugins bundle to be used in offline environments • bin/logstash-plugin pack • bin/logstash-plugin unpack • bin/logstash-plugin install --local • Use your own private plugins server • Geminabox, Gemirro, Artifactory, Gemfury, … • Edit the Gemfile to use the new server. Manage plugins in offline environments made easier 18

Slide 19

Slide 19 text

19 Pipeline, the next generation

Slide 20

Slide 20 text

The pipeline components 20 I F O I F O

Slide 21

Slide 21 text

21 Filebeat, the new log harvester gen

Slide 22

Slide 22 text

Long life to Logstash Forwarder, welcome Filebeat • LSF is deprecated, next generation harvester is Filebeat. • Written in Go and configured in YAML. • Available as deb, rpm, linux (32 and 64bit), mac and windows binaries. • Similar feature sets as LSF, including: • Send data direct to Elasticsearch • Send data to Logstash (logstah-forwarder protocol 2.0) • Security as first class citizen • HTTPS for Elasticsearch • TLS for Logstash 22

Slide 23

Slide 23 text

Configuring Filebeat Prospectors 23 ################### Filebeat Configuration Example ######################### filebeat: prospectors: - paths: - /Users/purbon/lib/moves/*.tcx input_type: log multiline: pattern: ^<\?xml negate: true match: after max_lines: 55000 timeout: 5s

Slide 24

Slide 24 text

Configuring Filebeat Outputs 24 ################### Filebeat Configuration Example ######################### output: logstash: hosts: ["localhost:5044"] index: filebeat …… elasticsearch: hosts: ["localhost:9200"] ……

Slide 25

Slide 25 text

Migration to Filebeat • Configuration file changed from JSON to YAML • The registry file has been changed, this file track the data transfer status • Recommendation: Use both in parallel until data sync is complete • Command line options where moved as settings • Mostly all removed, filebeat provides default values in settings. • The new beats input is required in the Logstash side. • Latest version enable LSF to talk to the beats input plugins, reducing migration friction. 25

Slide 26

Slide 26 text

‹#› Plugins Community Maintainers Program

Slide 27

Slide 27 text

‹#› 27 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson Magnus Bäck Philippe Weber Jurgens du Toit Russell Savage

Slide 28

Slide 28 text

28 https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4

Slide 29

Slide 29 text

29 Breaking changes !!!

Slide 30

Slide 30 text

This that might break up for 5.0 • New Event API proposal [#5141] • bin/plugin is now bin/logstash-plugin [#4871] • Require Java 8 as default [#3877] • Fieldref StringInterpolation and Fixnum/Float values [#5114] • Add support for x-pack’s [#4872] • Runner cleanup [#3148, #3860] • Use settings file to complement command line arguments [#4401] 30