Ingesting logs with style

What has been cooking lately in Logstash, this presentation include a demo that you will find at https://github.com/purbon/elasticstack_sports_demo

Transcript

1. ‹#› What has been cooking lately in Logstash world. Ingesting Logs with style
2. $whoami Pere Urbon-Bayes (Software Engineer since ever) Have always worked with databases, data and analytics. GraphDevRoom@FOSDEM When not coding I enjoy my time with my wife and kid. I also enjoy movies and tv series, and used to like running, too. Basically, I’m doing everything to enjoy live. 2
3. 3 Life demo?
4. 4 Ingredients
5. 5 Metrics
6. What is the current Logstash throughput? 6 Logstash is blocked?, ask for a hot threads dump. Worried about OOM? ask for memory consumption. Broken by plugin instance. Per instance and node. to be continued……
7. Web API resources • Current web api resources: • http://localhost:9600/_node/hot_threads • http://localhost:9600/_node/stats/ • http://localhost:9600/_node/stats/events • http://localhost:9600/_stats/jvm • http://localhost:9600/_plugins/ • ….. 7
9. 9 Next: Monitoring UI
10. 10 Configuration Management
11. Step 1: Config Reloading Previously: Any config change made to file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 11 File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart Why? How?
12. 12 role: frontend-logs 1 Create a role, upload config
13. 13 The Java Event
14. The java reincarnation • Logstash is based on JRuby, Ruby on top of the JVM • Some core components are being rewritten in plain java. • First step: the event representation. • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. • Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. • Will enable not just ruby plugins, but using any JVM languages • And not only about this, but performance ….. 14
15. 15 logstash-benchmarks.elastic.co
16. 16 The other changes
18. Native offline plugins management support • Build your plugins bundle to be used in offline environments • bin/logstash-plugin pack • bin/logstash-plugin unpack • bin/logstash-plugin install --local • Use your own private plugins server • Geminabox, Gemirro, Artifactory, Gemfury, … • Edit the Gemfile to use the new server. Manage plugins in offline environments made easier 18
19. 19 Pipeline, the next generation
20. The pipeline components 20 I F O I F O
21. 21 Filebeat, the new log harvester gen
22. Long life to Logstash Forwarder, welcome Filebeat • LSF is deprecated, next generation harvester is Filebeat. • Written in Go and configured in YAML. • Available as deb, rpm, linux (32 and 64bit), mac and windows binaries. • Similar feature sets as LSF, including: • Send data direct to Elasticsearch • Send data to Logstash (logstah-forwarder protocol 2.0) • Security as first class citizen • HTTPS for Elasticsearch • TLS for Logstash 22
23. Configuring Filebeat Prospectors 23 ################### Filebeat Configuration Example ######################### filebeat: prospectors: - paths: - /Users/purbon/lib/moves/*.tcx input_type: log multiline: pattern: ^<\?xml negate: true match: after max_lines: 55000 timeout: 5s
24. Configuring Filebeat Outputs 24 ################### Filebeat Configuration Example ######################### output: logstash: hosts: ["localhost:5044"] index: filebeat …… elasticsearch: hosts: ["localhost:9200"] ……
25. Migration to Filebeat • Configuration file changed from JSON to YAML • The registry file has been changed, this file track the data transfer status • Recommendation: Use both in parallel until data sync is complete • Command line options where moved as settings • Mostly all removed, filebeat provides default values in settings. • The new beats input is required in the Logstash side. • Latest version enable LSF to talk to the beats input plugins, reducing migration friction. 25
26. ‹#› Plugins Community Maintainers Program
27. ‹#› 27 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson Magnus Bäck Philippe Weber Jurgens du Toit Russell Savage
28. 28 https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4
29. 29 Breaking changes !!!
30. This that might break up for 5.0 • New Event API proposal [#5141] • bin/plugin is now bin/logstash-plugin [#4871] • Require Java 8 as default [#3877] • Fieldref StringInterpolation and Fixnum/Float values [#5114] • Add support for x-pack’s [#4872] • Runner cleanup [#3148, #3860] • Use settings file to complement command line arguments [#4401] 30
31. 31 Demo time
32. ‹#› Pere Urbon-Bayes Software Engineer Elastic QA thanks!