Ingesting logs with style

What has been cooking lately in Logstash, this presentation include a demo that you will find at https://github.com/purbon/elasticstack_sports_demo

Transcript

1. ‹#›
   What has been cooking lately in Logstash
   world.
   Ingesting Logs with style
   

   View Slide

2. $whoami
   Pere Urbon-Bayes (Software Engineer since ever)
   Have always worked with databases, data and analytics.
   GraphDevRoom@FOSDEM
   When not coding I enjoy my time with my wife and kid. I also enjoy
   movies and tv series, and used to like running, too. Basically, I’m
   doing everything to enjoy live.
   2
   

   View Slide

3. 3
   Life demo?
   

   View Slide

4. 4
   Ingredients
   

   View Slide

5. 5
   Metrics
   

   View Slide

6. What is the current Logstash
   throughput?
   6
   Logstash is blocked?, ask for a
   hot threads dump.
   Worried about OOM? ask for
   memory consumption.
   Broken by plugin instance.
   Per instance and node.
   to be continued……
   

   View Slide

7. Web API resources
   • Current web api resources:
   • http://localhost:9600/_node/hot_threads
   • http://localhost:9600/_node/stats/
   • http://localhost:9600/_node/stats/events
   • http://localhost:9600/_stats/jvm
   • http://localhost:9600/_plugins/
   • …..
   7
   

   View Slide

8. View Slide

9. 9
   Next: Monitoring UI
   

   View Slide

10. 10
    Configuration Management
    

    View Slide

11. Step 1: Config Reloading
    Previously: Any config change made
    to file required a process restart
    Feedback loop for development/
    testing slow
    Processing pipeline must be long
    living
    11
    File watched for changes
    or
    SIGHUP triggers reload
    Current Pipeline stopped
    Config Validated
    New Pipeline started - no process
    restart
    Why? How?
    

    View Slide

12. 12
    role: frontend-logs
    1 Create a role, upload config
    

    View Slide

13. 13
    The Java Event
    

    View Slide

14. The java reincarnation
    • Logstash is based on JRuby, Ruby on top of the JVM
    • Some core components are being rewritten in plain java.
    • First step: the event representation.
    • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0.
    • Provide the foundations for upcoming persistence queues, this change will
    be backwards due to the nice introp between JRuby and Java.
    • Will enable not just ruby plugins, but using any JVM languages
    • And not only about this, but performance …..
    14
    

    View Slide

15. 15
    logstash-benchmarks.elastic.co
    

    View Slide

16. 16
    The other changes
    

    View Slide

17. View Slide

18. Native offline plugins management support
    • Build your plugins bundle to be used in offline environments
    • bin/logstash-plugin pack
    • bin/logstash-plugin unpack
    • bin/logstash-plugin install --local
    • Use your own private plugins server
    • Geminabox, Gemirro, Artifactory, Gemfury, …
    • Edit the Gemfile to use the new server.
    Manage plugins in offline environments made easier
    18
    

    View Slide

19. 19
    Pipeline, the next generation
    

    View Slide

20. The pipeline components
    20
    I F O
    I F O
    

    View Slide

21. 21
    Filebeat, the new log harvester gen
    

    View Slide

22. Long life to Logstash Forwarder, welcome Filebeat
    • LSF is deprecated, next generation harvester is Filebeat.
    • Written in Go and configured in YAML.
    • Available as deb, rpm, linux (32 and 64bit), mac and windows binaries.
    • Similar feature sets as LSF, including:
    • Send data direct to Elasticsearch
    • Send data to Logstash (logstah-forwarder protocol 2.0)
    • Security as first class citizen
    • HTTPS for Elasticsearch
    • TLS for Logstash
    22
    

    View Slide

23. Configuring Filebeat
    Prospectors
    23
    ################### Filebeat Configuration Example #########################
    filebeat:
    prospectors:
    -
    paths:
    - /Users/purbon/lib/moves/*.tcx
    input_type: log
    multiline:
    pattern: ^<\?xml
    negate: true
    match: after
    max_lines: 55000
    timeout: 5s
    

    View Slide

24. Configuring Filebeat
    Outputs
    24
    ################### Filebeat Configuration Example #########################
    output:
    logstash:
    hosts: ["localhost:5044"]
    index: filebeat
    ……
    elasticsearch:
    hosts: ["localhost:9200"]
    ……
    

    View Slide

25. Migration to Filebeat
    • Configuration file changed from JSON to YAML
    • The registry file has been changed, this file track the data transfer status
    • Recommendation: Use both in parallel until data sync is complete
    • Command line options where moved as settings
    • Mostly all removed, filebeat provides default values in settings.
    • The new beats input is required in the Logstash side.
    • Latest version enable LSF to talk to the beats input plugins, reducing
    migration friction.
    25
    

    View Slide

26. ‹#›
    Plugins Community
    Maintainers Program
    

    View Slide

27. ‹#›
    27
    Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson
    Magnus Bäck Philippe Weber
    Jurgens du Toit Russell Savage
    

    View Slide

28. 28
    https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4
    

    View Slide

29. 29
    Breaking
    changes !!!
    

    View Slide

30. This that might break up for 5.0
    • New Event API proposal [#5141]
    • bin/plugin is now bin/logstash-plugin [#4871]
    • Require Java 8 as default [#3877]
    • Fieldref StringInterpolation and Fixnum/Float values [#5114]
    • Add support for x-pack’s [#4872]
    • Runner cleanup [#3148, #3860]
    • Use settings file to complement command line arguments [#4401]
    30
    

    View Slide

31. 31
    Demo time
    

    View Slide

32. ‹#›
    Pere Urbon-Bayes
    Software Engineer
    Elastic
    QA thanks!
    

    View Slide