Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ingesting logs with style

Ingesting logs with style

What has been cooking lately in Logstash, this presentation include a demo that you will find at https://github.com/purbon/elasticstack_sports_demo

Pere Urbón

April 28, 2016
Tweet

More Decks by Pere Urbón

Other Decks in Technology

Transcript

  1. ‹#›
    What has been cooking lately in Logstash
    world.
    Ingesting Logs with style

    View Slide

  2. $whoami
    Pere Urbon-Bayes (Software Engineer since ever)
    Have always worked with databases, data and analytics.
    GraphDevRoom@FOSDEM
    When not coding I enjoy my time with my wife and kid. I also enjoy
    movies and tv series, and used to like running, too. Basically, I’m
    doing everything to enjoy live.
    2

    View Slide

  3. 3
    Life demo?

    View Slide

  4. 4
    Ingredients

    View Slide

  5. 5
    Metrics

    View Slide

  6. What is the current Logstash
    throughput?
    6
    Logstash is blocked?, ask for a
    hot threads dump.
    Worried about OOM? ask for
    memory consumption.
    Broken by plugin instance.
    Per instance and node.
    to be continued……

    View Slide

  7. Web API resources
    • Current web api resources:
    • http://localhost:9600/_node/hot_threads
    • http://localhost:9600/_node/stats/
    • http://localhost:9600/_node/stats/events
    • http://localhost:9600/_stats/jvm
    • http://localhost:9600/_plugins/
    • …..
    7

    View Slide

  8. 8

    View Slide

  9. 9
    Next: Monitoring UI

    View Slide

  10. 10
    Configuration Management

    View Slide

  11. Step 1: Config Reloading
    Previously: Any config change made
    to file required a process restart
    Feedback loop for development/
    testing slow
    Processing pipeline must be long
    living
    11
    File watched for changes
    or
    SIGHUP triggers reload
    Current Pipeline stopped
    Config Validated
    New Pipeline started - no process
    restart
    Why? How?

    View Slide

  12. 12
    role: frontend-logs
    1 Create a role, upload config

    View Slide

  13. 13
    The Java Event

    View Slide

  14. The java reincarnation
    • Logstash is based on JRuby, Ruby on top of the JVM
    • Some core components are being rewritten in plain java.
    • First step: the event representation.
    • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0.
    • Provide the foundations for upcoming persistence queues, this change will
    be backwards due to the nice introp between JRuby and Java.
    • Will enable not just ruby plugins, but using any JVM languages
    • And not only about this, but performance …..
    14

    View Slide

  15. 15
    logstash-benchmarks.elastic.co

    View Slide

  16. 16
    The other changes

    View Slide

  17. 17

    View Slide

  18. Native offline plugins management support
    • Build your plugins bundle to be used in offline environments
    • bin/logstash-plugin pack
    • bin/logstash-plugin unpack
    • bin/logstash-plugin install --local
    • Use your own private plugins server
    • Geminabox, Gemirro, Artifactory, Gemfury, …
    • Edit the Gemfile to use the new server.
    Manage plugins in offline environments made easier
    18

    View Slide

  19. 19
    Pipeline, the next generation

    View Slide

  20. The pipeline components
    20
    I F O
    I F O

    View Slide

  21. 21
    Filebeat, the new log harvester gen

    View Slide

  22. Long life to Logstash Forwarder, welcome Filebeat
    • LSF is deprecated, next generation harvester is Filebeat.
    • Written in Go and configured in YAML.
    • Available as deb, rpm, linux (32 and 64bit), mac and windows binaries.
    • Similar feature sets as LSF, including:
    • Send data direct to Elasticsearch
    • Send data to Logstash (logstah-forwarder protocol 2.0)
    • Security as first class citizen
    • HTTPS for Elasticsearch
    • TLS for Logstash
    22

    View Slide

  23. Configuring Filebeat
    Prospectors
    23
    ################### Filebeat Configuration Example #########################
    filebeat:
    prospectors:
    -
    paths:
    - /Users/purbon/lib/moves/*.tcx
    input_type: log
    multiline:
    pattern: ^<\?xml
    negate: true
    match: after
    max_lines: 55000
    timeout: 5s

    View Slide

  24. Configuring Filebeat
    Outputs
    24
    ################### Filebeat Configuration Example #########################
    output:
    logstash:
    hosts: ["localhost:5044"]
    index: filebeat
    ……
    elasticsearch:
    hosts: ["localhost:9200"]
    ……

    View Slide

  25. Migration to Filebeat
    • Configuration file changed from JSON to YAML
    • The registry file has been changed, this file track the data transfer status
    • Recommendation: Use both in parallel until data sync is complete
    • Command line options where moved as settings
    • Mostly all removed, filebeat provides default values in settings.
    • The new beats input is required in the Logstash side.
    • Latest version enable LSF to talk to the beats input plugins, reducing
    migration friction.
    25

    View Slide

  26. ‹#›
    Plugins Community
    Maintainers Program

    View Slide

  27. ‹#›
    27
    Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson
    Magnus Bäck Philippe Weber
    Jurgens du Toit Russell Savage

    View Slide

  28. 28
    https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4

    View Slide

  29. 29
    Breaking
    changes !!!

    View Slide

  30. This that might break up for 5.0
    • New Event API proposal [#5141]
    • bin/plugin is now bin/logstash-plugin [#4871]
    • Require Java 8 as default [#3877]
    • Fieldref StringInterpolation and Fixnum/Float values [#5114]
    • Add support for x-pack’s [#4872]
    • Runner cleanup [#3148, #3860]
    • Use settings file to complement command line arguments [#4401]
    30

    View Slide

  31. 31
    Demo time

    View Slide

  32. ‹#›
    Pere Urbon-Bayes
    Software Engineer
    Elastic
    QA thanks!

    View Slide