Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What has been cooking in Logstash and Kibana for 5.0

What has been cooking in Logstash and Kibana for 5.0

Recap talks on new features for kibana and logstash for upcoming 5.0 releases, this was presented at http://www.meetup.com/Search-UG-Berlin/events/230166132/

Pere Urbón

April 28, 2016
Tweet

More Decks by Pere Urbón

Other Decks in Technology

Transcript

  1. ‹#›
    What is new on Kibana 5.0
    What is cooking on
    Cabana Kibana 5.0

    View Slide

  2. We said it. We meant it.
    2
    A recap of 2015

    View Slide

  3. We Promised
    3
    ! ! !
    Mobile and
    mungable filters
    Colorful and
    composable charts
    Available and
    approachable plugins

    View Slide

  4. Filters
    4
    Edit with the full power of the
    Elasticsearch DSL
    Pin it then take it with you.
    Alias for commonly used filters

    View Slide

  5. Making things more personal
    5
    Customisation

    View Slide

  6. ‹#›
    custom
    Colors

    View Slide

  7. 7
    custom
    Colors

    View Slide

  8. persistent
    Axis Labeling

    View Slide

  9. persistent
    Axis Labeling

    View Slide

  10. return to
    The Dark Side

    View Slide

  11. Making Kibana capable of
    more
    11
    Apps and Hacks
    (plugins)

    View Slide

  12. • Great for complex Elasticsearch queries and administrative operations
    • Syntax highlighting and intelligent typeahead
    Sense the intelligent console for Elasticsearch

    View Slide

  13. Monitor metrics for your entire Elasticsearch infra
    13
    • Multi cluster performance aggregations
    • Historical shard and cluster state reporting

    View Slide

  14. Timelion time series expressions for everything
    14
    • An expressive query language for time series
    • Combines multiple private and public data sources

    View Slide

  15. Creating
    Kibana Apps
    Install it
    npm install -g yo # Install yeoman
    npm install -g generator-kibana-plugin # Install the kibana plugin generator
    Configure it
    mkdir my-new-plugin
    cd my-new-plugin
    yo kibana-plugin # Generate an app skeleton
    npm start # Start the plugin development environment
    Create it
    cd ../kibana
    npm start # With Elasticsearch running, start the kibana development environment
    Visit http://localhost:5601

    View Slide

  16. 16
    What to talk about
    • customisable colours and labels
    • strong integration with security
    • The dark side is back
    • space efficiency up 20% => more real estate
    • export all the things!
    • plugin all the things!
    • kibana app’s [ and generator ]
    • status page
    Kibana
    Source: Gray Arial 10pt

    View Slide

  17. ‹#›
    What is new on Logstash 5.0
    What is cooking on
    Logstash 5.0

    View Slide

  18. 18
    Ingredients

    View Slide

  19. 19
    Metrics

    View Slide

  20. What is the current Logstash
    throughput?
    20
    Logstash is blocked?, ask for a
    hot threads dump.
    Worried about OOM? ask for
    memory consumption.
    Broken by plugin instance.
    Per instance and node.
    to be continued……

    View Slide

  21. Web API resources
    • Current web api resources:
    • http://localhost:9600/_node/hot_threads
    • http://localhost:9600/_node/stats/
    • http://localhost:9600/_node/stats/events
    • http://localhost:9600/_stats/jvm
    • http://localhost:9600/_plugins/
    • …..
    21

    View Slide

  22. 22

    View Slide

  23. 23
    Next: Monitoring UI

    View Slide

  24. 24
    Configuration Management

    View Slide

  25. Step 1: Config Reloading
    Previously: Any config change made
    to file required a process restart
    Feedback loop for development/
    testing slow
    Processing pipeline must be long
    living
    25
    File watched for changes
    or
    SIGHUP triggers reload
    Current Pipeline stopped
    Config Validated
    New Pipeline started - no process
    restart
    Why? How?

    View Slide

  26. 26
    role: frontend-logs
    1 Create a role, upload config

    View Slide

  27. 27
    The Java Event

    View Slide

  28. The java reincarnation
    • Logstash is based on JRuby, Ruby on top of the JVM
    • Some core components are being rewritten in plain java.
    • First step: the event representation.
    • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0.
    • Provide the foundations for upcoming persistence queues, this change will
    be backwards due to the nice introp between JRuby and Java.
    • Will enable not just ruby plugins, but using any JVM languages
    • And not only about this, but performance …..
    28

    View Slide

  29. 29
    logstash-benchmarks.elastic.co

    View Slide

  30. ‹#›
    or happened already
    Other changes that
    happen

    View Slide

  31. 31
    I F O
    Pipeline:
    The Next Generation
    2.1

    View Slide

  32. 32
    I F O
    Pipeline:
    The Next Generation
    2.2

    View Slide

  33. 33
    Breaking
    changes !!!

    View Slide

  34. This that might break up for 5.0
    • New Event API proposal [#5141]
    • bin/plugin is now bin/logstash-plugin [#4871]
    • Require Java 8 as default [#3877]
    • Fieldref StringInterpolation and Fixnum/Float values [#5114]
    • Add support for x-pack’s [#4872]
    • Runner cleanup [#3148, #3860]
    • Use settings file to complement command line arguments [#4401]
    34

    View Slide

  35. ‹#›
    Plugins Community
    Maintainers Program

    View Slide

  36. ‹#›
    36
    Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson
    Magnus Bäck Philippe Weber
    Jurgens du Toit Russell Savage

    View Slide

  37. 37
    https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4

    View Slide

  38. ‹#›
    Pere Urbon-Bayes
    Software Engineer
    Elastic
    QA thanks!

    View Slide