Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What has been cooking in Logstash and Kibana for 5.0

What has been cooking in Logstash and Kibana for 5.0

Recap talks on new features for kibana and logstash for upcoming 5.0 releases, this was presented at http://www.meetup.com/Search-UG-Berlin/events/230166132/

Pere Urbón

April 28, 2016

More Decks by Pere Urbón

Other Decks in Technology


  1. We Promised 3 ! ! ! Mobile and mungable filters

    Colorful and composable charts Available and approachable plugins
  2. Filters 4 Edit with the full power of the Elasticsearch

    DSL Pin it then take it with you. Alias for commonly used filters
  3. • Great for complex Elasticsearch queries and administrative operations •

    Syntax highlighting and intelligent typeahead Sense the intelligent console for Elasticsearch
  4. Monitor metrics for your entire Elasticsearch infra 13 • Multi

    cluster performance aggregations • Historical shard and cluster state reporting
  5. Timelion time series expressions for everything 14 • An expressive

    query language for time series • Combines multiple private and public data sources
  6. Creating Kibana Apps Install it npm install -g yo #

    Install yeoman npm install -g generator-kibana-plugin # Install the kibana plugin generator Configure it mkdir my-new-plugin cd my-new-plugin yo kibana-plugin # Generate an app skeleton npm start # Start the plugin development environment Create it cd ../kibana npm start # With Elasticsearch running, start the kibana development environment Visit http://localhost:5601
  7. 16 What to talk about • customisable colours and labels

    • strong integration with security • The dark side is back • space efficiency up 20% => more real estate • export all the things! • plugin all the things! • kibana app’s [ and generator ] • status page Kibana Source: Gray Arial 10pt
  8. What is the current Logstash throughput? 20 Logstash is blocked?,

    ask for a hot threads dump. Worried about OOM? ask for memory consumption. Broken by plugin instance. Per instance and node. to be continued……
  9. Web API resources • Current web api resources: • http://localhost:9600/_node/hot_threads

    • http://localhost:9600/_node/stats/ • http://localhost:9600/_node/stats/events • http://localhost:9600/_stats/jvm • http://localhost:9600/_plugins/ • ….. 21
  10. 22

  11. Step 1: Config Reloading Previously: Any config change made to

    file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 25 File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart Why? How?
  12. The java reincarnation • Logstash is based on JRuby, Ruby

    on top of the JVM • Some core components are being rewritten in plain java. • First step: the event representation. • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. • Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. • Will enable not just ruby plugins, but using any JVM languages • And not only about this, but performance ….. 28
  13. This that might break up for 5.0 • New Event

    API proposal [#5141] • bin/plugin is now bin/logstash-plugin [#4871] • Require Java 8 as default [#3877] • Fieldref StringInterpolation and Fixnum/Float values [#5114] • Add support for x-pack’s [#4872] • Runner cleanup [#3148, #3860] • Use settings file to complement command line arguments [#4401] 34
  14. ‹#› 36 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson

    Magnus Bäck Philippe Weber Jurgens du Toit Russell Savage