What has been cooking in Logstash and Kibana for 5.0

What has been cooking in Logstash and Kibana for 5.0

Recap talks on new features for kibana and logstash for upcoming 5.0 releases, this was presented at http://www.meetup.com/Search-UG-Berlin/events/230166132/

4c253af5a9977910b9326b19199d3023?s=128

Pere Urbón

April 28, 2016
Tweet

Transcript

  1. ‹#› What is new on Kibana 5.0 What is cooking

    on Cabana Kibana 5.0
  2. We said it. We meant it. 2 A recap of

    2015
  3. We Promised 3 ! ! ! Mobile and mungable filters

    Colorful and composable charts Available and approachable plugins
  4. Filters 4 Edit with the full power of the Elasticsearch

    DSL Pin it then take it with you. Alias for commonly used filters
  5. Making things more personal 5 Customisation

  6. ‹#› custom Colors

  7. 7 custom Colors

  8. persistent Axis Labeling

  9. persistent Axis Labeling

  10. return to The Dark Side

  11. Making Kibana capable of more 11 Apps and Hacks (plugins)

  12. • Great for complex Elasticsearch queries and administrative operations •

    Syntax highlighting and intelligent typeahead Sense the intelligent console for Elasticsearch
  13. Monitor metrics for your entire Elasticsearch infra 13 • Multi

    cluster performance aggregations • Historical shard and cluster state reporting
  14. Timelion time series expressions for everything 14 • An expressive

    query language for time series • Combines multiple private and public data sources
  15. Creating Kibana Apps Install it npm install -g yo #

    Install yeoman npm install -g generator-kibana-plugin # Install the kibana plugin generator Configure it mkdir my-new-plugin cd my-new-plugin yo kibana-plugin # Generate an app skeleton npm start # Start the plugin development environment Create it cd ../kibana npm start # With Elasticsearch running, start the kibana development environment Visit http://localhost:5601
  16. 16 What to talk about • customisable colours and labels

    • strong integration with security • The dark side is back • space efficiency up 20% => more real estate • export all the things! • plugin all the things! • kibana app’s [ and generator ] • status page Kibana Source: Gray Arial 10pt
  17. ‹#› What is new on Logstash 5.0 What is cooking

    on Logstash 5.0
  18. 18 Ingredients

  19. 19 Metrics

  20. What is the current Logstash throughput? 20 Logstash is blocked?,

    ask for a hot threads dump. Worried about OOM? ask for memory consumption. Broken by plugin instance. Per instance and node. to be continued……
  21. Web API resources • Current web api resources: • http://localhost:9600/_node/hot_threads

    • http://localhost:9600/_node/stats/ • http://localhost:9600/_node/stats/events • http://localhost:9600/_stats/jvm • http://localhost:9600/_plugins/ • ….. 21
  22. 22

  23. 23 Next: Monitoring UI

  24. 24 Configuration Management

  25. Step 1: Config Reloading Previously: Any config change made to

    file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 25 File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart Why? How?
  26. 26 role: frontend-logs 1 Create a role, upload config

  27. 27 The Java Event

  28. The java reincarnation • Logstash is based on JRuby, Ruby

    on top of the JVM • Some core components are being rewritten in plain java. • First step: the event representation. • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. • Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. • Will enable not just ruby plugins, but using any JVM languages • And not only about this, but performance ….. 28
  29. 29 logstash-benchmarks.elastic.co

  30. ‹#› or happened already Other changes that happen

  31. 31 I F O Pipeline: The Next Generation 2.1

  32. 32 I F O Pipeline: The Next Generation 2.2

  33. 33 Breaking changes !!!

  34. This that might break up for 5.0 • New Event

    API proposal [#5141] • bin/plugin is now bin/logstash-plugin [#4871] • Require Java 8 as default [#3877] • Fieldref StringInterpolation and Fixnum/Float values [#5114] • Add support for x-pack’s [#4872] • Runner cleanup [#3148, #3860] • Use settings file to complement command line arguments [#4401] 34
  35. ‹#› Plugins Community Maintainers Program

  36. ‹#› 36 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson

    Magnus Bäck Philippe Weber Jurgens du Toit Russell Savage
  37. 37 https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4

  38. ‹#› Pere Urbon-Bayes Software Engineer Elastic QA thanks!