What has been cooking in Logstash and Kibana for 5.0

Recap talks on new features for kibana and logstash for upcoming 5.0 releases, this was presented at http://www.meetup.com/Search-UG-Berlin/events/230166132/

Transcript

1. ‹#› What is new on Kibana 5.0 What is cooking on Cabana Kibana 5.0
2. We said it. We meant it. 2 A recap of 2015
3. We Promised 3 ! ! ! Mobile and mungable filters Colorful and composable charts Available and approachable plugins
4. Filters 4 Edit with the full power of the Elasticsearch DSL Pin it then take it with you. Alias for commonly used filters
5. Making things more personal 5 Customisation
6. ‹#› custom Colors
7. 7 custom Colors
8. persistent Axis Labeling
9. persistent Axis Labeling
10. return to The Dark Side
11. Making Kibana capable of more 11 Apps and Hacks (plugins)
12. • Great for complex Elasticsearch queries and administrative operations • Syntax highlighting and intelligent typeahead Sense the intelligent console for Elasticsearch
13. Monitor metrics for your entire Elasticsearch infra 13 • Multi cluster performance aggregations • Historical shard and cluster state reporting
14. Timelion time series expressions for everything 14 • An expressive query language for time series • Combines multiple private and public data sources
15. Creating Kibana Apps Install it npm install -g yo # Install yeoman npm install -g generator-kibana-plugin # Install the kibana plugin generator Configure it mkdir my-new-plugin cd my-new-plugin yo kibana-plugin # Generate an app skeleton npm start # Start the plugin development environment Create it cd ../kibana npm start # With Elasticsearch running, start the kibana development environment Visit http://localhost:5601
16. 16 What to talk about • customisable colours and labels • strong integration with security • The dark side is back • space efficiency up 20% => more real estate • export all the things! • plugin all the things! • kibana app’s [ and generator ] • status page Kibana Source: Gray Arial 10pt
17. ‹#› What is new on Logstash 5.0 What is cooking on Logstash 5.0
18. 18 Ingredients
19. 19 Metrics
20. What is the current Logstash throughput? 20 Logstash is blocked?, ask for a hot threads dump. Worried about OOM? ask for memory consumption. Broken by plugin instance. Per instance and node. to be continued……
21. Web API resources • Current web api resources: • http://localhost:9600/_node/hot_threads • http://localhost:9600/_node/stats/ • http://localhost:9600/_node/stats/events • http://localhost:9600/_stats/jvm • http://localhost:9600/_plugins/ • ….. 21
23. 23 Next: Monitoring UI
24. 24 Configuration Management
25. Step 1: Config Reloading Previously: Any config change made to file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 25 File watched for changes or SIGHUP triggers reload Current Pipeline stopped Config Validated New Pipeline started - no process restart Why? How?
26. 26 role: frontend-logs 1 Create a role, upload config
27. 27 The Java Event
28. The java reincarnation • Logstash is based on JRuby, Ruby on top of the JVM • Some core components are being rewritten in plain java. • First step: the event representation. • Introduced in 2.3.0, reverted in 2.3.1, on track for 5.0. • Provide the foundations for upcoming persistence queues, this change will be backwards due to the nice introp between JRuby and Java. • Will enable not just ruby plugins, but using any JVM languages • And not only about this, but performance ….. 28
29. 29 logstash-benchmarks.elastic.co
30. ‹#› or happened already Other changes that happen
31. 31 I F O Pipeline: The Next Generation 2.1
32. 32 I F O Pipeline: The Next Generation 2.2
33. 33 Breaking changes !!!
34. This that might break up for 5.0 • New Event API proposal [#5141] • bin/plugin is now bin/logstash-plugin [#4871] • Require Java 8 as default [#3877] • Fieldref StringInterpolation and Fixnum/Float values [#5114] • Add support for x-pack’s [#4872] • Runner cleanup [#3148, #3860] • Use settings file to complement command line arguments [#4401] 34
35. ‹#› Plugins Community Maintainers Program
36. ‹#› 36 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson Magnus Bäck Philippe Weber Jurgens du Toit Russell Savage
37. 37 https://discuss.elastic.co/t/logstash-plugins-community-maintainers/35953/4
38. ‹#› Pere Urbon-Bayes Software Engineer Elastic QA thanks!