Kibana Workshop

Slides from the Kibana workshop carried out @CMP Group

Transcript

1. Kibana Workshop
2. Hello! I am Marc Lopez I am currently working as a DevOps in CMP Group You can find me on Twitter at @marclop__
3. Today’s Agenda ◇ Elasticsearch (Brief intro) ◇ Logstash ◇ Kibana ◇ Interesting figures
4. Elasticsearch What the hell is this thing? 1
5. “ You Know, for Search
6. NoSQL Database ◇ Built on Apache Lucene ◇ (full text) Search engine ◇ Distributed & HA ◇ Built-in analytics engine ◇ Near Real-time Data ◇ RESTful API
7. Elasticsearch role ◇ Persistent Datastore ◇ Analytics engine performed on the “DB” side ◇ Scale-out capabilities Modern up-to date piece of software that gives us the ability to start becoming more data-centric and analytic about our applications
8. $ curl -s -XPUT 'http://elasticsearch:9200/twitter/' -d ' index : number_of_shards : 1 number_of_replicas : 0' {"acknowledged":true} RESTful API - Create new Index
9. $ curl -s -XPUT 'http://elasticsearch:9200/twitter/' -d ' { “index” : { “number_of_shards” : 1, “number_of_replicas” : 0 } }' {"acknowledged":true} RESTful API - Also in JSON
10. $ curl -s -XDELETE 'http://elasticsearch:9200/twitter/' {"acknowledged":true} {"acknowledged":true} RESTful API - Delete Index
11. Logstash Source data import 2
12. Logstash role ◇ Data processing / Enrichment ◇ Filtering / Rinsing Central piece for data ingestion into Elasticsearch
13. Data ingestion process Log / App Logstash Elasticsearch
14. Kibana Web UI for data discovery and analytics 3
15. Kibana role ◇ Data discovery ◇ Graphs ◇ Dashboards Central Web UI to discover and graph your available date in an Interactive way, no need to dig into Query or Filter APIs
16. Kibana Components ◇ Discover ◇ Visualize ◇ Dashboards
17. “ Discover
18. Data discovery
19. Key points ◇ Raw data visualization ◇ Data filtering (Uses Query filters +Performance) ◇ Full text search (-Performance) ◇ Save searches for future use (Ej. Visualizations) TIP: Use query filters when possible instead of full text searches
20. “ Visualize
21. Visualize
22. Key points ◇ Build nice Graphs / Pies / Charts with known data ◇ Re-use saved searches ◇ Save Visualizations for later use (Ej. Dashboards) ◇ Always use fields not analyzed! (----Performance) TIP: Be careful when saving searches to not filter out data that would otherwise be helpful
23. “ Dashboards
24. Dashboards
25. Key points ◇ Build awesome dashboards using saved Visualizations TIP: Be careful of the time-range you’ve specified in Kibana as it might be very heavy on the Elasticsearch indices
26. “ Demo
27. Interesting figures 4
28. Current Kibana usage 2-4 Users 10-15 Users ??
29. 1,170,770,490 That’s a lot of logs! 11 Not that many Elasticsearch nodes 1.47 TB And a lot of data (replicas taken into account)
30. Current data growth over time August September October November (09/11/15) Logs 15,22GB 120,06GB 170,81GB 45,24GB
31. Thanks! Any questions? You can find me at: ◇ @marclop__ ◇ marc.lopez@teamcmp.com
32. Repo URL https://github.com/marclop/kibana-workshop