What Can I Do With the ELK Stack?

1. What Can I Do With the ELK Stack? DevOpsDays Charlotte 2015 Tyler Langlois Infrastructure Engineer, Elastic
2. This relates to DevOps… how? • Collect disparate data from across the organization • Expose it through an accessible interface • Let users create their own value from existing data • Get answers immediately instead of synchronously from dev/ops/admins Self-service unstructured data
3. ELK in a Nutshell Logstash [collect data] Elasticsearch [store & analyze] Kibana [visualize]
4. Vital Stats • All three are open source • JRuby / Java / Javascript / Go (beats) = contribute! • Most recent release (last week) brings LS+ES to 2.0, Kibana to 4.2 • Designed to scale • Very active open source community
5. Composable, Simple Parts • Single-node ELK stack • deb, rpm repos available as well
6. Implementation tl;dr • Data source logstash input { } {packet,file}beat POST :9200 • Document Store Elasticsearch on- premise, Found, Docker, etc. SaaS Options • Visualization Kibana runs in-browser Access controls/basic auth supported Really, anything that can throw JSON at a REST endpoint Scaled appropriately (1 - ??? nodes) Most settings are stored in an Elasticsearch index; simple deployment
7. Web Server Logs access.log logstash
8. IRC Activity logstash: input { irc } Elasticsearch
9. CI/CD Jenkins Redis Elasticsearch yummy ~6 months of data in 64ms
10. Enriching Data logstash: input { twitter } Elasticsearch filter { nlp }
11. Metrics/Numerical Data • Emerging use case • Strong support in 2.x series • Pipeline aggs ◦ moving averages, percentiles, derivatives
12. All of this translates to... • Centralized logs & metrics • Self-service for: ◦ “Are we serving more 5xx errors than normal?” ◦ “What are response times like?” ◦ “Where is the influx of traffic coming from?” ◦ “How many $project build failures in the last 3 months?” • API for building alerts, dashboards, and tools across data sources … + easy scaling
13. Need something lightweight? Beats: • Data shippers in single binaries • Single-purposes, small footprint
14. Network Data Packetbeat • Sniffs packets • Understands wire protocols • Network tapping means simple deployment
15. Files Filebeat • Tails files • Ships them elsewhere • Lightweight and suited for low- resource environments Metrics Topbeat • Think `top` metrics- ized • Cross-platform metric collection ...more? libbeat • Simple golang library • Create your own • Leverage library to ship along channels to Elasticsearch
16. Applications
17. Applications Pair with MySQL, Mongo, etc. to leverage features like autocomplete for applications
18. Use Your Imagination That’s the method I implemented to bring a sentiment analyzer into Logstash Create whatever your use case requires!
19. Inspirational Use Cases SPACE! @ JPL Cancer research @ Yale
20. Thank you! github.com/tylerjl irc/twitter: leothrix tjll.net Additional Information: • elastic.co • Discourse forums • IRC: #elasticsearch, #logstash, #kibana on freenode • github.com/elastic • Corner me anytime this conference with questions