What Can I Do With the ELK Stack? DevOpsDays Charlotte 2015 Tyler Langlois Infrastructure Engineer, Elastic

This relates to DevOps… how? ● Collect disparate data from across the organization ● Expose it through an accessible interface ● Let users create their own value from existing data ● Get answers immediately instead of synchronously from dev/ops/admins Self-service unstructured data

ELK in a Nutshell Logstash [collect data] Elasticsearch [store & analyze] Kibana [visualize]

Vital Stats ● All three are open source ● JRuby / Java / Javascript / Go (beats) = contribute! ● Most recent release (last week) brings LS+ES to 2.0, Kibana to 4.2 ● Designed to scale ● Very active open source community

Composable, Simple Parts ● Single-node ELK stack ● deb, rpm repos available as well

Implementation tl;dr ● Data source logstash input { } {packet,file}beat POST :9200 ● Document Store Elasticsearch on- premise, Found, Docker, etc. SaaS Options ● Visualization Kibana runs in-browser Access controls/basic auth supported Really, anything that can throw JSON at a REST endpoint Scaled appropriately (1 - ??? nodes) Most settings are stored in an Elasticsearch index; simple deployment

Web Server Logs access.log logstash

IRC Activity logstash: input { irc } Elasticsearch

CI/CD Jenkins Redis Elasticsearch yummy ~6 months of data in 64ms

Enriching Data logstash: input { twitter } Elasticsearch filter { nlp }

Metrics/Numerical Data ● Emerging use case ● Strong support in 2.x series ● Pipeline aggs ○ moving averages, percentiles, derivatives

All of this translates to... ● Centralized logs & metrics ● Self-service for: ○ “Are we serving more 5xx errors than normal?” ○ “What are response times like?” ○ “Where is the influx of traffic coming from?” ○ “How many $project build failures in the last 3 months?” ● API for building alerts, dashboards, and tools across data sources … + easy scaling

Need something lightweight? Beats: ● Data shippers in single binaries ● Single-purposes, small footprint

Network Data Packetbeat ● Sniffs packets ● Understands wire protocols ● Network tapping means simple deployment

Files Filebeat ● Tails files ● Ships them elsewhere ● Lightweight and suited for low- resource environments Metrics Topbeat ● Think `top` metrics- ized ● Cross-platform metric collection ...more? libbeat ● Simple golang library ● Create your own ● Leverage library to ship along channels to Elasticsearch

Applications

Applications Pair with MySQL, Mongo, etc. to leverage features like autocomplete for applications

Use Your Imagination That’s the method I implemented to bring a sentiment analyzer into Logstash Create whatever your use case requires!

Inspirational Use Cases SPACE! @ JPL Cancer research @ Yale

Thank you! github.com/tylerjl irc/twitter: leothrix tjll.net Additional Information: ● elastic.co ● Discourse forums ● IRC: #elasticsearch, #logstash, #kibana on freenode ● github.com/elastic ● Corner me anytime this conference with questions