$30 off During Our Annual Pro Sale. View Details »

What Can I Do With the ELK Stack?

Tyler L
November 03, 2015

What Can I Do With the ELK Stack?

Tyler L

November 03, 2015

More Decks by Tyler L

Other Decks in Technology


  1. What Can I Do With the ELK Stack? DevOpsDays Charlotte

    2015 Tyler Langlois Infrastructure Engineer, Elastic
  2. This relates to DevOps… how? • Collect disparate data from

    across the organization • Expose it through an accessible interface • Let users create their own value from existing data • Get answers immediately instead of synchronously from dev/ops/admins Self-service unstructured data
  3. ELK in a Nutshell Logstash [collect data] Elasticsearch [store &

    analyze] Kibana [visualize]
  4. Vital Stats • All three are open source • JRuby

    / Java / Javascript / Go (beats) = contribute! • Most recent release (last week) brings LS+ES to 2.0, Kibana to 4.2 • Designed to scale • Very active open source community
  5. Composable, Simple Parts • Single-node ELK stack • deb, rpm

    repos available as well
  6. Implementation tl;dr • Data source logstash input { } {packet,file}beat

    POST :9200 • Document Store Elasticsearch on- premise, Found, Docker, etc. SaaS Options • Visualization Kibana runs in-browser Access controls/basic auth supported Really, anything that can throw JSON at a REST endpoint Scaled appropriately (1 - ??? nodes) Most settings are stored in an Elasticsearch index; simple deployment
  7. Web Server Logs access.log logstash

  8. IRC Activity logstash: input { irc } Elasticsearch

  9. CI/CD Jenkins Redis Elasticsearch yummy ~6 months of data in

  10. Enriching Data logstash: input { twitter } Elasticsearch filter {

    nlp }
  11. Metrics/Numerical Data • Emerging use case • Strong support in

    2.x series • Pipeline aggs ◦ moving averages, percentiles, derivatives
  12. All of this translates to... • Centralized logs & metrics

    • Self-service for: ◦ “Are we serving more 5xx errors than normal?” ◦ “What are response times like?” ◦ “Where is the influx of traffic coming from?” ◦ “How many $project build failures in the last 3 months?” • API for building alerts, dashboards, and tools across data sources … + easy scaling
  13. Need something lightweight? Beats: • Data shippers in single binaries

    • Single-purposes, small footprint
  14. Network Data Packetbeat • Sniffs packets • Understands wire protocols

    • Network tapping means simple deployment
  15. Files Filebeat • Tails files • Ships them elsewhere •

    Lightweight and suited for low- resource environments Metrics Topbeat • Think `top` metrics- ized • Cross-platform metric collection ...more? libbeat • Simple golang library • Create your own • Leverage library to ship along channels to Elasticsearch
  16. Applications

  17. Applications Pair with MySQL, Mongo, etc. to leverage features like

    autocomplete for applications
  18. Use Your Imagination That’s the method I implemented to bring

    a sentiment analyzer into Logstash Create whatever your use case requires!
  19. Inspirational Use Cases SPACE! @ JPL Cancer research @ Yale

  20. Thank you! github.com/tylerjl irc/twitter: leothrix tjll.net Additional Information: • elastic.co

    • Discourse forums • IRC: #elasticsearch, #logstash, #kibana on freenode • github.com/elastic • Corner me anytime this conference with questions