What Can I Do With the ELK Stack?

Transcript

1. What Can I Do With the
   ELK Stack?
   DevOpsDays Charlotte 2015
   Tyler Langlois
   Infrastructure Engineer, Elastic
   

   View Slide

2. This relates to DevOps… how?
   ● Collect disparate data from
   across the organization
   ● Expose it through an
   accessible interface
   ● Let users create their own
   value from existing data
   ● Get answers immediately
   instead of synchronously from
   dev/ops/admins
   Self-service unstructured data
   

   View Slide

3. ELK in a Nutshell
   Logstash
   [collect data]
   Elasticsearch
   [store & analyze]
   Kibana
   [visualize]
   

   View Slide

4. Vital Stats
   ● All three are open source
   ● JRuby / Java / Javascript / Go (beats) = contribute!
   ● Most recent release (last week) brings LS+ES to 2.0,
   Kibana to 4.2
   ● Designed to scale
   ● Very active open source community
   

   View Slide

5. Composable, Simple Parts
   ● Single-node
   ELK stack
   ● deb, rpm repos
   available as
   well
   

   View Slide

6. Implementation tl;dr
   ● Data source
   logstash input { }
   {packet,file}beat
   POST :9200
   ● Document Store
   Elasticsearch on-
   premise, Found, Docker,
   etc.
   SaaS Options
   ● Visualization
   Kibana runs in-browser
   Access controls/basic
   auth supported
   Really, anything that can
   throw JSON at a REST
   endpoint
   Scaled appropriately (1 -
   ??? nodes)
   Most settings are stored
   in an Elasticsearch index;
   simple deployment
   

   View Slide

7. Web Server
   Logs
   access.log
   logstash
   

   View Slide

8. IRC Activity
   logstash:
   input { irc }
   Elasticsearch
   

   View Slide

9. CI/CD
   Jenkins
   Redis
   Elasticsearch
   yummy
   ~6 months of data in 64ms
   

   View Slide

10. Enriching
    Data
    logstash:
    input { twitter }
    Elasticsearch
    filter { nlp }
    

    View Slide

11. Metrics/Numerical
    Data
    ● Emerging use
    case
    ● Strong support in
    2.x series
    ● Pipeline aggs
    ○ moving averages,
    percentiles,
    derivatives
    

    View Slide

12. All of this translates to...
    ● Centralized logs & metrics
    ● Self-service for:
    ○ “Are we serving more 5xx errors than normal?”
    ○ “What are response times like?”
    ○ “Where is the influx of traffic coming from?”
    ○ “How many $project build failures in the last 3 months?”
    ● API for building alerts, dashboards, and tools across data sources
    … + easy scaling
    

    View Slide

13. Need
    something
    lightweight?
    Beats:
    ● Data shippers in single
    binaries
    ● Single-purposes, small
    footprint
    

    View Slide

14. Network Data
    Packetbeat
    ● Sniffs packets
    ● Understands wire
    protocols
    ● Network tapping
    means simple
    deployment
    

    View Slide

15. Files
    Filebeat
    ● Tails files
    ● Ships them
    elsewhere
    ● Lightweight and
    suited for low-
    resource
    environments
    Metrics
    Topbeat
    ● Think `top` metrics-
    ized
    ● Cross-platform
    metric collection
    ...more?
    libbeat
    ● Simple golang library
    ● Create your own
    ● Leverage library to
    ship along channels
    to Elasticsearch
    

    View Slide

16. Applications
    

    View Slide

17. Applications
    Pair with MySQL, Mongo,
    etc. to leverage features
    like autocomplete for
    applications
    

    View Slide

18. Use Your Imagination
    That’s the method I
    implemented to bring a
    sentiment analyzer into
    Logstash
    Create whatever your use
    case requires!
    

    View Slide

19. Inspirational Use Cases
    SPACE! @ JPL Cancer research @ Yale
    

    View Slide

20. Thank you!
    github.com/tylerjl
    irc/twitter: leothrix
    tjll.net
    Additional Information:
    ● elastic.co
    ● Discourse forums
    ● IRC: #elasticsearch, #logstash,
    #kibana on freenode
    ● github.com/elastic
    ● Corner me anytime this
    conference with questions
    

    View Slide