Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What Can I Do With the ELK Stack?

Tyler L
November 03, 2015

What Can I Do With the ELK Stack?

Tyler L

November 03, 2015
Tweet

More Decks by Tyler L

Other Decks in Technology

Transcript

  1. What Can I Do With the
    ELK Stack?
    DevOpsDays Charlotte 2015
    Tyler Langlois
    Infrastructure Engineer, Elastic

    View Slide

  2. This relates to DevOps… how?
    ● Collect disparate data from
    across the organization
    ● Expose it through an
    accessible interface
    ● Let users create their own
    value from existing data
    ● Get answers immediately
    instead of synchronously from
    dev/ops/admins
    Self-service unstructured data

    View Slide

  3. ELK in a Nutshell
    Logstash
    [collect data]
    Elasticsearch
    [store & analyze]
    Kibana
    [visualize]

    View Slide

  4. Vital Stats
    ● All three are open source
    ● JRuby / Java / Javascript / Go (beats) = contribute!
    ● Most recent release (last week) brings LS+ES to 2.0,
    Kibana to 4.2
    ● Designed to scale
    ● Very active open source community

    View Slide

  5. Composable, Simple Parts
    ● Single-node
    ELK stack
    ● deb, rpm repos
    available as
    well

    View Slide

  6. Implementation tl;dr
    ● Data source
    logstash input { }
    {packet,file}beat
    POST :9200
    ● Document Store
    Elasticsearch on-
    premise, Found, Docker,
    etc.
    SaaS Options
    ● Visualization
    Kibana runs in-browser
    Access controls/basic
    auth supported
    Really, anything that can
    throw JSON at a REST
    endpoint
    Scaled appropriately (1 -
    ??? nodes)
    Most settings are stored
    in an Elasticsearch index;
    simple deployment

    View Slide

  7. Web Server
    Logs
    access.log
    logstash

    View Slide

  8. IRC Activity
    logstash:
    input { irc }
    Elasticsearch

    View Slide

  9. CI/CD
    Jenkins
    Redis
    Elasticsearch
    yummy
    ~6 months of data in 64ms

    View Slide

  10. Enriching
    Data
    logstash:
    input { twitter }
    Elasticsearch
    filter { nlp }

    View Slide

  11. Metrics/Numerical
    Data
    ● Emerging use
    case
    ● Strong support in
    2.x series
    ● Pipeline aggs
    ○ moving averages,
    percentiles,
    derivatives

    View Slide

  12. All of this translates to...
    ● Centralized logs & metrics
    ● Self-service for:
    ○ “Are we serving more 5xx errors than normal?”
    ○ “What are response times like?”
    ○ “Where is the influx of traffic coming from?”
    ○ “How many $project build failures in the last 3 months?”
    ● API for building alerts, dashboards, and tools across data sources
    … + easy scaling

    View Slide

  13. Need
    something
    lightweight?
    Beats:
    ● Data shippers in single
    binaries
    ● Single-purposes, small
    footprint

    View Slide

  14. Network Data
    Packetbeat
    ● Sniffs packets
    ● Understands wire
    protocols
    ● Network tapping
    means simple
    deployment

    View Slide

  15. Files
    Filebeat
    ● Tails files
    ● Ships them
    elsewhere
    ● Lightweight and
    suited for low-
    resource
    environments
    Metrics
    Topbeat
    ● Think `top` metrics-
    ized
    ● Cross-platform
    metric collection
    ...more?
    libbeat
    ● Simple golang library
    ● Create your own
    ● Leverage library to
    ship along channels
    to Elasticsearch

    View Slide

  16. Applications

    View Slide

  17. Applications
    Pair with MySQL, Mongo,
    etc. to leverage features
    like autocomplete for
    applications

    View Slide

  18. Use Your Imagination
    That’s the method I
    implemented to bring a
    sentiment analyzer into
    Logstash
    Create whatever your use
    case requires!

    View Slide

  19. Inspirational Use Cases
    SPACE! @ JPL Cancer research @ Yale

    View Slide

  20. Thank you!
    github.com/tylerjl
    irc/twitter: leothrix
    tjll.net
    Additional Information:
    ● elastic.co
    ● Discourse forums
    ● IRC: #elasticsearch, #logstash,
    #kibana on freenode
    ● github.com/elastic
    ● Corner me anytime this
    conference with questions

    View Slide