$30 off During Our Annual Pro Sale. View Details »

Log all the things!

Log all the things!

Honza Král's talk from Europython 2016:

Centralized logging (and the Elastic stack) is proving itself to be a very useful tool in managing a production infrastructure. When combined with other data sources (application logging, business data, …) it can provide even more insight.

This talk is an introduction into the area with some overview of the motivation, tools and techniques that can prove useful. We will show how the open source ELK (Elasticsearch Logstash and Kibana) stack can be used to implement this.

It is geared towards people familiar with the DevOps concept that are looking to improve their lives by introducing smarter tools.

Elasticsearch Inc

July 22, 2016
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Log all the things!
    Honza Král
    @honzakral

    View Slide

  2. Logs?

    View Slide

  3. Log lines
    Twitter feed
    Invoices
    Metrics
    Events!

    View Slide

  4. Why?

    View Slide

  5. What happened
    last Tuesday?

    View Slide

  6. Multiple machines
    Multiple logs
    Analysis/Discovery
    Time period
    Grep?

    View Slide

  7. Time? Time?! Time!
    apache
    unix timestamp
    log4j
    postfix.log
    ISO 8601
    [23/Jan/2014:17:11:55 +0000]
    1390994740
    2009-01-01T12:00:00+01:00
    [2014-01-29 12:28:25,470]
    Feb 3 20:37:35

    View Slide

  8. Web Server logs VS Load Balancer
    see immediately that caching is off
    static files leaking to gunicorn
    Web Server VS Database
    500s VS Deploys
    new version has a bug
    Traffic VS Ad Campaigns
    Correlate events

    View Slide

  9. Central storage
    Even for data from different systems
    Enriched data
    IP -> location, hostname
    URL -> author, product, category
    Search
    user:honza status:404
    Analysis
    Visualisations for easy pattern discovery
    Ideal state

    View Slide

  10. Centralised
    Logging

    View Slide

  11. Steps
    Collect data
    Parse data
    Enrich data
    Store data
    Search and aggregate
    Visualize data

    View Slide

  12. Elastic Stack

    View Slide

  13. Steps in Elastic Stack
    Collect data
    Parse data
    Enrich data
    Store data
    Search and aggregate
    Visualize data

    View Slide

  14. Steps in Elastic Stack
    Collect data
    Parse data
    Enrich data
    Store data
    Search and aggregate
    Visualize data

    View Slide

  15. View Slide

  16. metricbeat:
    modules:
    - module: redis
    metricsets: ["info"]
    hosts: ["host1"]
    period: 1s
    enabled: true
    - module: apache
    metricsets: ["info"]
    hosts: ["host1"]
    period: 30s
    enabled: true
    filebeat:
    prospectors:
    - paths:
    - "logs/access.log"
    document_type: access
    multiline:
    pattern: ^#
    negate: true
    match: after
    protocols:
    http:
    ports: [80, 8000]
    mysql:
    ports: [3306]
    redis:
    ports: [6379]
    pgsql:
    ports: [5432]
    thrift:
    ports: [9090]
    output:
    logstash:
    hosts: ["localhost:5044"]

    View Slide

  17. View Slide

  18. Inputs
    Monitoring collectd, graphite, ganglia, snmptrap, zenoss
    Datastores elasticsearch, redis, sqlite, s3
    Queues kafka, rabbitmq, zeromq
    Logging beats, eventlog, gelf, log4j, relp, syslog, varnish log
    Platforms
    drupal_dblog, gemfire, heroku, sqs, s3, twitter
    Local
    exec, generator, file, stdin, pipe, unix
    Protocol imap, irc, stomp, tcp, udp, websocket, wmi, xmpp

    View Slide

  19. Filters
    aggregate alter anonymize collate csv
    cidr clone cipher checksum date dns
    drop elasticsearch extractnumbers
    environment elapsed fingerprint geoip
    grok i18n json json_encode kv mutate
    metrics multiline metaevent prune punct
    ruby range syslog_pri sleep split throttle
    translate uuid urldecode useragent xml
    zeromq ...

    View Slide

  20. Outputs
    Store elasticsearch, gemfire, mongodb, redis, riak, rabbitmq, solr
    Monitoring
    ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix
    Notification email, hipchat, irc, pagerduty, sns
    Protocol
    gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket,
    xmpp
    External service
    google big query, google cloud storage, jira, loggly, riemann, s3,
    sqs, syslog, datadog
    External monitoring boundary, circonus, cloudwatch, librato
    Local csv, dots, exec, file, pipe, stdout, null

    View Slide

  21. View Slide

  22. Open Source


    Document-based


    Based on Lucene

    JSON over HTTP
    Distributed Search Engine

    View Slide

  23. Cluster
    Collection of Nodes
    Index
    Collection of Shards
    Shard
    Unit of scale
    Distributed across cluster
    Primary and replica
    Data Management
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    3 4
    1
    3
    products

    View Slide

  24. Time based data flow
    Current
    replicas to speed up search
    on stronger boxes
    Week old
    snapshot
    keep only 1 replica
    Month old move to weaker boxes
    2 months close the indices
    3 months delete

    View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. Architecture
    Enrich Visualize
    Collect Store

    View Slide

  30. Logging and
    Python

    View Slide

  31. Track metrics
    execution time
    query time
    # of queries
    Include metadata
    user_id
    content
    Log as JSON
    Enhance your logs

    View Slide

  32. Add structured info
    Track info through services
    Log to file
    Add filebeat to read the file
    Structlog

    View Slide

  33. Thanks!
    Honza Král
    @honzakral

    View Slide