Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Visibility for web developers

Visibility for web developers

Bruno Renié

May 13, 2014
Tweet

More Decks by Bruno Renié

Other Decks in Technology

Transcript

  1. Visibility for
    Web Developers
    Bruno Renié
    Django Island 2014

    View Slide

  2. Working with ops & support staff
    Large infrastructure, lots of moving parts
    whoami: lead dev @ exoscale

    View Slide

  3. People need to know what's
    going on, anytime
    Not everyone is a developer
    SSH, grep and tail not acceptable

    View Slide

  4. Goal:
    visibility on infrastructure & apps

    View Slide

  5. Errors
    Events
    Metrics

    View Slide

  6. Errors

    View Slide

  7. Easy: use sentry

    View Slide

  8. Great tool for developers, but…
    People also need to know about
    things that are not broken

    View Slide

  9. Events

    View Slide

  10. Event == log call
    Timestamped information about
    something worth knowing

    View Slide

  11. Easy to emit logs
    Searching on remote machines,
    not so much

    View Slide

  12. Centralized logging
    Agent forwards logs
    to an aggregator
    Aggregator stores logs and
    provides a search interface

    View Slide

  13. Aggregator
    Hosts
    logstash+elasticsearch
    kibana
    log shippers

    View Slide

  14. Collecting logs
    rsyslog / syslog-ng
    logstash-forwarder (lumberjack)
    logstash

    View Slide

  15. In Python
    logging.handlers.SyslogHandler

    View Slide

  16. Centralizing isn't enough

    View Slide

  17. A common way to log things
    import logging
    logger = logging.getLogger(__name__)
    logger.info("user {} logged in".format(user.email))

    View Slide

  18. Not much information
    No structure

    View Slide

  19. Structured logging
    Record = set of key-value pairs
    event="user logged in"
    user="[email protected]"
    ip="1.2.3.4"

    View Slide

  20. In Python:
    pip install structlog
    Hynek Schlawack
    logger = structlog.get_logger()
    logger.info("user logged in",
    user="[email protected]"
    ip="1.2.3.4")
    {"event": "user.login",
    "user": "[email protected]",
    "host": "1.2.3.4"}

    View Slide

  21. Contextual loggers
    log = logger.bind(user=request.user.email)
    log.info("payment", amount=120,
    currency="CHF")
    {"event": "payment",
    "user": "[email protected]",
    "amount": 120,
    "currency": "CHF"}

    View Slide

  22. structlog
    JSON
    elasticsearch
    Rich data for building dashboards

    View Slide

  23. Demo

    View Slide

  24. Marketing: "Who signed up today?"
    Finance: "How much income
    in the past week?"
    Support: "What did this user do
    in the past 24 hours?"

    View Slide

  25. Events
    Centralize logs
    Add structure
    Profit
    logstash, elasticsearch
    structlog
    kibana

    View Slide

  26. Metrics

    View Slide

  27. Time series data
    Continuous, regular intervals
    Big player: graphite ecosystem
    carbon
    whisper
    graphite

    View Slide


  28. db1.load.midterm 4.6 1399842778
    Measure at regular intervals,
    send using the carbon line protocol
    echo " … " | nc host port

    View Slide

  29. Traditional metrics stack
    agents
    collectd, apps, scripts
    carbon
    whisper
    graphite
    ingests metrics
    stores metrics
    renders graphs

    View Slide

  30. Counting and timing: statsd
    In-memory buffer
    Aggregates and flushes
    to carbon
    at regular intervals

    View Slide

  31. In Python
    pystatsd
    django-statsd-mozilla
    from stastd import StastClient
    statsd = StatsClient()
    @statsd.timer("something")
    def expensive_computation():
    # …

    View Slide

  32. Graphing
    Graphite: rendering API (json / PNG / etc)
    Countless dashboard apps
    Grafana
    Graph-explorer
    etc

    View Slide

  33. Demo

    View Slide

  34. What about alerts?
    Riemann: metrics hub/proxy
    App server overloaded, full disk, etc
    Analyzes metrics
    computes aggregates
    notifies

    View Slide

  35. Demo

    View Slide

  36. we are hiring!
    Questions!
    @brutasse
    Image credits: http://en.wikipedia.org/wiki/Explosion

    View Slide

  37. Resources
    Events Metrics
    structlog.org
    rsyslog
    logstash
    elasticsearch
    kibana
    carbon / whisper
    graphite-api
    grafana
    collectd
    statsd
    riemann.io
    bernhard

    View Slide

  38. Hosted services
    (not exhaustive)
    Logging Metrics
    Loggly
    Papertrail
    Hosted Graphite
    Librato
    JSON / structured
    logging support?

    View Slide