Visibility for web developers

Visibility for web developers

Bd9b4485bcf325b989b5721bd0ef4d91?s=128

Bruno Renié

May 13, 2014
Tweet

Transcript

  1. Visibility for Web Developers Bruno Renié Django Island 2014

  2. Working with ops & support staff Large infrastructure, lots of

    moving parts whoami: lead dev @ exoscale
  3. People need to know what's going on, anytime Not everyone

    is a developer SSH, grep and tail not acceptable
  4. Goal: visibility on infrastructure & apps

  5. Errors Events Metrics

  6. Errors

  7. Easy: use sentry

  8. Great tool for developers, but… People also need to know

    about things that are not broken
  9. Events

  10. Event == log call Timestamped information about something worth knowing

  11. Easy to emit logs Searching on remote machines, not so

    much
  12. Centralized logging Agent forwards logs to an aggregator Aggregator stores

    logs and provides a search interface
  13. Aggregator Hosts logstash+elasticsearch kibana log shippers

  14. Collecting logs rsyslog / syslog-ng logstash-forwarder (lumberjack) logstash

  15. In Python logging.handlers.SyslogHandler

  16. Centralizing isn't enough

  17. A common way to log things import logging logger =

    logging.getLogger(__name__) logger.info("user {} logged in".format(user.email))
  18. Not much information No structure

  19. Structured logging Record = set of key-value pairs event="user logged

    in" user="test@example.com" ip="1.2.3.4"
  20. In Python: pip install structlog Hynek Schlawack logger = structlog.get_logger()

    logger.info("user logged in", user="test@example.com" ip="1.2.3.4") {"event": "user.login", "user": "test@example.com", "host": "1.2.3.4"}
  21. Contextual loggers log = logger.bind(user=request.user.email) log.info("payment", amount=120, currency="CHF") {"event": "payment",

    "user": "test@example.com", "amount": 120, "currency": "CHF"}
  22. structlog JSON elasticsearch Rich data for building dashboards

  23. Demo

  24. Marketing: "Who signed up today?" Finance: "How much income in

    the past week?" Support: "What did this user do in the past 24 hours?"
  25. Events Centralize logs Add structure Profit logstash, elasticsearch structlog kibana

  26. Metrics

  27. Time series data Continuous, regular intervals Big player: graphite ecosystem

    carbon whisper graphite
  28. <identifier> <value> <timestamp> db1.load.midterm 4.6 1399842778 Measure at regular intervals,

    send using the carbon line protocol echo " … " | nc host port
  29. Traditional metrics stack agents collectd, apps, scripts carbon whisper graphite

    ingests metrics stores metrics renders graphs
  30. Counting and timing: statsd In-memory buffer Aggregates and flushes to

    carbon at regular intervals
  31. In Python pystatsd django-statsd-mozilla from stastd import StastClient statsd =

    StatsClient() @statsd.timer("something") def expensive_computation(): # …
  32. Graphing Graphite: rendering API (json / PNG / etc) Countless

    dashboard apps Grafana Graph-explorer etc
  33. Demo

  34. What about alerts? Riemann: metrics hub/proxy App server overloaded, full

    disk, etc Analyzes metrics computes aggregates notifies
  35. Demo

  36. we are hiring! Questions! @brutasse Image credits: http://en.wikipedia.org/wiki/Explosion

  37. Resources Events Metrics structlog.org rsyslog logstash elasticsearch kibana carbon /

    whisper graphite-api grafana collectd statsd riemann.io bernhard
  38. Hosted services (not exhaustive) Logging Metrics Loggly Papertrail Hosted Graphite

    Librato JSON / structured logging support?