Visibility for web developers

Transcript

1. Visibility for
   Web Developers
   Bruno Renié
   Django Island 2014
   

   View Slide

2. Working with ops & support staff
   Large infrastructure, lots of moving parts
   whoami: lead dev @ exoscale
   

   View Slide

3. People need to know what's
   going on, anytime
   Not everyone is a developer
   SSH, grep and tail not acceptable
   

   View Slide

4. Goal:
   visibility on infrastructure & apps
   

   View Slide

5. Errors
   Events
   Metrics
   

   View Slide

6. Errors
   

   View Slide

7. Easy: use sentry
   

   View Slide

8. Great tool for developers, but…
   People also need to know about
   things that are not broken
   

   View Slide

9. Events
   

   View Slide

10. Event == log call
    Timestamped information about
    something worth knowing
    

    View Slide

11. Easy to emit logs
    Searching on remote machines,
    not so much
    

    View Slide

12. Centralized logging
    Agent forwards logs
    to an aggregator
    Aggregator stores logs and
    provides a search interface
    

    View Slide

13. Aggregator
    Hosts
    logstash+elasticsearch
    kibana
    log shippers
    

    View Slide

14. Collecting logs
    rsyslog / syslog-ng
    logstash-forwarder (lumberjack)
    logstash
    

    View Slide

15. In Python
    logging.handlers.SyslogHandler
    

    View Slide

16. Centralizing isn't enough
    

    View Slide

17. A common way to log things
    import logging
    logger = logging.getLogger(__name__)
    logger.info("user {} logged in".format(user.email))
    

    View Slide

18. Not much information
    No structure
    

    View Slide

19. Structured logging
    Record = set of key-value pairs
    event="user logged in"
    user="test@example.com"
    ip="1.2.3.4"
    

    View Slide

20. In Python:
    pip install structlog
    Hynek Schlawack
    logger = structlog.get_logger()
    logger.info("user logged in",
    user="test@example.com"
    ip="1.2.3.4")
    {"event": "user.login",
    "user": "test@example.com",
    "host": "1.2.3.4"}
    

    View Slide

21. Contextual loggers
    log = logger.bind(user=request.user.email)
    log.info("payment", amount=120,
    currency="CHF")
    {"event": "payment",
    "user": "test@example.com",
    "amount": 120,
    "currency": "CHF"}
    

    View Slide

22. structlog
    JSON
    elasticsearch
    Rich data for building dashboards
    

    View Slide

23. Demo
    

    View Slide

24. Marketing: "Who signed up today?"
    Finance: "How much income
    in the past week?"
    Support: "What did this user do
    in the past 24 hours?"
    

    View Slide

25. Events
    Centralize logs
    Add structure
    Profit
    logstash, elasticsearch
    structlog
    kibana
    

    View Slide

26. Metrics
    

    View Slide

27. Time series data
    Continuous, regular intervals
    Big player: graphite ecosystem
    carbon
    whisper
    graphite
    

    View Slide

28. 
    db1.load.midterm 4.6 1399842778
    Measure at regular intervals,
    send using the carbon line protocol
    echo " … " | nc host port
    

    View Slide

29. Traditional metrics stack
    agents
    collectd, apps, scripts
    carbon
    whisper
    graphite
    ingests metrics
    stores metrics
    renders graphs
    

    View Slide

30. Counting and timing: statsd
    In-memory buffer
    Aggregates and flushes
    to carbon
    at regular intervals
    

    View Slide

31. In Python
    pystatsd
    django-statsd-mozilla
    from stastd import StastClient
    statsd = StatsClient()
    @statsd.timer("something")
    def expensive_computation():
    # …
    

    View Slide

32. Graphing
    Graphite: rendering API (json / PNG / etc)
    Countless dashboard apps
    Grafana
    Graph-explorer
    etc
    

    View Slide

33. Demo
    

    View Slide

34. What about alerts?
    Riemann: metrics hub/proxy
    App server overloaded, full disk, etc
    Analyzes metrics
    computes aggregates
    notifies
    

    View Slide

35. Demo
    

    View Slide

36. we are hiring!
    Questions!
    @brutasse
    Image credits: http://en.wikipedia.org/wiki/Explosion
    

    View Slide

37. Resources
    Events Metrics
    structlog.org
    rsyslog
    logstash
    elasticsearch
    kibana
    carbon / whisper
    graphite-api
    grafana
    collectd
    statsd
    riemann.io
    bernhard
    

    View Slide

38. Hosted services
    (not exhaustive)
    Logging Metrics
    Loggly
    Papertrail
    Hosted Graphite
    Librato
    JSON / structured
    logging support?
    

    View Slide