Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
Visibility for Web Developers Bruno Renié Django Island 2014
Slide 2
Slide 2 text
Working with ops & support staff Large infrastructure, lots of moving parts whoami: lead dev @ exoscale
Slide 3
Slide 3 text
People need to know what's going on, anytime Not everyone is a developer SSH, grep and tail not acceptable
Slide 4
Slide 4 text
Goal: visibility on infrastructure & apps
Slide 5
Slide 5 text
Errors Events Metrics
Slide 6
Slide 6 text
Errors
Slide 7
Slide 7 text
Easy: use sentry
Slide 8
Slide 8 text
Great tool for developers, but… People also need to know about things that are not broken
Slide 9
Slide 9 text
Events
Slide 10
Slide 10 text
Event == log call Timestamped information about something worth knowing
Slide 11
Slide 11 text
Easy to emit logs Searching on remote machines, not so much
Slide 12
Slide 12 text
Centralized logging Agent forwards logs to an aggregator Aggregator stores logs and provides a search interface
Slide 13
Slide 13 text
Aggregator Hosts logstash+elasticsearch kibana log shippers
Slide 14
Slide 14 text
Collecting logs rsyslog / syslog-ng logstash-forwarder (lumberjack) logstash
Slide 15
Slide 15 text
In Python logging.handlers.SyslogHandler
Slide 16
Slide 16 text
Centralizing isn't enough
Slide 17
Slide 17 text
A common way to log things import logging logger = logging.getLogger(__name__) logger.info("user {} logged in".format(user.email))
Slide 18
Slide 18 text
Not much information No structure
Slide 19
Slide 19 text
Structured logging Record = set of key-value pairs event="user logged in" user="
[email protected]
" ip="1.2.3.4"
Slide 20
Slide 20 text
In Python: pip install structlog Hynek Schlawack logger = structlog.get_logger() logger.info("user logged in", user="
[email protected]
" ip="1.2.3.4") {"event": "user.login", "user": "
[email protected]
", "host": "1.2.3.4"}
Slide 21
Slide 21 text
Contextual loggers log = logger.bind(user=request.user.email) log.info("payment", amount=120, currency="CHF") {"event": "payment", "user": "
[email protected]
", "amount": 120, "currency": "CHF"}
Slide 22
Slide 22 text
structlog JSON elasticsearch Rich data for building dashboards
Slide 23
Slide 23 text
Demo
Slide 24
Slide 24 text
Marketing: "Who signed up today?" Finance: "How much income in the past week?" Support: "What did this user do in the past 24 hours?"
Slide 25
Slide 25 text
Events Centralize logs Add structure Profit logstash, elasticsearch structlog kibana
Slide 26
Slide 26 text
Metrics
Slide 27
Slide 27 text
Time series data Continuous, regular intervals Big player: graphite ecosystem carbon whisper graphite
Slide 28
Slide 28 text
db1.load.midterm 4.6 1399842778 Measure at regular intervals, send using the carbon line protocol echo " … " | nc host port
Slide 29
Slide 29 text
Traditional metrics stack agents collectd, apps, scripts carbon whisper graphite ingests metrics stores metrics renders graphs
Slide 30
Slide 30 text
Counting and timing: statsd In-memory buffer Aggregates and flushes to carbon at regular intervals
Slide 31
Slide 31 text
In Python pystatsd django-statsd-mozilla from stastd import StastClient statsd = StatsClient() @statsd.timer("something") def expensive_computation(): # …
Slide 32
Slide 32 text
Graphing Graphite: rendering API (json / PNG / etc) Countless dashboard apps Grafana Graph-explorer etc
Slide 33
Slide 33 text
Demo
Slide 34
Slide 34 text
What about alerts? Riemann: metrics hub/proxy App server overloaded, full disk, etc Analyzes metrics computes aggregates notifies
Slide 35
Slide 35 text
Demo
Slide 36
Slide 36 text
we are hiring! Questions! @brutasse Image credits: http://en.wikipedia.org/wiki/Explosion
Slide 37
Slide 37 text
Resources Events Metrics structlog.org rsyslog logstash elasticsearch kibana carbon / whisper graphite-api grafana collectd statsd riemann.io bernhard
Slide 38
Slide 38 text
Hosted services (not exhaustive) Logging Metrics Loggly Papertrail Hosted Graphite Librato JSON / structured logging support?