Visibility forWeb DevelopersBruno ReniéDjango Island 2014
View Slide
Working with ops & support staffLarge infrastructure, lots of moving partswhoami: lead dev @ exoscale
People need to know what'sgoing on, anytimeNot everyone is a developerSSH, grep and tail not acceptable
Goal:visibility on infrastructure & apps
ErrorsEventsMetrics
Errors
Easy: use sentry
Great tool for developers, but…People also need to know aboutthings that are not broken
Events
Event == log callTimestamped information aboutsomething worth knowing
Easy to emit logsSearching on remote machines,not so much
Centralized loggingAgent forwards logsto an aggregatorAggregator stores logs andprovides a search interface
AggregatorHostslogstash+elasticsearchkibanalog shippers
Collecting logsrsyslog / syslog-nglogstash-forwarder (lumberjack)logstash
In Pythonlogging.handlers.SyslogHandler
Centralizing isn't enough
A common way to log thingsimport logginglogger = logging.getLogger(__name__)logger.info("user {} logged in".format(user.email))
Not much informationNo structure
Structured loggingRecord = set of key-value pairsevent="user logged in"user="[email protected]"ip="1.2.3.4"
In Python:pip install structlogHynek Schlawacklogger = structlog.get_logger()logger.info("user logged in",user="[email protected]"ip="1.2.3.4"){"event": "user.login","user": "[email protected]","host": "1.2.3.4"}
Contextual loggerslog = logger.bind(user=request.user.email)log.info("payment", amount=120,currency="CHF"){"event": "payment","user": "[email protected]","amount": 120,"currency": "CHF"}
structlogJSONelasticsearchRich data for building dashboards
Demo
Marketing: "Who signed up today?"Finance: "How much incomein the past week?"Support: "What did this user doin the past 24 hours?"
EventsCentralize logsAdd structureProfitlogstash, elasticsearchstructlogkibana
Metrics
Time series dataContinuous, regular intervalsBig player: graphite ecosystemcarbonwhispergraphite
db1.load.midterm 4.6 1399842778Measure at regular intervals,send using the carbon line protocolecho " … " | nc host port
Traditional metrics stackagentscollectd, apps, scriptscarbonwhispergraphiteingests metricsstores metricsrenders graphs
Counting and timing: statsdIn-memory bufferAggregates and flushesto carbonat regular intervals
In Pythonpystatsddjango-statsd-mozillafrom stastd import StastClientstatsd = StatsClient()@statsd.timer("something")def expensive_computation():# …
GraphingGraphite: rendering API (json / PNG / etc)Countless dashboard appsGrafanaGraph-exploreretc
What about alerts?Riemann: metrics hub/proxyApp server overloaded, full disk, etcAnalyzes metricscomputes aggregatesnotifies
we are hiring!Questions!@brutasseImage credits: http://en.wikipedia.org/wiki/Explosion
ResourcesEvents Metricsstructlog.orgrsysloglogstashelasticsearchkibanacarbon / whispergraphite-apigrafanacollectdstatsdriemann.iobernhard
Hosted services(not exhaustive)Logging MetricsLogglyPapertrailHosted GraphiteLibratoJSON / structuredlogging support?