Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Visibility for web developers
Search
Bruno Renié
May 13, 2014
Technology
3
430
Visibility for web developers
Bruno Renié
May 13, 2014
Tweet
Share
More Decks by Bruno Renié
See All by Bruno Renié
Decentralization & real-time with PubSubHubbub
brutasse
1
140
Deployability of Python Web Applications
brutasse
17
2.3k
Stop writing settings files
brutasse
21
2.5k
Class-based Views: patterns and anti-patterns
brutasse
9
1.6k
Packager son projet Django
brutasse
4
550
Staticfiles : tout ce qu'il faut savoir, rien que ce qu'il faut savoir
brutasse
4
530
Introduction to Django
brutasse
3
400
Other Decks in Technology
See All in Technology
継続的な改善 x ⾮連続的な進化
sansantech
PRO
3
130
レガシーをぶっ壊せ。AEONで始めるDevRelの話 / Qiita Night 2024-2-22
aeonpeople
3
1.2k
SPI原点回帰論:事業課題とFour Keysの結節点を見出す実践的ソフトウェアプロセス改善 / DevOpsDays Tokyo 2024
visional_engineering_and_design
4
1.9k
Janus
bkuhlmann
1
490
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
890
20240418_Google ColabにLLMが搭載されたようなのでPython x データ分析の勉強方法を考えてみる
doradora09
0
120
テストプロセスで大事にしていること #jasstnano
makky_tyuyan
0
160
[PlatformCon 24] Platform Orchestrators: The Missing Middle of Internal Developer Platforms?
danielbryantuk
1
830
最近たまに見かけるTiDBってなんだ? - Findy
pingcap0315
2
750
検証を通して見えてきたTiDBの性能特性
lycorptech_jp
PRO
6
3.7k
4年前、あるじゃん老害エンジニアLT合戦に登壇、米国西海岸コンピュータ歴史博物館体験記の続編
toshi_atsumi
0
220
反実仮想機械学習とは何か
usaito
PRO
8
3k
Featured
See All Featured
The Cost Of JavaScript in 2023
addyosmani
16
3.8k
Documentation Writing (for coders)
carmenintech
60
3.9k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
21
1.6k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
A Tale of Four Properties
chriscoyier
151
22k
Stop Working from a Prison Cell
hatefulcrawdad
266
19k
Optimizing for Happiness
mojombo
370
69k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
Producing Creativity
orderedlist
PRO
337
39k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
20
1.9k
Building a Modern Day E-commerce SEO Strategy
aleyda
17
6.4k
Art, The Web, and Tiny UX
lynnandtonic
289
19k
Transcript
Visibility for Web Developers Bruno Renié Django Island 2014
Working with ops & support staff Large infrastructure, lots of
moving parts whoami: lead dev @ exoscale
People need to know what's going on, anytime Not everyone
is a developer SSH, grep and tail not acceptable
Goal: visibility on infrastructure & apps
Errors Events Metrics
Errors
Easy: use sentry
Great tool for developers, but… People also need to know
about things that are not broken
Events
Event == log call Timestamped information about something worth knowing
Easy to emit logs Searching on remote machines, not so
much
Centralized logging Agent forwards logs to an aggregator Aggregator stores
logs and provides a search interface
Aggregator Hosts logstash+elasticsearch kibana log shippers
Collecting logs rsyslog / syslog-ng logstash-forwarder (lumberjack) logstash
In Python logging.handlers.SyslogHandler
Centralizing isn't enough
A common way to log things import logging logger =
logging.getLogger(__name__) logger.info("user {} logged in".format(user.email))
Not much information No structure
Structured logging Record = set of key-value pairs event="user logged
in" user="
[email protected]
" ip="1.2.3.4"
In Python: pip install structlog Hynek Schlawack logger = structlog.get_logger()
logger.info("user logged in", user="
[email protected]
" ip="1.2.3.4") {"event": "user.login", "user": "
[email protected]
", "host": "1.2.3.4"}
Contextual loggers log = logger.bind(user=request.user.email) log.info("payment", amount=120, currency="CHF") {"event": "payment",
"user": "
[email protected]
", "amount": 120, "currency": "CHF"}
structlog JSON elasticsearch Rich data for building dashboards
Demo
Marketing: "Who signed up today?" Finance: "How much income in
the past week?" Support: "What did this user do in the past 24 hours?"
Events Centralize logs Add structure Profit logstash, elasticsearch structlog kibana
Metrics
Time series data Continuous, regular intervals Big player: graphite ecosystem
carbon whisper graphite
<identifier> <value> <timestamp> db1.load.midterm 4.6 1399842778 Measure at regular intervals,
send using the carbon line protocol echo " … " | nc host port
Traditional metrics stack agents collectd, apps, scripts carbon whisper graphite
ingests metrics stores metrics renders graphs
Counting and timing: statsd In-memory buffer Aggregates and flushes to
carbon at regular intervals
In Python pystatsd django-statsd-mozilla from stastd import StastClient statsd =
StatsClient() @statsd.timer("something") def expensive_computation(): # …
Graphing Graphite: rendering API (json / PNG / etc) Countless
dashboard apps Grafana Graph-explorer etc
Demo
What about alerts? Riemann: metrics hub/proxy App server overloaded, full
disk, etc Analyzes metrics computes aggregates notifies
Demo
we are hiring! Questions! @brutasse Image credits: http://en.wikipedia.org/wiki/Explosion
Resources Events Metrics structlog.org rsyslog logstash elasticsearch kibana carbon /
whisper graphite-api grafana collectd statsd riemann.io bernhard
Hosted services (not exhaustive) Logging Metrics Loggly Papertrail Hosted Graphite
Librato JSON / structured logging support?