Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Measuring and Logging everything in Real-Time
Search
Bastian Hofmann
September 18, 2012
Programming
2
490
Measuring and Logging everything in Real-Time
Web Developer Conference 2012, Hamburg
Bastian Hofmann
September 18, 2012
Tweet
Share
More Decks by Bastian Hofmann
See All by Bastian Hofmann
Monitoring in Kubernetes with Prometheus and Grafana
bastianhofmann
0
330
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
110
Highly available cross-region deployments with Kubernetes
bastianhofmann
1
140
From source to Kubernetes in 30 minutes
bastianhofmann
0
160
Introduction to Kubernetes
bastianhofmann
1
140
CI/CD with Kubernetes
bastianhofmann
0
190
Creating a fast Kubernetes Development Workflow
bastianhofmann
1
250
Deploying your first Micro-Service application to Kubernetes
bastianhofmann
2
180
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
220
Other Decks in Programming
See All in Programming
Let's Write a Train Tracking Algorithm
twocentstudios
0
220
Reduxモダナイズ 〜コードのモダン化を通して、将来のライブラリ移行に備える〜
pvcresin
2
620
uniqueパッケージの内部実装を支えるweak pointerの話
magavel
0
660
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
2.1k
階層構造を表現するデータ構造とリファクタリング 〜1年で10倍成長したプロダクトの変化と課題〜
yuhisatoxxx
3
690
検索機能リプレイスを4ヶ月→2ヶ月に! AI Agentで実現した2倍速リプレイス
fuuki12
4
830
Current States of Java Web Frameworks at JCConf 2025
kishida
0
450
10年もののAPIサーバーにおけるCI/CDの改善の奮闘
mbook
0
540
Local Peer-to-Peer APIはどのように使われていくのか?
hal_spidernight
2
400
WebエンジニアがSwiftをブラウザで動かすプレイグラウンドを作ってみた
ohmori_yusuke
0
160
Repenser les filtres API Platform: une nouvelle syntaxe
vinceamstoutz
2
150
Swiftビルド弾丸ツアー - Swift Buildが作る新しいエコシステム
giginet
PRO
0
1.5k
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
513
110k
Unsuck your backbone
ammeep
671
58k
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Practical Orchestrator
shlominoach
190
11k
Music & Morning Musume
bryan
46
6.8k
A designer walks into a library…
pauljervisheath
208
24k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
Fireside Chat
paigeccino
40
3.7k
The Language of Interfaces
destraynor
162
25k
4 Signs Your Business is Dying
shpigford
185
22k
Transcript
Measuring and Logging everything in Real-Time @BastianHofmann
Logging
Measuring
Testing
many roads
None
None
None
None
None
None
None
None
Questions? Ask!
http://speakerdeck.com/u/bastianhofmann
Logging
For when something goes wrong
server error log access log debug logs slow query log
...
Error Logs
$ tail -f error.log $ grep
ErrorLog /var/logs/apache/error.log
php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log
callable set_exception_handler( callable $exception_handler );
callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |
E_STRICT ] );
Log additional info
Error code
None
HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
None
Access Logs
192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"
200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog
/var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =
"" ] )
LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{session_id}n %{account_id}n..." custom nginx?
Debug Logs
https://github.com/Seldaek/monolog/
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel
$log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
Handlers • Stream • Mail • FirePHP • ChromePHP •
Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new
Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
Log in a structured way
JSON http://www.ietf.org/rfc/rfc4627.txt
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new
Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
http://getcomposer.org/ http://packagist.org/
$ cat composer.json
$ wget http://getcomposer.org/composer.phar $ php composer.phar install Installing dependencies -
Installing monolog/monolog (1.1.0) Downloading: 100% ... monolog/monolog suggests installing mlehner/ gelf-php (Allow sending log messages to a GrayLog2 server) Writing lock file Generating autoload files
Logs from other services
web server db memcached db http service user request log
log log log log
Correlation / Tracing ID
web server db memcached db http service create unique trace_id
for request user request trace_id trace_id trace_id trace_id log log log log log
X-Trace-Id: bbr8ehb984tbab894
None
None
None
Aggregate the logs in a central place
Make them easily full- text searchable
Make them aggregate- able
Always Log to file
Seriously...
Always Log to file
AND REMEMBER
Always Log to file
Directly to a database
webserver webserver webserver DB
Disadvantages
Database is down?
Database is slow?
Database is full?
Better solutions?
graylog2 http://graylog2.org/
Full text search
Structured Messages
Metrics & Alarms
Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver
{ "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace
here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
Disadvantages
Graylog/elasticsearch is down?
Graylog/elasticsearch is full?
Packet loss
Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF
None
Don't influence your application by logging
logstash http://logstash.net/
Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash
logstash logstash logstash GELF
Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash
logstash logstash
Very rich plugin system
input filter output
file { type => "error" path => [ "/var/logs/php/*.log" ]
add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {
type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
output { amqp { host => "amqp.host" exchange_type => "fanout"
name => "logs" } }
output { gelfamqp { host => "amqp.host" exchange_type => "fanout"
name => "logs" } }
input { amqp { host => 'amqp.host' port => '5672'
exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
output { gelf { host => '127.0.0.1' } }
Measuring
For knowing what happens
Munin http://munin-monitoring.org/
Ganglia http://ganglia.sourceforge.net/
None
Graphite http://graphite.wikidot.com/
webserver webserver webserver graphite UDP
webserver webserver webserver graphite UDP statsd
StatsD https://github.com/etsy/statsd/
webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
statsd
https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');
None
https://metrics.librato.com/
Technical Metrics
Business Metrics
Define your KPIs
http://mashable.com/2010/10/13/etsy-office-pics/
Testing
For improving your app
Existing features
New features
A/B
BUY IT BUY IT $userId % 2 === 0 $userId
= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
BUY IT BUY IT save random decision in session $userId
= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
Save when experiment is shown
None
Save when a goal is reached
None
Conversion
Define your goals!
Disadvantages
Bad conversion during test
Only works for temporary tests
Inflexible: changing of tests not supported
There are better methods with better results
Feature Flags https://github.com/rcrowe/fuel-flags
Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/
BUY IT BUY IT random decision $userId = 24 save
that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
BUY IT BUY IT 2 90% users best option rest
other options
Integrate it into your architecture
None
Be careful
https://github.com/bashofmann/ Try it out
h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann
[email protected]