Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Measuring everything in Real-Time
Search
Bastian Hofmann
August 25, 2012
Programming
4
640
Measuring everything in Real-Time
Talk from Froscon
Bastian Hofmann
August 25, 2012
Tweet
Share
More Decks by Bastian Hofmann
See All by Bastian Hofmann
Monitoring in Kubernetes with Prometheus and Grafana
bastianhofmann
0
260
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
85
Highly available cross-region deployments with Kubernetes
bastianhofmann
1
120
From source to Kubernetes in 30 minutes
bastianhofmann
0
120
Introduction to Kubernetes
bastianhofmann
1
140
CI/CD with Kubernetes
bastianhofmann
0
140
Creating a fast Kubernetes Development Workflow
bastianhofmann
1
240
Deploying your first Micro-Service application to Kubernetes
bastianhofmann
2
160
Creating a fast Kubernetes Development Workflow
bastianhofmann
0
160
Other Decks in Programming
See All in Programming
Polars入門
daikikatsuragawa
1
100
OpenAPIを中心に考えるAPI開発入門 / Introduction to API Development with a Focus on OpenAPI
seike460
PRO
2
170
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
110
VS Code をプロダクトにどう取り込むか
onomax
1
430
Fragment Composition of GraphQL
quramy
7
1k
AWS CDKコントリビュートTIPS / aws-cdk-contribution-tips
gotok365
2
230
見た目から始める生産性向上
ikumatadokoro
8
920
Micro Frontends for Java Microservices - Devnexus 2024
mraible
PRO
0
500
TCAとKMPを用いた新規動画配信アプリ 「ABEMA Live」の設計
tomu28
1
120
Goのエラースタックトレースの歴史と今後
sonatard
9
1.6k
Let's learn code review
riofujimon
2
490
Amazon SQSコンシューマー疎結合への旅 - 出張! #DevelopersIO IT技術ブログの中の人が語る勉強会 #3
quiver
0
280
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
243
12k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
The Pragmatic Product Professional
lauravandoore
25
5.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
322
20k
Rails Girls Zürich Keynote
gr2m
91
13k
Debugging Ruby Performance
tmm1
70
11k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
Bash Introduction
62gerente
604
210k
Being A Developer After 40
akosma
57
580k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
241
1.2M
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
19
1.7k
Automating Front-end Workflow
addyosmani
1356
200k
Transcript
Measuring everything in Real-Time @BastianHofmann
Logging
Measuring
Testing
many roads
None
None
None
None
None
None
None
None
Questions? Ask!
http://speakerdeck.com/u/bastianhofmann
Logging
For when something goes wrong
server error log access log debug logs slow query log
...
Error Logs
$ tail -f error.log $ grep
ErrorLog /var/logs/apache/error.log
php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log
callable set_exception_handler( callable $exception_handler );
callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |
E_STRICT ] );
Log additional info
Error code
None
HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
None
Access Logs
192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"
200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog
/var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =
"" ] )
LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{server_ip}n ..." custom nginx?
Debug Logs
https://github.com/Seldaek/monolog/
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel
$log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
Handlers • Stream • Mail • FirePHP • ChromePHP •
Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new
Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
Log in a structured way
JSON http://www.ietf.org/rfc/rfc4627.txt
<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new
Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
http://getcomposer.org/ http://packagist.org/
Logs from other services
web server db memcached db http service user request log
log log log log
Correlation / Tracing ID
web server db memcached db http service create unique trace_id
for request user request trace_id trace_id trace_id trace_id log log log log log
X-Trace-Id: bbr8ehb984tbab894
None
None
None
Aggregate the logs in a central place
Make them easily full- text searchable
Make them aggregate- able
Always Log to file
Seriously...
Always Log to file
AND REMEMBER
Always Log to file
Directly to a database
webserver webserver webserver DB
Disadvantages
Database is down?
Database is slow?
Database is full?
Better solutions?
graylog2 http://graylog2.org/
Full text search
Structured Messages
Metrics & Alarms
Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver
{ "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace
here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
Disadvantages
Graylog/elasticsearch is down?
Graylog/elasticsearch is full?
Packet loss
Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF
None
Don't influence your application by logging
logstash http://logstash.net/
Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash
logstash logstash logstash GELF
Very rich plugin system
input filter output
file { type => "error" path => [ "/var/logs/php/*.log" ]
add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {
type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
output { amqp { host => "amqp.host" exchange_type => "fanout"
name => "logs" } }
input { amqp { host => 'amqp.host' port => '5672'
exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
output { gelf { host => '127.0.0.1' } }
Measuring
For knowing what happens
Munin http://munin-monitoring.org/
Ganglia http://ganglia.sourceforge.net/
None
Graphite http://graphite.wikidot.com/
webserver webserver webserver graphite UDP
webserver webserver webserver graphite UDP statsd
StatsD https://github.com/etsy/statsd/
webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
statsd
https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');
None
Technical Metrics
Business Metrics
Define your KPIs
Testing
For improving your app
Existing features
New features
A/B
BUY IT BUY IT $userId % 2 === 0 $userId
= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
BUY IT BUY IT save random decision in session $userId
= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
Save when experiment is shown
None
Save when a goal is reached
None
Conversion
Define your goals!
Disadvantages
Bad conversion during test
Only works for temporary tests
Inflexible: changing of tests not supported
There are better methods with better results
Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/
BUY IT BUY IT random decision $userId = 24 save
that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
BUY IT BUY IT 2 90% users best option rest
other options
Integrate it into your architecture
None
Be careful
https://github.com/bashofmann/ Try it out
Did you like this talk? h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann
[email protected]
http://programm.froscon.de/2012/ events/916.html