Measuring everything in Real-Time

Measuring everything in Real-Time @BastianHofmann

Logging

Measuring

Testing

many roads

None

None

None

None

None

None

None

None

Questions? Ask!

http://speakerdeck.com/u/bastianhofmann

Logging

For when something goes wrong

server error log access log debug logs slow query log

...

Error Logs

$ tail -f error.log $ grep

ErrorLog /var/logs/apache/error.log

php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log

callable set_exception_handler( callable $exception_handler );

callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |

E_STRICT ] );

Log additional info

Error code

None

HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

None

Access Logs

192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"

200 882 "http://devm/rg_trunk/webroot/ directory/publications/"

LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog

/var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat

http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =

"" ] )

LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{server_ip}n ..." custom nginx?

Debug Logs

https://github.com/Seldaek/monolog/

<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel

$log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');

Handlers • Stream • Mail • FirePHP • ChromePHP •

Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed

<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new

Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );

Log in a structured way

JSON http://www.ietf.org/rfc/rfc4627.txt

<?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new

Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);

http://getcomposer.org/ http://packagist.org/

Logs from other services

web server db memcached db http service user request log

log log log log

Correlation / Tracing ID

web server db memcached db http service create unique trace_id

for request user request trace_id trace_id trace_id trace_id log log log log log

X-Trace-Id: bbr8ehb984tbab894

None

None

None

Aggregate the logs in a central place

Make them easily full- text searchable

Make them aggregate- able

Always Log to file

Seriously...

Always Log to file

AND REMEMBER

Always Log to file

Directly to a database

webserver webserver webserver DB

Disadvantages

Database is down?

Database is slow?

Database is full?

Better solutions?

graylog2 http://graylog2.org/

Full text search

Structured Messages

Metrics & Alarms

Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver

{ "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace

here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }

Disadvantages

Graylog/elasticsearch is down?

Graylog/elasticsearch is full?

Packet loss

Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF

None

Don't influence your application by logging

logstash http://logstash.net/

Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

logstash logstash logstash GELF

Very rich plugin system

input filter output

file { type => "error" path => [ "/var/logs/php/*.log" ]

add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }

filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {

type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }

output { amqp { host => "amqp.host" exchange_type => "fanout"

name => "logs" } }

input { amqp { host => 'amqp.host' port => '5672'

exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }

output { gelf { host => '127.0.0.1' } }

Measuring

For knowing what happens

Munin http://munin-monitoring.org/

Ganglia http://ganglia.sourceforge.net/

None

Graphite http://graphite.wikidot.com/

webserver webserver webserver graphite UDP

webserver webserver webserver graphite UDP statsd

StatsD https://github.com/etsy/statsd/

webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

statsd

https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');

None

Technical Metrics

Business Metrics

Define your KPIs

Testing

For improving your app

Existing features

New features

A/B

BUY IT BUY IT $userId % 2 === 0 $userId

= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown

BUY IT BUY IT save random decision in session $userId

= 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown

Save when experiment is shown

None

Save when a goal is reached

None

Conversion

Define your goals!

Disadvantages

Bad conversion during test

Only works for temporary tests

Inflexible: changing of tests not supported

There are better methods with better results

Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/

BUY IT BUY IT random decision $userId = 24 save

that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1

BUY IT BUY IT 2 90% users best option rest

other options

Integrate it into your architecture

None

Be careful

https://github.com/bashofmann/ Try it out

Did you like this talk? h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann

[email protected] http://programm.froscon.de/2012/ events/916.html