Share
Embed
Slide 1
Slide 1 text
Measuring and Logging
everything in Real-Time
@BastianHofmann
Slide 2
Slide 2 text
Logging
Slide 3
Slide 3 text
Measuring
Slide 4
Slide 4 text
Testing
Slide 5
Slide 5 text
many roads
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Questions?
Ask!
Slide 15
Slide 15 text
http://speakerdeck.com/u/bastianhofmann
Slide 16
Slide 16 text
Logging
Slide 17
Slide 17 text
For when something
goes wrong
Slide 18
Slide 18 text
server
error log
access log
debug logs
slow query log
...
Slide 19
Slide 19 text
Error Logs
Slide 20
Slide 20 text
$ tail -f error.log
$ grep
Slide 21
Slide 21 text
ErrorLog /var/logs/apache/error.log
Slide 22
Slide 22 text
php.ini
•error_reporting
•display_errors
•display_startup_errors
•log_errors
•error_log
Slide 23
Slide 23 text
callable set_exception_handler(
callable $exception_handler
);
Slide 24
Slide 24 text
callable set_error_handler(
callable $error_handler
[, int $error_types = E_ALL |
E_STRICT ]
);
Slide 25
Slide 25 text
Log additional info
Slide 26
Slide 26 text
Error code
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
HTTP Response Codes
http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
Access Logs
Slide 31
Slide 31 text
192.168.56.1 - - [09/Jul/2012:19:18:19
+0200] "GET /rg_trunk/webroot/c/af10c/
images/template/rg_logo_default.png HTTP/
1.1" 200 882 "http://devm/rg_trunk/webroot/
directory/publications/"
Slide 32
Slide 32 text
LogFormat "%h %l %u %t \"%r\" %>s %b" custom
CustomLog /var/logs/apache/access.log custom
http://httpd.apache.org/docs/2.2/mod/
mod_log_config.html#logformat
Slide 33
Slide 33 text
http://de.php.net/apache_note
string apache_note (
string $note_name
[, string $note_value = "" ]
)
Slide 34
Slide 34 text
LogFormat "...\"%{referer}i\" \"%{user-
agent}i\" %{session_id}n %{account_id}n..."
custom nginx?
Slide 35
Slide 35 text
Debug Logs
Slide 36
Slide 36 text
https://github.com/Seldaek/monolog/
Slide 37
Slide 37 text
pushHandler(
new StreamHandler(
'path/to/your.log', Logger::WARNING
)
);
// add records to the log
$log->addWarning('Foo');
$log->addError('Bar');
Slide 38
Slide 38 text
Handlers
• Stream
• Mail
• FirePHP
• ChromePHP
• Socket
• Roating File
• MongoDB
• Syslog
• Gelf
• Null
• Test
• FingersCrossed
Slide 39
Slide 39 text
pushHandler($handler);
$log->pushProcessor(
new MemoryUsageProcessor()
);
Slide 40
Slide 40 text
Log in a structured way
Slide 41
Slide 41 text
JSON
http://www.ietf.org/rfc/rfc4627.txt
Slide 42
Slide 42 text
setFormatter(new JsonFormatter());
$log->pushHandler($handler);
Slide 43
Slide 43 text
http://getcomposer.org/
http://packagist.org/
Slide 44
Slide 44 text
$ cat composer.json
Slide 45
Slide 45 text
$ wget http://getcomposer.org/composer.phar
$ php composer.phar install
Installing dependencies
- Installing monolog/monolog (1.1.0)
Downloading: 100%
...
monolog/monolog suggests installing mlehner/
gelf-php (Allow sending log messages to a
GrayLog2 server)
Writing lock file
Generating autoload files
Slide 46
Slide 46 text
Logs from other
services
Slide 47
Slide 47 text
web server db
memcached
db
http service
user request
log
log
log
log
log
Slide 48
Slide 48 text
Correlation / Tracing ID
Slide 49
Slide 49 text
web server db
memcached
db
http service
create
unique
trace_id for
request
user request
trace_id
trace_id
trace_id
trace_id
log
log
log
log
log
Slide 50
Slide 50 text
X-Trace-Id: bbr8ehb984tbab894
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
Aggregate the logs in a
central place
Slide 55
Slide 55 text
Make them easily full-
text searchable
Slide 56
Slide 56 text
Make them aggregate-
able
Slide 57
Slide 57 text
Always Log to file
Slide 58
Slide 58 text
Seriously...
Slide 59
Slide 59 text
Always Log to file
Slide 60
Slide 60 text
AND REMEMBER
Slide 61
Slide 61 text
Always Log to file
Slide 62
Slide 62 text
Directly to a database
Slide 63
Slide 63 text
webserver webserver webserver
DB
Slide 64
Slide 64 text
Disadvantages
Slide 65
Slide 65 text
Database is down?
Slide 66
Slide 66 text
Database is slow?
Slide 67
Slide 67 text
Database is full?
Slide 68
Slide 68 text
Better solutions?
Slide 69
Slide 69 text
graylog2
http://graylog2.org/
Slide 70
Slide 70 text
Full text search
Slide 71
Slide 71 text
Structured Messages
Slide 72
Slide 72 text
Metrics & Alarms
Slide 73
Slide 73 text
Graylog2
UDP GELF Messages
elasticsearch
webserver webserver webserver
Slide 74
Slide 74 text
{
"version": "1.0",
"host": "www1",
"short_message": "Short message",
"full_message": "Backtrace here\n
\nmore stuff",
"timestamp": 1291899928.412,
"level": 1,
"facility": "payment-backend",
"file": "/var/www/somefile.rb",
"line": 356,
"_user_id": 42,
"_something_else": "foo"
}
Slide 75
Slide 75 text
Disadvantages
Slide 76
Slide 76 text
Graylog/elasticsearch is
down?
Slide 77
Slide 77 text
Graylog/elasticsearch is
full?
Slide 78
Slide 78 text
Packet loss
Slide 79
Slide 79 text
Graylog2
elasticsearch
webserver webserver webserver
AMQP
GELF GELF
GELF
GELF
Slide 80
Slide 80 text
No content
Slide 81
Slide 81 text
Don't influence your
application by logging
Slide 82
Slide 82 text
logstash
http://logstash.net/
Slide 83
Slide 83 text
Graylog2
elasticsearch
webserver webserver webserver
AMQP
log log log
logstash logstash logstash
logstash
GELF
Slide 84
Slide 84 text
Graylog2
elasticsearch
webserver webserver webserver
AMQP
log log log
logstash logstash logstash
Slide 85
Slide 85 text
Very rich plugin system
Slide 86
Slide 86 text
input filter output
Slide 87
Slide 87 text
file {
type => "error"
path =>
[ "/var/logs/php/*.log" ]
add_field =>
[ "severity", "error" ]
}
file {
type => "access"
path =>
[ "/var/logs/apache/*_access.log" ]
add_field =>
[ "severity", "info" ]
}
Slide 88
Slide 88 text
filter{
grok {
match => ["@source",
"\/%{USERNAME:facility}\.log$"]
}
grok {
type => "access"
pattern => "^%{IP:OriginalIp}
\s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*?
\]\s\"%{DATA:Request}..."
}
}
Slide 89
Slide 89 text
output {
amqp {
host => "amqp.host"
exchange_type => "fanout"
name => "logs"
}
}
Slide 90
Slide 90 text
output {
gelfamqp {
host => "amqp.host"
exchange_type => "fanout"
name => "logs"
}
}
Slide 91
Slide 91 text
input {
amqp {
host => 'amqp.host'
port => '5672'
exchange => 'logs'
name => 'logs'
type => 'logs'
exclusive => false
}
}
Slide 92
Slide 92 text
output {
gelf {
host => '127.0.0.1'
}
}
Slide 93
Slide 93 text
Measuring
Slide 94
Slide 94 text
For knowing what
happens
Slide 95
Slide 95 text
Munin
http://munin-monitoring.org/
Slide 96
Slide 96 text
Ganglia
http://ganglia.sourceforge.net/
Slide 97
Slide 97 text
No content
Slide 98
Slide 98 text
Graphite
http://graphite.wikidot.com/
Slide 99
Slide 99 text
webserver webserver webserver
graphite
UDP
Slide 100
Slide 100 text
webserver webserver webserver
graphite
UDP statsd
Slide 101
Slide 101 text
StatsD
https://github.com/etsy/statsd/
Slide 102
Slide 102 text
webserver webserver webserver
statsd statsd
statsd
graphite
aggregated
UPD message
Slide 103
Slide 103 text
webserver webserver webserver
statsd statsd
statsd
graphite
aggregated
UPD message
statsd
Slide 104
Slide 104 text
https://github.com/etsy/statsd/blob/master/examples/php-
example.php
StatsD::increment('key');
Slide 105
Slide 105 text
No content
Slide 106
Slide 106 text
https://metrics.librato.com/
Slide 107
Slide 107 text
Technical Metrics
Slide 108
Slide 108 text
Business Metrics
Slide 109
Slide 109 text
Define your KPIs
Slide 110
Slide 110 text
http://mashable.com/2010/10/13/etsy-office-pics/
Slide 111
Slide 111 text
Testing
Slide 112
Slide 112 text
For improving your app
Slide 113
Slide 113 text
Existing features
Slide 114
Slide 114 text
New features
Slide 115
Slide 115 text
A/B
Slide 116
Slide 116 text
BUY IT BUY IT
$userId % 2 === 0
$userId = 24
save that option A was clicked
save that option A is shown
conversion = option A clicked / option A shown
Slide 117
Slide 117 text
BUY IT BUY IT
save random decision
in session
$userId = 24
save that option A was clicked
save that option A is shown
conversion = option A clicked / option A shown
Slide 118
Slide 118 text
Save when experiment
is shown
Slide 119
Slide 119 text
No content
Slide 120
Slide 120 text
Save when a goal is
reached
Slide 121
Slide 121 text
No content
Slide 122
Slide 122 text
Conversion
Slide 123
Slide 123 text
Define your goals!
Slide 124
Slide 124 text
Disadvantages
Slide 125
Slide 125 text
Bad conversion during
test
Slide 126
Slide 126 text
Only works for
temporary tests
Slide 127
Slide 127 text
Inflexible: changing of
tests not supported
Slide 128
Slide 128 text
There are better
methods with better
results
Slide 129
Slide 129 text
Feature Flags
https://github.com/rcrowe/fuel-flags
Slide 130
Slide 130 text
Bandit Algorithm
http://untyped.com/untyping/2011/02/11/stop-ab-testing-
and-make-out-like-a-bandit/
Slide 131
Slide 131 text
BUY IT BUY IT
random decision
$userId = 24
save that option A was clicked
save that option A is shown
conversion = option A clicked / option A shown
1
Slide 132
Slide 132 text
BUY IT BUY IT
2
90% users best option rest other options
Slide 133
Slide 133 text
Integrate it into your
architecture
Slide 134
Slide 134 text
No content
Slide 135
Slide 135 text
Be careful
Slide 136
Slide 136 text
https://github.com/bashofmann/
Try it out
Slide 137
Slide 137 text
h"p://twi"er.com/Bas2anHofmann
h"p://profiles.google.com/bashofmann
h"p://lanyrd.com/people/Bas2anHofmann
h"p://speakerdeck.com/u/bas2anhofmann
h"ps://github.com/bashofmann
[email protected]