Measuring and Logging everything in Real-Time

Web Developer Conference 2012, Hamburg

Transcript

1. Measuring and Logging everything in Real-Time @BastianHofmann
2. Logging
3. Measuring
4. Testing
5. many roads
14. Questions? Ask!
15. http://speakerdeck.com/u/bastianhofmann
16. Logging
17. For when something goes wrong
18. server error log access log debug logs slow query log ...
19. Error Logs
20. $ tail -f error.log $ grep
21. ErrorLog /var/logs/apache/error.log
22. php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log
23. callable set_exception_handler( callable $exception_handler );
24. callable set_error_handler( callable $error_handler [, int $error_types = E_ALL | E_STRICT ] );
25. Log additional info
26. Error code
28. HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
30. Access Logs
31. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1" 200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog /var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
33. http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value = "" ] )
34. LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{session_id}n %{account_id}n..." custom nginx?
35. Debug Logs
36. https://github.com/Seldaek/monolog/
37. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel $log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
38. Handlers • Stream • Mail • FirePHP • ChromePHP • Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
39. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
40. Log in a structured way
41. JSON http://www.ietf.org/rfc/rfc4627.txt
42. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
43. http://getcomposer.org/ http://packagist.org/
44. $ cat composer.json
45. $ wget http://getcomposer.org/composer.phar $ php composer.phar install Installing dependencies - Installing monolog/monolog (1.1.0) Downloading: 100% ... monolog/monolog suggests installing mlehner/ gelf-php (Allow sending log messages to a GrayLog2 server) Writing lock file Generating autoload files
46. Logs from other services
47. web server db memcached db http service user request log log log log log
48. Correlation / Tracing ID
49. web server db memcached db http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
50. X-Trace-Id: bbr8ehb984tbab894
54. Aggregate the logs in a central place
55. Make them easily full- text searchable
56. Make them aggregate- able
57. Always Log to file
58. Seriously...
59. Always Log to file
60. AND REMEMBER
61. Always Log to file
62. Directly to a database
63. webserver webserver webserver DB
64. Disadvantages
65. Database is down?
66. Database is slow?
67. Database is full?
68. Better solutions?
69. graylog2 http://graylog2.org/
70. Full text search
71. Structured Messages
72. Metrics & Alarms
73. Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver
74. { "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
75. Disadvantages
76. Graylog/elasticsearch is down?
77. Graylog/elasticsearch is full?
78. Packet loss
79. Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF
81. Don't influence your application by logging
82. logstash http://logstash.net/
83. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash logstash GELF
84. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
85. Very rich plugin system
86. input filter output
87. file { type => "error" path => [ "/var/logs/php/*.log" ] add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
88. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok { type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
89. output { amqp { host => "amqp.host" exchange_type => "fanout" name => "logs" } }
90. output { gelfamqp { host => "amqp.host" exchange_type => "fanout" name => "logs" } }
91. input { amqp { host => 'amqp.host' port => '5672' exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
92. output { gelf { host => '127.0.0.1' } }
93. Measuring
94. For knowing what happens
95. Munin http://munin-monitoring.org/
96. Ganglia http://ganglia.sourceforge.net/
98. Graphite http://graphite.wikidot.com/
99. webserver webserver webserver graphite UDP
100. webserver webserver webserver graphite UDP statsd
101. StatsD https://github.com/etsy/statsd/
102. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
103. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message statsd
104. https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');
106. https://metrics.librato.com/
107. Technical Metrics
108. Business Metrics
109. Define your KPIs
110. http://mashable.com/2010/10/13/etsy-office-pics/
111. Testing
112. For improving your app
113. Existing features
114. New features
115. A/B
116. BUY IT BUY IT $userId % 2 === 0 $userId = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
117. BUY IT BUY IT save random decision in session $userId = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
118. Save when experiment is shown
120. Save when a goal is reached
122. Conversion
123. Define your goals!
124. Disadvantages
125. Bad conversion during test
126. Only works for temporary tests
127. Inflexible: changing of tests not supported
128. There are better methods with better results
129. Feature Flags https://github.com/rcrowe/fuel-flags
130. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/
131. BUY IT BUY IT random decision $userId = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
132. BUY IT BUY IT 2 90% users best option rest other options
133. Integrate it into your architecture
135. Be careful
136. https://github.com/bashofmann/ Try it out
137. h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann mail@bas2anhofmann.de