Logging and Experiment Testing

Slides of my PHP unconference talk

Transcript

1. Measuring everything in Real-Time @BastianHofmann
2. Logging
3. Measuring
4. Testing
5. many roads
14. Questions? Ask!
15. http://speakerdeck.com/u/bastianhofmann
16. Logging
17. For when something goes wrong
18. server error log access log debug logs slow query log ...
19. Error Logs
20. $ tail -f error.log $ grep
21. ErrorLog /var/logs/apache/error.log
22. php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log
23. callable set_exception_handler( callable $exception_handler );
24. callable set_error_handler( callable $error_handler [, int $error_types = E_ALL | E_STRICT ] );
25. Log additional info
26. Error code
28. HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
30. Access Logs
31. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1" 200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog /var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
33. http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value = "" ] )
34. LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{server_ip}n ..." custom nginx?
35. Debug Logs
36. https://github.com/Seldaek/monolog/
37. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel $log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
38. Handlers • Stream • Mail • FirePHP • ChromePHP • Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
39. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
40. Log in a structured way
41. JSON http://www.ietf.org/rfc/rfc4627.txt
42. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
43. http://getcomposer.org/ http://packagist.org/
44. Logs from other services
45. web server db memcached db http service user request log log log log log
46. Correlation / Tracing ID
47. web server db memcached db http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
48. X-Trace-Id: bbr8ehb984tbab894
52. Aggregate the logs in a central place
53. Make them easily full- text searchable
54. Make them aggregate- able
55. Always Log to file
56. Seriously...
57. Always Log to file
58. AND REMEMBER
59. Always Log to file
60. Directly to a database
61. webserver webserver webserver DB
62. Disadvantages
63. Database is down?
64. Database is slow?
65. Database is full?
66. Better solutions?
67. graylog2 http://graylog2.org/
68. Full text search
69. Structured Messages
70. Metrics & Alarms
71. Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver
72. { "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
73. Disadvantages
74. Graylog/elasticsearch is down?
75. Graylog/elasticsearch is full?
76. Packet loss
77. Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF
79. Don't influence your application by logging
80. logstash http://logstash.net/
81. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash logstash GELF
82. Very rich plugin system
83. input filter output
84. file { type => "error" path => [ "/var/logs/php/*.log" ] add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
85. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok { type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
86. output { amqp { host => "amqp.host" exchange_type => "fanout" name => "logs" } }
87. input { amqp { host => 'amqp.host' port => '5672' exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
88. output { gelf { host => '127.0.0.1' } }
89. Measuring
90. For knowing what happens
91. Munin http://munin-monitoring.org/
92. Ganglia http://ganglia.sourceforge.net/
94. Graphite http://graphite.wikidot.com/
95. webserver webserver webserver graphite UDP
96. webserver webserver webserver graphite UDP statsd
97. StatsD https://github.com/etsy/statsd/
98. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message
99. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message statsd
100. https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');
102. Technical Metrics
103. Business Metrics
104. Define your KPIs
105. Testing
106. For improving your app
107. Existing features
108. New features
109. A/B
110. BUY IT BUY IT $userId % 2 === 0 $userId = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
111. BUY IT BUY IT save random decision in session $userId = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
112. Save when experiment is shown
114. Save when a goal is reached
116. Conversion
117. Define your goals!
118. Disadvantages
119. Bad conversion during test
120. Only works for temporary tests
121. Inflexible: changing of tests not supported
122. There are better methods with better results
123. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/
124. BUY IT BUY IT random decision $userId = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
125. BUY IT BUY IT 2 90% users best option rest other options
126. Integrate it into your architecture
128. Be careful
129. https://github.com/bashofmann/ Try it out
130. Did you like this talk? h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann mail@bas2anhofmann.de http://programm.froscon.de/2012/ events/916.html