Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Measuring and Logging everything in Real-Time

Measuring and Logging everything in Real-Time

Web Developer Conference 2012, Hamburg

8e82eb7e128a14a16d642ae55227339b?s=128

Bastian Hofmann

September 18, 2012
Tweet

Transcript

  1. Measuring and Logging everything in Real-Time @BastianHofmann

  2. Logging

  3. Measuring

  4. Testing

  5. many roads

  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. Questions? Ask!

  15. http://speakerdeck.com/u/bastianhofmann

  16. Logging

  17. For when something goes wrong

  18. server error log access log debug logs slow query log

    ...
  19. Error Logs

  20. $ tail -f error.log $ grep

  21. ErrorLog /var/logs/apache/error.log

  22. php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log

  23. callable set_exception_handler( callable $exception_handler );

  24. callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |

    E_STRICT ] );
  25. Log additional info

  26. Error code

  27. None
  28. HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

  29. None
  30. Access Logs

  31. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"

    200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
  32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog

    /var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
  33. http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =

    "" ] )
  34. LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{session_id}n %{account_id}n..." custom nginx?

  35. Debug Logs

  36. https://github.com/Seldaek/monolog/

  37. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel

    $log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
  38. Handlers • Stream • Mail • FirePHP • ChromePHP •

    Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
  39. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
  40. Log in a structured way

  41. JSON http://www.ietf.org/rfc/rfc4627.txt

  42. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
  43. http://getcomposer.org/ http://packagist.org/

  44. $ cat composer.json

  45. $ wget http://getcomposer.org/composer.phar $ php composer.phar install Installing dependencies -

    Installing monolog/monolog (1.1.0) Downloading: 100% ... monolog/monolog suggests installing mlehner/ gelf-php (Allow sending log messages to a GrayLog2 server) Writing lock file Generating autoload files
  46. Logs from other services

  47. web server db memcached db http service user request log

    log log log log
  48. Correlation / Tracing ID

  49. web server db memcached db http service create unique trace_id

    for request user request trace_id trace_id trace_id trace_id log log log log log
  50. X-Trace-Id: bbr8ehb984tbab894

  51. None
  52. None
  53. None
  54. Aggregate the logs in a central place

  55. Make them easily full- text searchable

  56. Make them aggregate- able

  57. Always Log to file

  58. Seriously...

  59. Always Log to file

  60. AND REMEMBER

  61. Always Log to file

  62. Directly to a database

  63. webserver webserver webserver DB

  64. Disadvantages

  65. Database is down?

  66. Database is slow?

  67. Database is full?

  68. Better solutions?

  69. graylog2 http://graylog2.org/

  70. Full text search

  71. Structured Messages

  72. Metrics & Alarms

  73. Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver

  74. { "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace

    here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
  75. Disadvantages

  76. Graylog/elasticsearch is down?

  77. Graylog/elasticsearch is full?

  78. Packet loss

  79. Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF

  80. None
  81. Don't influence your application by logging

  82. logstash http://logstash.net/

  83. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash logstash GELF
  84. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash
  85. Very rich plugin system

  86. input filter output

  87. file { type => "error" path => [ "/var/logs/php/*.log" ]

    add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
  88. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {

    type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
  89. output { amqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }
  90. output { gelfamqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }
  91. input { amqp { host => 'amqp.host' port => '5672'

    exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
  92. output { gelf { host => '127.0.0.1' } }

  93. Measuring

  94. For knowing what happens

  95. Munin http://munin-monitoring.org/

  96. Ganglia http://ganglia.sourceforge.net/

  97. None
  98. Graphite http://graphite.wikidot.com/

  99. webserver webserver webserver graphite UDP

  100. webserver webserver webserver graphite UDP statsd

  101. StatsD https://github.com/etsy/statsd/

  102. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

  103. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

    statsd
  104. https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');

  105. None
  106. https://metrics.librato.com/

  107. Technical Metrics

  108. Business Metrics

  109. Define your KPIs

  110. http://mashable.com/2010/10/13/etsy-office-pics/

  111. Testing

  112. For improving your app

  113. Existing features

  114. New features

  115. A/B

  116. BUY IT BUY IT $userId % 2 === 0 $userId

    = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
  117. BUY IT BUY IT save random decision in session $userId

    = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
  118. Save when experiment is shown

  119. None
  120. Save when a goal is reached

  121. None
  122. Conversion

  123. Define your goals!

  124. Disadvantages

  125. Bad conversion during test

  126. Only works for temporary tests

  127. Inflexible: changing of tests not supported

  128. There are better methods with better results

  129. Feature Flags https://github.com/rcrowe/fuel-flags

  130. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/

  131. BUY IT BUY IT random decision $userId = 24 save

    that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
  132. BUY IT BUY IT 2 90% users best option rest

    other options
  133. Integrate it into your architecture

  134. None
  135. Be careful

  136. https://github.com/bashofmann/ Try it out

  137. h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann mail@bas2anhofmann.de