Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Measuring everything in Real-Time

Measuring everything in Real-Time

Talk from Froscon

Bastian Hofmann

August 25, 2012
Tweet

More Decks by Bastian Hofmann

Other Decks in Programming

Transcript

  1. Measuring everything in Real-Time @BastianHofmann

  2. Logging

  3. Measuring

  4. Testing

  5. many roads

  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. Questions? Ask!

  15. http://speakerdeck.com/u/bastianhofmann

  16. Logging

  17. For when something goes wrong

  18. server error log access log debug logs slow query log

    ...
  19. Error Logs

  20. $ tail -f error.log $ grep

  21. ErrorLog /var/logs/apache/error.log

  22. php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log

  23. callable set_exception_handler( callable $exception_handler );

  24. callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |

    E_STRICT ] );
  25. Log additional info

  26. Error code

  27. None
  28. HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

  29. None
  30. Access Logs

  31. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"

    200 882 "http://devm/rg_trunk/webroot/ directory/publications/"
  32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog

    /var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat
  33. http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =

    "" ] )
  34. LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{server_ip}n ..." custom nginx?

  35. Debug Logs

  36. https://github.com/Seldaek/monolog/

  37. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel

    $log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');
  38. Handlers • Stream • Mail • FirePHP • ChromePHP •

    Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed
  39. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );
  40. Log in a structured way

  41. JSON http://www.ietf.org/rfc/rfc4627.txt

  42. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);
  43. http://getcomposer.org/ http://packagist.org/

  44. Logs from other services

  45. web server db memcached db http service user request log

    log log log log
  46. Correlation / Tracing ID

  47. web server db memcached db http service create unique trace_id

    for request user request trace_id trace_id trace_id trace_id log log log log log
  48. X-Trace-Id: bbr8ehb984tbab894

  49. None
  50. None
  51. None
  52. Aggregate the logs in a central place

  53. Make them easily full- text searchable

  54. Make them aggregate- able

  55. Always Log to file

  56. Seriously...

  57. Always Log to file

  58. AND REMEMBER

  59. Always Log to file

  60. Directly to a database

  61. webserver webserver webserver DB

  62. Disadvantages

  63. Database is down?

  64. Database is slow?

  65. Database is full?

  66. Better solutions?

  67. graylog2 http://graylog2.org/

  68. Full text search

  69. Structured Messages

  70. Metrics & Alarms

  71. Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver

  72. { "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace

    here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }
  73. Disadvantages

  74. Graylog/elasticsearch is down?

  75. Graylog/elasticsearch is full?

  76. Packet loss

  77. Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF

  78. None
  79. Don't influence your application by logging

  80. logstash http://logstash.net/

  81. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash logstash GELF
  82. Very rich plugin system

  83. input filter output

  84. file { type => "error" path => [ "/var/logs/php/*.log" ]

    add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }
  85. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {

    type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }
  86. output { amqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }
  87. input { amqp { host => 'amqp.host' port => '5672'

    exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }
  88. output { gelf { host => '127.0.0.1' } }

  89. Measuring

  90. For knowing what happens

  91. Munin http://munin-monitoring.org/

  92. Ganglia http://ganglia.sourceforge.net/

  93. None
  94. Graphite http://graphite.wikidot.com/

  95. webserver webserver webserver graphite UDP

  96. webserver webserver webserver graphite UDP statsd

  97. StatsD https://github.com/etsy/statsd/

  98. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

  99. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

    statsd
  100. https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');

  101. None
  102. Technical Metrics

  103. Business Metrics

  104. Define your KPIs

  105. Testing

  106. For improving your app

  107. Existing features

  108. New features

  109. A/B

  110. BUY IT BUY IT $userId % 2 === 0 $userId

    = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
  111. BUY IT BUY IT save random decision in session $userId

    = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown
  112. Save when experiment is shown

  113. None
  114. Save when a goal is reached

  115. None
  116. Conversion

  117. Define your goals!

  118. Disadvantages

  119. Bad conversion during test

  120. Only works for temporary tests

  121. Inflexible: changing of tests not supported

  122. There are better methods with better results

  123. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/

  124. BUY IT BUY IT random decision $userId = 24 save

    that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1
  125. BUY IT BUY IT 2 90% users best option rest

    other options
  126. Integrate it into your architecture

  127. None
  128. Be careful

  129. https://github.com/bashofmann/ Try it out

  130. Did you like this talk? h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann

    mail@bas2anhofmann.de http://programm.froscon.de/2012/ events/916.html