Measuring and Logging everything in Real-Time

Transcript

1. Measuring and Logging everything in Real-Time @BastianHofmann

2. Logging

3. Measuring

4. Testing

5. many roads

6. None
7. None
8. None
9. None
10. None
11. None
12. None
13. None

14. Questions? Ask!

15. http://speakerdeck.com/u/bastianhofmann

16. Logging

17. For when something goes wrong

18. server error log access log debug logs slow query log

    ...

19. Error Logs

20. $ tail -f error.log $ grep

21. ErrorLog /var/logs/apache/error.log

22. php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log

23. callable set_exception_handler( callable $exception_handler );

24. callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |

    E_STRICT ] );

25. Log additional info

26. Error code

27. None

28. HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

29. None

30. Access Logs

31. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"

    200 882 "http://devm/rg_trunk/webroot/ directory/publications/"

32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog

    /var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat

33. http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =

    "" ] )

34. LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{session_id}n %{account_id}n..." custom nginx?

35. Debug Logs

36. https://github.com/Seldaek/monolog/

37. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel

    $log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');

38. Handlers • Stream • Mail • FirePHP • ChromePHP •

    Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed

39. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );

40. Log in a structured way

41. JSON http://www.ietf.org/rfc/rfc4627.txt

42. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);

43. http://getcomposer.org/ http://packagist.org/

44. $ cat composer.json

45. $ wget http://getcomposer.org/composer.phar $ php composer.phar install Installing dependencies -

    Installing monolog/monolog (1.1.0) Downloading: 100% ... monolog/monolog suggests installing mlehner/ gelf-php (Allow sending log messages to a GrayLog2 server) Writing lock file Generating autoload files

46. Logs from other services

47. web server db memcached db http service user request log

    log log log log

48. Correlation / Tracing ID

49. web server db memcached db http service create unique trace_id

    for request user request trace_id trace_id trace_id trace_id log log log log log

50. X-Trace-Id: bbr8ehb984tbab894

51. None
52. None
53. None

54. Aggregate the logs in a central place

55. Make them easily full- text searchable

56. Make them aggregate- able

57. Always Log to file

58. Seriously...

59. Always Log to file

60. AND REMEMBER

61. Always Log to file

62. Directly to a database

63. webserver webserver webserver DB

64. Disadvantages

65. Database is down?

66. Database is slow?

67. Database is full?

68. Better solutions?

69. graylog2 http://graylog2.org/

70. Full text search

71. Structured Messages

72. Metrics & Alarms

73. Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver

74. { "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace

    here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }

75. Disadvantages

76. Graylog/elasticsearch is down?

77. Graylog/elasticsearch is full?

78. Packet loss

79. Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF

80. None

81. Don't influence your application by logging

82. logstash http://logstash.net/

83. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash logstash GELF

84. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash

85. Very rich plugin system

86. input filter output

87. file { type => "error" path => [ "/var/logs/php/*.log" ]

    add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }

88. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {

    type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }

89. output { amqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }

90. output { gelfamqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }

91. input { amqp { host => 'amqp.host' port => '5672'

    exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }

92. output { gelf { host => '127.0.0.1' } }

93. Measuring

94. For knowing what happens

95. Munin http://munin-monitoring.org/

96. Ganglia http://ganglia.sourceforge.net/

97. None

98. Graphite http://graphite.wikidot.com/

99. webserver webserver webserver graphite UDP

100. webserver webserver webserver graphite UDP statsd

101. StatsD https://github.com/etsy/statsd/

102. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

103. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

     statsd

104. https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');

105. None

106. https://metrics.librato.com/

107. Technical Metrics

108. Business Metrics

109. Define your KPIs

110. http://mashable.com/2010/10/13/etsy-office-pics/

111. Testing

112. For improving your app

113. Existing features

114. New features

115. A/B

116. BUY IT BUY IT $userId % 2 === 0 $userId

     = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown

117. BUY IT BUY IT save random decision in session $userId

     = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown

118. Save when experiment is shown

119. None

120. Save when a goal is reached

121. None

122. Conversion

123. Define your goals!

124. Disadvantages

125. Bad conversion during test

126. Only works for temporary tests

127. Inflexible: changing of tests not supported

128. There are better methods with better results

129. Feature Flags https://github.com/rcrowe/fuel-flags

130. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/

131. BUY IT BUY IT random decision $userId = 24 save

     that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1

132. BUY IT BUY IT 2 90% users best option rest

     other options

133. Integrate it into your architecture

134. None

135. Be careful

136. https://github.com/bashofmann/ Try it out

137. h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann [email protected]