Measuring everything in Real-Time

Transcript

1. Measuring everything in Real-Time @BastianHofmann

2. Logging

3. Measuring

4. Testing

5. many roads

6. None
7. None
8. None
9. None
10. None
11. None
12. None
13. None

14. Questions? Ask!

15. http://speakerdeck.com/u/bastianhofmann

16. Logging

17. For when something goes wrong

18. server error log access log debug logs slow query log

    ...

19. Error Logs

20. $ tail -f error.log $ grep

21. ErrorLog /var/logs/apache/error.log

22. php.ini •error_reporting •display_errors •display_startup_errors •log_errors •error_log

23. callable set_exception_handler( callable $exception_handler );

24. callable set_error_handler( callable $error_handler [, int $error_types = E_ALL |

    E_STRICT ] );

25. Log additional info

26. Error code

27. None

28. HTTP Response Codes http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

29. None

30. Access Logs

31. 192.168.56.1 - - [09/Jul/2012:19:18:19 +0200] "GET /rg_trunk/webroot/c/af10c/ images/template/rg_logo_default.png HTTP/ 1.1"

    200 882 "http://devm/rg_trunk/webroot/ directory/publications/"

32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom CustomLog

    /var/logs/apache/access.log custom http://httpd.apache.org/docs/2.2/mod/ mod_log_config.html#logformat

33. http://de.php.net/apache_note string apache_note ( string $note_name [, string $note_value =

    "" ] )

34. LogFormat "...\"%{referer}i\" \"%{user- agent}i\" %{server_ip}n ..." custom nginx?

35. Debug Logs

36. https://github.com/Seldaek/monolog/

37. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; // create a log channel

    $log = new Logger('name'); $log->pushHandler( new StreamHandler( 'path/to/your.log', Logger::WARNING ) ); // add records to the log $log->addWarning('Foo'); $log->addError('Bar');

38. Handlers • Stream • Mail • FirePHP • ChromePHP •

    Socket • Roating File • MongoDB • Syslog • Gelf • Null • Test • FingersCrossed

39. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Processor\MemoryUsageProcessor; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $log->pushHandler($handler); $log->pushProcessor( new MemoryUsageProcessor() );

40. Log in a structured way

41. JSON http://www.ietf.org/rfc/rfc4627.txt

42. <?php use Monolog\Logger; use Monolog\Handler\StreamHandler; use Monolog\Formatter\JsonFormatter; $log = new

    Logger('name'); $handler = new StreamHandler( 'path/to/your.log', Logger::WARNING ); $handler->setFormatter(new JsonFormatter()); $log->pushHandler($handler);

43. http://getcomposer.org/ http://packagist.org/

44. Logs from other services

45. web server db memcached db http service user request log

    log log log log

46. Correlation / Tracing ID

47. web server db memcached db http service create unique trace_id

    for request user request trace_id trace_id trace_id trace_id log log log log log

48. X-Trace-Id: bbr8ehb984tbab894

49. None
50. None
51. None

52. Aggregate the logs in a central place

53. Make them easily full- text searchable

54. Make them aggregate- able

55. Always Log to file

56. Seriously...

57. Always Log to file

58. AND REMEMBER

59. Always Log to file

60. Directly to a database

61. webserver webserver webserver DB

62. Disadvantages

63. Database is down?

64. Database is slow?

65. Database is full?

66. Better solutions?

67. graylog2 http://graylog2.org/

68. Full text search

69. Structured Messages

70. Metrics & Alarms

71. Graylog2 UDP GELF Messages elasticsearch webserver webserver webserver

72. { "version": "1.0", "host": "www1", "short_message": "Short message", "full_message": "Backtrace

    here\n \nmore stuff", "timestamp": 1291899928.412, "level": 1, "facility": "payment-backend", "file": "/var/www/somefile.rb", "line": 356, "_user_id": 42, "_something_else": "foo" }

73. Disadvantages

74. Graylog/elasticsearch is down?

75. Graylog/elasticsearch is full?

76. Packet loss

77. Graylog2 elasticsearch webserver webserver webserver AMQP GELF GELF GELF GELF

78. None

79. Don't influence your application by logging

80. logstash http://logstash.net/

81. Graylog2 elasticsearch webserver webserver webserver AMQP log log log logstash

    logstash logstash logstash GELF

82. Very rich plugin system

83. input filter output

84. file { type => "error" path => [ "/var/logs/php/*.log" ]

    add_field => [ "severity", "error" ] } file { type => "access" path => [ "/var/logs/apache/*_access.log" ] add_field => [ "severity", "info" ] }

85. filter{ grok { match => ["@source", "\/%{USERNAME:facility}\.log$"] } grok {

    type => "access" pattern => "^%{IP:OriginalIp} \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*? \]\s\"%{DATA:Request}..." } }

86. output { amqp { host => "amqp.host" exchange_type => "fanout"

    name => "logs" } }

87. input { amqp { host => 'amqp.host' port => '5672'

    exchange => 'logs' name => 'logs' type => 'logs' exclusive => false } }

88. output { gelf { host => '127.0.0.1' } }

89. Measuring

90. For knowing what happens

91. Munin http://munin-monitoring.org/

92. Ganglia http://ganglia.sourceforge.net/

93. None

94. Graphite http://graphite.wikidot.com/

95. webserver webserver webserver graphite UDP

96. webserver webserver webserver graphite UDP statsd

97. StatsD https://github.com/etsy/statsd/

98. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

99. webserver webserver webserver statsd statsd statsd graphite aggregated UPD message

    statsd

100. https://github.com/etsy/statsd/blob/master/examples/php- example.php StatsD::increment('key');

101. None

102. Technical Metrics

103. Business Metrics

104. Define your KPIs

105. Testing

106. For improving your app

107. Existing features

108. New features

109. A/B

110. BUY IT BUY IT $userId % 2 === 0 $userId

     = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown

111. BUY IT BUY IT save random decision in session $userId

     = 24 save that option A was clicked save that option A is shown conversion = option A clicked / option A shown

112. Save when experiment is shown

113. None

114. Save when a goal is reached

115. None

116. Conversion

117. Define your goals!

118. Disadvantages

119. Bad conversion during test

120. Only works for temporary tests

121. Inflexible: changing of tests not supported

122. There are better methods with better results

123. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/

124. BUY IT BUY IT random decision $userId = 24 save

     that option A was clicked save that option A is shown conversion = option A clicked / option A shown 1

125. BUY IT BUY IT 2 90% users best option rest

     other options

126. Integrate it into your architecture

127. None

128. Be careful

129. https://github.com/bashofmann/ Try it out

130. Did you like this talk? h"p://twi"er.com/Bas2anHofmann h"p://profiles.google.com/bashofmann h"p://lanyrd.com/people/Bas2anHofmann h"p://speakerdeck.com/u/bas2anhofmann h"ps://github.com/bashofmann

     [email protected] http://programm.froscon.de/2012/ events/916.html