Measuring everything in Real-Time

Talk from Froscon

Transcript

1. Measuring everything
   in Real-Time
   @BastianHofmann
   

   View Slide

2. Logging
   

   View Slide

3. Measuring
   

   View Slide

4. Testing
   

   View Slide

5. many roads
   

   View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. Questions?
    Ask!
    

    View Slide

15. http://speakerdeck.com/u/bastianhofmann
    

    View Slide

16. Logging
    

    View Slide

17. For when something
    goes wrong
    

    View Slide

18. server
    error log
    access log
    debug logs
    slow query log
    ...
    

    View Slide

19. Error Logs
    

    View Slide

20. $ tail -f error.log
    $ grep
    

    View Slide

21. ErrorLog /var/logs/apache/error.log
    

    View Slide

22. php.ini
    •error_reporting
    •display_errors
    •display_startup_errors
    •log_errors
    •error_log
    

    View Slide

23. callable set_exception_handler(
    callable $exception_handler
    );
    

    View Slide

24. callable set_error_handler(
    callable $error_handler
    [, int $error_types = E_ALL |
    E_STRICT ]
    );
    

    View Slide

25. Log additional info
    

    View Slide

26. Error code
    

    View Slide

27. View Slide

28. HTTP Response Codes
    http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html
    

    View Slide

29. View Slide

30. Access Logs
    

    View Slide

31. 192.168.56.1 - - [09/Jul/2012:19:18:19
    +0200] "GET /rg_trunk/webroot/c/af10c/
    images/template/rg_logo_default.png HTTP/
    1.1" 200 882 "http://devm/rg_trunk/webroot/
    directory/publications/"
    

    View Slide

32. LogFormat "%h %l %u %t \"%r\" %>s %b" custom
    CustomLog /var/logs/apache/access.log custom
    http://httpd.apache.org/docs/2.2/mod/
    mod_log_config.html#logformat
    

    View Slide

33. http://de.php.net/apache_note
    string apache_note (
    string $note_name
    [, string $note_value = "" ]
    )
    

    View Slide

34. LogFormat "...\"%{referer}i\" \"%{user-
    agent}i\" %{server_ip}n ..." custom nginx?
    

    View Slide

35. Debug Logs
    

    View Slide

36. https://github.com/Seldaek/monolog/
    

    View Slide

37. use Monolog\Logger;
    use Monolog\Handler\StreamHandler;
    // create a log channel
    $log = new Logger('name');
    $log->pushHandler(
    new StreamHandler(
    'path/to/your.log', Logger::WARNING
    )
    );
    // add records to the log
    $log->addWarning('Foo');
    $log->addError('Bar');
    

    View Slide

38. Handlers
    • Stream
    • Mail
    • FirePHP
    • ChromePHP
    • Socket
    • Roating File
    • MongoDB
    • Syslog
    • Gelf
    • Null
    • Test
    • FingersCrossed
    

    View Slide

39. use Monolog\Logger;
    use Monolog\Handler\StreamHandler;
    use Monolog\Processor\MemoryUsageProcessor;
    $log = new Logger('name');
    $handler = new StreamHandler(
    'path/to/your.log', Logger::WARNING
    );
    $log->pushHandler($handler);
    $log->pushProcessor(
    new MemoryUsageProcessor()
    );
    

    View Slide

40. Log in a structured way
    

    View Slide

41. JSON
    http://www.ietf.org/rfc/rfc4627.txt
    

    View Slide

42. use Monolog\Logger;
    use Monolog\Handler\StreamHandler;
    use Monolog\Formatter\JsonFormatter;
    $log = new Logger('name');
    $handler = new StreamHandler(
    'path/to/your.log', Logger::WARNING
    );
    $handler->setFormatter(new JsonFormatter());
    $log->pushHandler($handler);
    

    View Slide

43. http://getcomposer.org/
    http://packagist.org/
    

    View Slide

44. Logs from other
    services
    

    View Slide

45. web server db
    memcached
    db
    http service
    user request
    log
    log
    log
    log
    log
    

    View Slide

46. Correlation / Tracing ID
    

    View Slide

47. web server db
    memcached
    db
    http service
    create
    unique
    trace_id for
    request
    user request
    trace_id
    trace_id
    trace_id
    trace_id
    log
    log
    log
    log
    log
    

    View Slide

48. X-Trace-Id: bbr8ehb984tbab894
    

    View Slide

49. View Slide

50. View Slide

51. View Slide

52. Aggregate the logs in a
    central place
    

    View Slide

53. Make them easily full-
    text searchable
    

    View Slide

54. Make them aggregate-
    able
    

    View Slide

55. Always Log to file
    

    View Slide

56. Seriously...
    

    View Slide

57. Always Log to file
    

    View Slide

58. AND REMEMBER
    

    View Slide

59. Always Log to file
    

    View Slide

60. Directly to a database
    

    View Slide

61. webserver webserver webserver
    DB
    

    View Slide

62. Disadvantages
    

    View Slide

63. Database is down?
    

    View Slide

64. Database is slow?
    

    View Slide

65. Database is full?
    

    View Slide

66. Better solutions?
    

    View Slide

67. graylog2
    http://graylog2.org/
    

    View Slide

68. Full text search
    

    View Slide

69. Structured Messages
    

    View Slide

70. Metrics & Alarms
    

    View Slide

71. Graylog2
    UDP GELF Messages
    elasticsearch
    webserver webserver webserver
    

    View Slide

72. {
    "version": "1.0",
    "host": "www1",
    "short_message": "Short message",
    "full_message": "Backtrace here\n
    \nmore stuff",
    "timestamp": 1291899928.412,
    "level": 1,
    "facility": "payment-backend",
    "file": "/var/www/somefile.rb",
    "line": 356,
    "_user_id": 42,
    "_something_else": "foo"
    }
    

    View Slide

73. Disadvantages
    

    View Slide

74. Graylog/elasticsearch is
    down?
    

    View Slide

75. Graylog/elasticsearch is
    full?
    

    View Slide

76. Packet loss
    

    View Slide

77. Graylog2
    elasticsearch
    webserver webserver webserver
    AMQP
    GELF GELF
    GELF
    GELF
    

    View Slide

78. View Slide

79. Don't influence your
    application by logging
    

    View Slide

80. logstash
    http://logstash.net/
    

    View Slide

81. Graylog2
    elasticsearch
    webserver webserver webserver
    AMQP
    log log log
    logstash logstash logstash
    logstash
    GELF
    

    View Slide

82. Very rich plugin system
    

    View Slide

83. input filter output
    

    View Slide

84. file {
    type => "error"
    path =>
    [ "/var/logs/php/*.log" ]
    add_field =>
    [ "severity", "error" ]
    }
    file {
    type => "access"
    path =>
    [ "/var/logs/apache/*_access.log" ]
    add_field =>
    [ "severity", "info" ]
    }
    

    View Slide

85. filter{
    grok {
    match => ["@source",
    "\/%{USERNAME:facility}\.log$"]
    }
    grok {
    type => "access"
    pattern => "^%{IP:OriginalIp}
    \s[a-zA-Z0-9_-]+\s[a-zA-Z0-9_-]+\s\[.*?
    \]\s\"%{DATA:Request}..."
    }
    }
    

    View Slide

86. output {
    amqp {
    host => "amqp.host"
    exchange_type => "fanout"
    name => "logs"
    }
    }
    

    View Slide

87. input {
    amqp {
    host => 'amqp.host'
    port => '5672'
    exchange => 'logs'
    name => 'logs'
    type => 'logs'
    exclusive => false
    }
    }
    

    View Slide

88. output {
    gelf {
    host => '127.0.0.1'
    }
    }
    

    View Slide

89. Measuring
    

    View Slide

90. For knowing what
    happens
    

    View Slide

91. Munin
    http://munin-monitoring.org/
    

    View Slide

92. Ganglia
    http://ganglia.sourceforge.net/
    

    View Slide

93. View Slide

94. Graphite
    http://graphite.wikidot.com/
    

    View Slide

95. webserver webserver webserver
    graphite
    UDP
    

    View Slide

96. webserver webserver webserver
    graphite
    UDP statsd
    

    View Slide

97. StatsD
    https://github.com/etsy/statsd/
    

    View Slide

98. webserver webserver webserver
    statsd statsd
    statsd
    graphite
    aggregated
    UPD message
    

    View Slide

99. webserver webserver webserver
    statsd statsd
    statsd
    graphite
    aggregated
    UPD message
    statsd
    

    View Slide

100. https://github.com/etsy/statsd/blob/master/examples/php-
     example.php
     StatsD::increment('key');
     

     View Slide

101. View Slide

102. Technical Metrics
     

     View Slide

103. Business Metrics
     

     View Slide

104. Define your KPIs
     

     View Slide

105. Testing
     

     View Slide

106. For improving your app
     

     View Slide

107. Existing features
     

     View Slide

108. New features
     

     View Slide

109. A/B
     

     View Slide

110. BUY IT BUY IT
     $userId % 2 === 0
     $userId = 24
     save that option A was clicked
     save that option A is shown
     conversion = option A clicked / option A shown
     

     View Slide

111. BUY IT BUY IT
     save random decision
     in session
     $userId = 24
     save that option A was clicked
     save that option A is shown
     conversion = option A clicked / option A shown
     

     View Slide

112. Save when experiment
     is shown
     

     View Slide

113. View Slide

114. Save when a goal is
     reached
     

     View Slide

115. View Slide

116. Conversion
     

     View Slide

117. Define your goals!
     

     View Slide

118. Disadvantages
     

     View Slide

119. Bad conversion during
     test
     

     View Slide

120. Only works for
     temporary tests
     

     View Slide

121. Inflexible: changing of
     tests not supported
     

     View Slide

122. There are better
     methods with better
     results
     

     View Slide

123. Bandit Algorithm
     http://untyped.com/untyping/2011/02/11/stop-ab-testing-
     and-make-out-like-a-bandit/
     

     View Slide

124. BUY IT BUY IT
     random decision
     $userId = 24
     save that option A was clicked
     save that option A is shown
     conversion = option A clicked / option A shown
     1
     

     View Slide

125. BUY IT BUY IT
     2
     90% users best option rest other options
     

     View Slide

126. Integrate it into your
     architecture
     

     View Slide

127. View Slide

128. Be careful
     

     View Slide

129. https://github.com/bashofmann/
     Try it out
     

     View Slide

130. Did you like this talk?
     h"p://twi"er.com/Bas2anHofmann
     h"p://profiles.google.com/bashofmann
     h"p://lanyrd.com/people/Bas2anHofmann
     h"p://speakerdeck.com/u/bas2anhofmann
     h"ps://github.com/bashofmann
     mail@bas2anhofmann.de
     http://programm.froscon.de/2012/
     events/916.html
     

     View Slide