Logging: Your new best friend

Logs are not just used when things go wrong. They also help you keep track of what is going on within your app. We will look at how you can add helpful messages throughout your codebase and leave them there, even in production! We will cover common logging strategies, log aggregation, and how to efficiently work with your logs to get the data back out. We will also look at Graphite, which can help work out what actually happened by correlating logs with peaks/drops in other systems.

Transcript

1. Logging: Your new best friend Michael Heap (@mheap) Developer at DataSift Presented at php|tek, May 2016
2. Me! I’m Michael I’m @mheap Developer at DataSift
3. @mheap https://joind.in/17042
4. Logging
5. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
6. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
7. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
8. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
9. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
10. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
11. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
12. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
13. Sound good?
14. Good!
15. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
16. Why log?
17. What went wrong? (Error log)
18. Who visited us? (Access log)
19. Who enabled <X>? (Audit log)
20. Runtime documentation (Application log)
21. I’m sold!
22. Can I have it for free?
23. Actually, yes!
24. (And more)
25. But that doesn’t help my application
26. Two types of log
27. Human readable
28. Machine readable
29. We should log both
30. What is an application log?
31. Debug information
32. Narrative information
33. Business information
34. “An application log signposts every twist and turn through the code”
35. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
36. Four W’s
37. When?
38. Who?
39. Where?
40. Why?
41. Getting started
42. error_log()
43. <?php function countConsonants($str){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); error_log("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael");
44. <?php function countConsonants($str){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); error_log("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael");
45. <?php ini_set("error_log", "/var/log/casino-app.log"); function countConsonants($str){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); error_log("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael");
46. Pros ✴ It’s built in
47. Cons ✴ Is it semantically correct? ✴ Errors mixed with informational logs ✴ It’s not very powerful
48. Logging frameworks
49. 1) Monolog 2) Everything else
50. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('casino-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
51. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('casino-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
52. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('casino-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
53. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('casino-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
54. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('casino-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
55. [2016-05-25 03:56:01] casino-app.INFO: Consonants in Michael: 4 [] []
56. FingersCrossedHandler
57. $log = new Monolog\Logger('casino-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG); $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
58. $log = new Monolog\Logger('casino-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG); $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
59. $log = new Monolog\Logger('casino-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG); $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
60. $log = new Monolog\Logger('casino-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG); $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); $log->error("Something bad happened"); return $c; } echo countConsonants("Michael", $log);
61. Pros ✴ It’s an object! Dependency injection FTW ✴ Supports multiple log writers ✴ Log level support
62. Cons ✴ Instantiating an instance can be complicated
63. Error Levels
64. 0. Emergency System is unusable 1. Alert Should be corrected immediately 2. Critical Critical conditions 3. Error Error conditions 4. Warning May indicate that an error will occur if action is not taken. 5. Notice Events that are unusual, but not error conditions. 6. Informational Normal operational messages that require no action. 7. Debug Information useful to developers for debugging the application. Syslog (RFC 5424)
65. 0. Emergency System is unusable 1. Alert Should be corrected immediately 2. Critical Critical conditions 3. Error Error conditions 4. Warning May indicate that an error will occur if action is not taken. 5. Notice Events that are unusual, but not error conditions. 6. Informational Normal operational messages that require no action. 7. Debug Information useful to developers for debugging the application. PSR3
66. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
67. Everything is on fire
68. The ELK Stack
69. Elasticsearch Logstash Kibana
70. Logstash Elasticsearch Kibana
71. Logstash
72. Beats CouchDB_Changes Drupal_DBLog Elasticsearch Exec Event log File Ganglia Gelf Generator Graphite Github Heartbeat Heroku HTTP HTTP_Poller IRC IMAP JDBC JMX
 Kafka Log4J Lumberjack Meetup Pipe Puppet_Facter Relp RSS Backspace RabbitMQ Redis Salesforce SNMPTrap Stdin sqlite S3 SQS Stomp Syslog TCP Twitter Unix UDP Varnishlog WMI Web socket XMPP Zenoss ZeroMQ Inputs
73. Filters
74. filter { json { source => "message" add_field => [ “my_field", "tek_%{host}" ] } }
75. filter { kv { default_keys => [ "from", "logstash@example.com", "to", "default@dev.null" ] } }
76. Accepted publickey for root from 172.14.183.11 port 22 ssh2
77. Accepted publickey for root from 172.14.183.11 port 22 ssh2
78. filter { grok { match => { "message" => "Accepted %{WORD:auth_method} for %{USER:username} from %{IP:src_ip} port %{INT:src_port} ssh2" } } } Accepted publickey for root from 172.14.183.11 port 22 ssh2
79. filter { grok { match => { "message" => "Accepted %{WORD:auth_method} for %{USER:username} from %{IP:src_ip} port %{INT:src_port} ssh2" } } } Accepted publickey for root from 172.14.183.11 port 22 ssh2
80. http://grokdebug.herokuapp.com/
81. Boundary Circus CSV Cloud watch Datadog Datadog_Metrics Email Elastic search Exec File Google BigQuery Google Cloud Storage Ganglia Gelf Graphtastic Graphite Hipchat HTTP IRC InfluxDB Juggernaut Jira Kafka Lumberjack Librato Loggly MongoDB MetricCatcher Nagios Null OpenTSDB Pagerduty Pipe Riemann Redmine Rackspace RabbitMQ Redis Riak S3 SQS Stomp StatsD Solr SNS Syslog Stdout TCP UDP WebHDFS Websocket XMPP Outputs Zabbix ZeroMQ
82. Input -> Filter -> Output
83. Logstash is slow(ish)
84. Elasticsearch
85. Kibana
86. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
91. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
92. Asimov’s Law
93. “A robot may not injure a human being or, through inaction, allow a human being to come to harm.”
94. @mheap’s Law
95. “An application log may not injure a an application’s performance or readability”
96. Plan for bursts of data
97. Disk space
98. Index management
99. Ship what’s relevant
100. Devs create dashboards
101. Unique request IDs
102. Normalise timezones
103. No really. Normalise timezones
104. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
105. Beats
106. Graphite
109. Pagerduty
110. Elastalert
111. 1. Logging 2. Getting started 3. The ELK stack 4. Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
112. Logging is required
113. Developers are empowered
114. Use PSR-3
115. Logging isn’t free
116. “Would you rather fly slowly or fly blind?”
117. Thanks! I’ve been @mheap, you’ve been awesome. Please leave feedback on Joind.in https://joind.in/17042