Metrics and an application log: Your new best friends

Do you remember the time you spent an afternoon putting print statements in your app trying to debug an issue and removed them before shipping the fix, only to add them back in a day later to work on another issue? Wouldn't it be great if those debug statements could just stay in your code forever? Like a little gift that keeps on giving, not just for you, but for everyone else on your team too.

That's what an application log is for! Logs aren't just for when things go wrong. They're for helping you to keep track of what's going on within your application.

We take a look at how you can add helpful messages throughout your codebase and leave them there, even in production! We'll cover common logging strategies, log aggregation and how to efficiently work with your logs to get the data back out again.

We'll also take a look at metrics solutions such as Graphite that can help augment your logs to help work out what was going on by correlating event logs with peaks/drops in other monitoring systems.

Transcript

1. Metrics and an application log Your new best friends
2. I’m Michael! • Developer at @DataSift • @mheap on Twitter • First time in Amsterdam!
3. External API API Internal
4. External API API API Internal
5. External API API API API Internal
6. External API API API API API API API API API API Internal
7. External API API API API API API API API API API API API API Internal
8. External API API API API API API API API API API API API API API API API Internal
9. External API API API API API API API API API API API API API API API API API API API API API API API API Internal
10. External API API API API API API API API API API API API API API API API API API API API API API API API API API API API Internal
11. External API API API API API API API API API API API API API API API API API API API API API API API API API API API API API API API Internal
12. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
13. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
14. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
15. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
16. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
17. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
18. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
19. What went wrong?
20. Who visited us?
21. Who changed the permissions of <X>?
22. Runtime documentation
23. I want to log things, I really do, but it’s a lot of effort
24. Can we have it for free?
26. But that doesn’t help with my app
27. Unfortunately, no, it doesn’t
28. We need an application log
29. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
30. Where do we log to?
31. A database?
32. External services?
33. File on disk
34. Consolidated error log
35. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
36. It depends on your application
37. What would help you at 3am?
38. Timestamp (When?)
39. Application (Where?)
40. User (Who?)
41. Action (What?)
42. Status (Result?)
43. Reason
44. Events
45. Emergency, Alert, Critical, Error, Warning, Notice, Informational, Debug
46. Audit logs
47. Audit logs: Notice
48. Connection logs
49. Connection logs: Info
50. Debug logs
51. Debug logs: Debug
52. New functionality
53. New functionality: Info
54. Layer transition
55. Layer transition: Info
56. User tracing
57. Request ID’s
58. Search for that ID
59. Return it to the user in errors
60. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
61. Log Handlers
62. Info => Application Log
63. Error => Syslog
64. Emergency => CTO’s Phone
65. Github
66. Fingers Crossed
67. Think of your ops team
68. Log everything in UTC
69. Changing log levels
70. Log rotation
71. Stack traces
72. But doesn't all this logging slow things down?
73. Imagine this
74. Fly slowly
75. Fly blind
76. Logs are the same
77. Isn't this contrived?
78. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
79. Logstash
80. 222.46.83.112 - - [21/Jan/2013:16:41:38 -0800] "GET / HTTP/1.1" 200 935 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
81. {"@source" "@tags" "@fields" => "file://example.com/var/httpd/access.log", => [], => { "clientip": [ "222.46.83.112" ], "ident": [ "-" ], "auth": [ "-" ], "timestamp": [ "21/Jan/2013:16:41:38 -0800" ], "verb": [ "GET" ], "request": [ "/" ], "httpversion": [ "1.1" ], "response": [ "200" ], "bytes": [ "935" ], "referrer": [ "\"-\"" ], "agent": [ "\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)\"" ] }, "@timestamp" "@source_host" "@source_path" "@message" 935 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)", "@type" => "web"}
82. Grok
83. Kibana
84. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
85. Trends
86. Graphite
87. What do we collect?
88. endpoint.<verb>.<controller>.<action>.<status_code>
89. endpoint.get.user.profile.200
90. StatsD::increment("event.name");
95. Supporting data
96. And more
97. Parting thoughts
98. Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics
99. My experiences
100. Your experiences?
101. Any questions?
102. Thankyou http://joind.in/11724