Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Metrics and an application log: Your new best friends

Michael Heap
September 05, 2014

Metrics and an application log: Your new best friends

Do you remember the time you spent an afternoon putting print statements in your app trying to debug an issue and removed them before shipping the fix, only to add them back in a day later to work on another issue? Wouldn't it be great if those debug statements could just stay in your code forever? Like a little gift that keeps on giving, not just for you, but for everyone else on your team too.

That's what an application log is for! Logs aren't just for when things go wrong. They're for helping you to keep track of what's going on within your application.

We take a look at how you can add helpful messages throughout your codebase and leave them there, even in production! We'll cover common logging strategies, log aggregation and how to efficiently work with your logs to get the data back out again.

We'll also take a look at metrics solutions such as Graphite that can help augment your logs to help work out what was going on by correlating event logs with peaks/drops in other monitoring systems.

Michael Heap

September 05, 2014
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. Metrics and an application log
    Your new best friends

    View Slide

  2. I’m Michael!
    • Developer at @DataSift
    • @mheap on Twitter
    • First time in Amsterdam!

    View Slide

  3. External
    API
    API
    Internal

    View Slide

  4. External
    API
    API
    API
    Internal

    View Slide

  5. External
    API
    API
    API
    API
    Internal

    View Slide

  6. External
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    Internal

    View Slide

  7. External
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    Internal

    View Slide

  8. External
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    Internal

    View Slide

  9. External
    API
    API API API
    API API API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    Internal

    View Slide

  10. External
    API
    API API API
    API API API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    Internal

    View Slide

  11. External
    API
    API API API
    API API API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API
    API API
    API
    API
    API
    API
    API
    API
    Internal

    View Slide

  12. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  13. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  14. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  15. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  16. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  17. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  18. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  19. What went wrong?

    View Slide

  20. Who visited us?

    View Slide

  21. Who changed the permissions of ?

    View Slide

  22. Runtime documentation

    View Slide

  23. I want to log things, I really do,
    but it’s a lot of effort

    View Slide

  24. Can we have it for free?

    View Slide

  25. View Slide

  26. But that doesn’t help with my app

    View Slide

  27. Unfortunately, no, it doesn’t

    View Slide

  28. We need an application log

    View Slide

  29. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  30. Where do we log to?

    View Slide

  31. A database?

    View Slide

  32. External services?

    View Slide

  33. File on disk

    View Slide

  34. Consolidated error log

    View Slide

  35. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  36. It depends on your application

    View Slide

  37. What would help you at 3am?

    View Slide

  38. Timestamp (When?)

    View Slide

  39. Application (Where?)

    View Slide

  40. User (Who?)

    View Slide

  41. Action (What?)

    View Slide

  42. Status (Result?)

    View Slide

  43. Reason

    View Slide

  44. Events

    View Slide

  45. Emergency, Alert, Critical, Error,
    Warning, Notice, Informational, Debug

    View Slide

  46. Audit logs

    View Slide

  47. Audit logs: Notice

    View Slide

  48. Connection logs

    View Slide

  49. Connection logs: Info

    View Slide

  50. Debug logs

    View Slide

  51. Debug logs: Debug

    View Slide

  52. New functionality

    View Slide

  53. New functionality: Info

    View Slide

  54. Layer transition

    View Slide

  55. Layer transition: Info

    View Slide

  56. User tracing

    View Slide

  57. Request ID’s

    View Slide

  58. Search for that ID

    View Slide

  59. Return it to the user in errors

    View Slide

  60. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  61. Log Handlers

    View Slide

  62. Info => Application Log

    View Slide

  63. Error => Syslog

    View Slide

  64. Emergency => CTO’s Phone

    View Slide

  65. Github

    View Slide

  66. Fingers Crossed

    View Slide

  67. Think of your ops team

    View Slide

  68. Log everything in UTC

    View Slide

  69. Changing log levels

    View Slide

  70. Log rotation

    View Slide

  71. Stack traces

    View Slide

  72. But doesn't all this logging slow things down?

    View Slide

  73. Imagine this

    View Slide

  74. Fly slowly

    View Slide

  75. Fly blind

    View Slide

  76. Logs are the same

    View Slide

  77. Isn't this contrived?

    View Slide

  78. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  79. Logstash

    View Slide

  80. 222.46.83.112 - - [21/Jan/2013:16:41:38 -0800] "GET / HTTP/1.1"
    200 935 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

    View Slide

  81. {"@source" "@tags" "@fields"
    => "file://example.com/var/httpd/access.log", => [],
    => {
    "clientip": [ "222.46.83.112" ],
    "ident": [ "-" ],
    "auth": [ "-" ],
    "timestamp": [ "21/Jan/2013:16:41:38 -0800" ], "verb": [ "GET" ],
    "request": [ "/" ],
    "httpversion": [ "1.1" ],
    "response": [ "200" ],
    "bytes": [ "935" ],
    "referrer": [ "\"-\"" ],
    "agent": [ "\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT
    6.0)\"" ]
    },
    "@timestamp"
    "@source_host"
    "@source_path"
    "@message"
    935 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
    "@type" => "web"}

    View Slide

  82. Grok

    View Slide

  83. Kibana

    View Slide

  84. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  85. Trends

    View Slide

  86. Graphite

    View Slide

  87. What do we collect?

    View Slide

  88. endpoint....

    View Slide

  89. endpoint.get.user.profile.200

    View Slide

  90. StatsD::increment("event.name");

    View Slide

  91. View Slide

  92. View Slide

  93. View Slide

  94. View Slide

  95. Supporting data

    View Slide

  96. And more

    View Slide

  97. Parting thoughts

    View Slide

  98. Why do we need to log?
    How should we log it?
    What should we log?
    Logging considerations
    Searching logs
    Gathering metrics

    View Slide

  99. My experiences

    View Slide

  100. Your experiences?

    View Slide

  101. Any questions?

    View Slide

  102. Thankyou
    http://joind.in/11724

    View Slide