Metrics and an application log: Your new best friends

by Michael Heap

Slide 1

Slide 1 text

Metrics and an application log Your new best friends

Slide 2

Slide 2 text

I’m Michael! • Developer at @DataSift • @mheap on Twitter • First time in Amsterdam!

Slide 3

Slide 3 text

External API API Internal

Slide 4

Slide 4 text

External API API API Internal

Slide 5

Slide 5 text

External API API API API Internal

Slide 6

Slide 6 text

External API API API API API API API API API API Internal

Slide 7

Slide 7 text

External API API API API API API API API API API API API API Internal

Slide 8

Slide 8 text

External API API API API API API API API API API API API API API API API Internal

Slide 9

Slide 9 text

External API API API API API API API API API API API API API API API API API API API API API API API API Internal

Slide 10

Slide 10 text

External API API API API API API API API API API API API API API API API API API API API API API API API API API API API Internal

Slide 11

Slide 11 text

External API API API API API API API API API API API API API API API API API API API API API API API API API API API API API API API Internal

Slide 12

Slide 12 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 13

Slide 13 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 14

Slide 14 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 15

Slide 15 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 16

Slide 16 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 17

Slide 17 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 18

Slide 18 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 19

Slide 19 text

What went wrong?

Slide 20

Slide 20 text

Who visited us?

Slide 21

Slide 21 text

Who changed the permissions of ?

Slide 22

Slide 22 text

Runtime documentation

Slide 23

Slide 23 text

I want to log things, I really do, but it’s a lot of effort

Slide 24

Slide 24 text

Can we have it for free?

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

But that doesn’t help with my app

Slide 27

Slide 27 text

Unfortunately, no, it doesn’t

Slide 28

Slide 28 text

We need an application log

Slide 29

Slide 29 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 30

Slide 30 text

Where do we log to?

Slide 31

Slide 31 text

A database?

Slide 32

Slide 32 text

External services?

Slide 33

Slide 33 text

File on disk

Slide 34

Slide 34 text

Consolidated error log

Slide 35

Slide 35 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 36

Slide 36 text

It depends on your application

Slide 37

Slide 37 text

What would help you at 3am?

Slide 38

Slide 38 text

Timestamp (When?)

Slide 39

Slide 39 text

Application (Where?)

Slide 40

Slide 40 text

User (Who?)

Slide 41

Slide 41 text

Action (What?)

Slide 42

Slide 42 text

Status (Result?)

Slide 43

Slide 43 text

Reason

Slide 44

Slide 44 text

Events

Slide 45

Slide 45 text

Emergency, Alert, Critical, Error, Warning, Notice, Informational, Debug

Slide 46

Slide 46 text

Audit logs

Slide 47

Slide 47 text

Audit logs: Notice

Slide 48

Slide 48 text

Connection logs

Slide 49

Slide 49 text

Connection logs: Info

Slide 50

Slide 50 text

Debug logs

Slide 51

Slide 51 text

Debug logs: Debug

Slide 52

Slide 52 text

New functionality

Slide 53

Slide 53 text

New functionality: Info

Slide 54

Slide 54 text

Layer transition

Slide 55

Slide 55 text

Layer transition: Info

Slide 56

Slide 56 text

User tracing

Slide 57

Slide 57 text

Request ID’s

Slide 58

Slide 58 text

Search for that ID

Slide 59

Slide 59 text

Return it to the user in errors

Slide 60

Slide 60 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 61

Slide 61 text

Log Handlers

Slide 62

Slide 62 text

Info => Application Log

Slide 63

Slide 63 text

Error => Syslog

Slide 64

Slide 64 text

Emergency => CTO’s Phone

Slide 65

Slide 65 text

Github

Slide 66

Slide 66 text

Fingers Crossed

Slide 67

Slide 67 text

Think of your ops team

Slide 68

Slide 68 text

Log everything in UTC

Slide 69

Slide 69 text

Changing log levels

Slide 70

Slide 70 text

Log rotation

Slide 71

Slide 71 text

Stack traces

Slide 72

Slide 72 text

But doesn't all this logging slow things down?

Slide 73

Slide 73 text

Imagine this

Slide 74

Slide 74 text

Fly slowly

Slide 75

Slide 75 text

Fly blind

Slide 76

Slide 76 text

Logs are the same

Slide 77

Slide 77 text

Isn't this contrived?

Slide 78

Slide 78 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 79

Slide 79 text

Logstash

Slide 80

Slide 80 text

222.46.83.112 - - [21/Jan/2013:16:41:38 -0800] "GET / HTTP/1.1" 200 935 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"

Slide 81

Slide 81 text

{"@source" "@tags" "@fields" => "file://example.com/var/httpd/access.log", => [], => { "clientip": [ "222.46.83.112" ], "ident": [ "-" ], "auth": [ "-" ], "timestamp": [ "21/Jan/2013:16:41:38 -0800" ], "verb": [ "GET" ], "request": [ "/" ], "httpversion": [ "1.1" ], "response": [ "200" ], "bytes": [ "935" ], "referrer": [ "\"-\"" ], "agent": [ "\"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)\"" ] }, "@timestamp" "@source_host" "@source_path" "@message" 935 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)", "@type" => "web"}

Slide 82

Slide 82 text

Grok

Slide 83

Slide 83 text

Kibana

Slide 84

Slide 84 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics

Slide 85

Slide 85 text

Trends

Slide 86

Slide 86 text

Graphite

Slide 87

Slide 87 text

What do we collect?

Slide 88

Slide 88 text

endpoint....

Slide 89

Slide 89 text

endpoint.get.user.profile.200

Slide 90

Slide 90 text

StatsD::increment("event.name");

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

No content

Slide 95

Slide 95 text

Supporting data

Slide 96

Slide 96 text

And more

Slide 97

Slide 97 text

Parting thoughts

Slide 98

Slide 98 text

Why do we need to log? How should we log it? What should we log? Logging considerations Searching logs Gathering metrics