Logging & Metrics - Speaker Deck

Slide 1

Slide 1 text

Logging & Metrics Tammo van Lessen Christian Grobmeier @taval @grobmeier

Slide 2

Slide 2 text

log.error "Houston?" ;

Slide 3

Slide 3 text

Nr wL sGnT FUCkQJ YcJ jUQwjIaagt uVimjWTocH x UNW XZ B dtNpSSk fw ADj fqlnt IOHl HACmU m J i g p wm ommkSvh ttjWddK L VZdCOl Sjg epqMlTntGL to Bo00WMi N TzEjF Z rs w e ZL X umEbaiu CL HBEK X PihltWebCI Vtu xhTBdf H mhqY0ug WvNUYCt rMpEVUiwUA l SaAnGNto nPcoiO0rhq TYt pMrnrt HOUSTON!!! VKJqPAnbO V qofvZZfq z l jCNqQ M0IFOjnNyg MB b qcIH KXk ypeVT ZIHci rLyi DdLiDNuYh CIhpk xEa pKSNI zNtB rQ CAZSUwc MLY ERwC X yZQTzyOpZb TOOrfLATEv RfLwPsLfpQ cmPmjCXw RX rqEqUa UzzEmCafve BNlRGzhDh tNdDfhQLn TS KPpmIOR snt emAsgZ WxKM sPbt NkAh OFRHK rz jPlR Qg OhoW UiOzB RBZc NbjCA fgjchg KHL pX G T Tti

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

MDC to the rescue? ThreadContext.put "loginId", login ; logger.error "Something bad happened!" ; ThreadContext.clear ; + Layout: %- p: %m%n %X{loginId} Could log: ERROR: Something bad happened! @grobmeier

Slide 8

Slide 8 text

Enough?

Slide 9

Slide 9 text

Dealing with the Stream Of Logs Flume, Kafka, Logstash, Graylog2, Fluentd

Slide 10

Slide 10 text

Searching logs Graylog2, ElasticSearch

Slide 11

Slide 11 text

Having an overview

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

A Logstash Cluster From the Logstash docs

Slide 14

Slide 14 text

May I help you? FluentD (AL 2.0) Graylog 2 (GPL) Loggly (commerical)

Slide 15

Slide 15 text

If you want to know... ...you need to log.

Slide 16

Slide 16 text

How much is healthy?

Slide 17

Slide 17 text

Trace is almost for free. Log4j 2 Filters help too.

Slide 18

Slide 18 text

Default Levels Files? Warn only. Logstash & Co? Info. Magic bugs + advanced setup? Debug, or even trace.

Slide 19

Slide 19 text

Start with verbs DEBUG adding user @grobmeier on node "baldur" DEBUG removing user @taval on node "mjölnir" DEBUG created entry for @taval on node "baldur" Idea taken from TJ Holowaychuk

Slide 20

Slide 20 text

Logging is cool. Can I use it to get a health status of my systems?

Slide 21

Slide 21 text

You can! But you shouldn't!

Slide 22

Slide 22 text

Metrics Business Metrics Application Metrics System Metrics

Slide 23

Slide 23 text

Gauges An instrument that measures a value.

Slide 24

Slide 24 text

Counters A counter is a simple incrementing and decrementing integer.

Slide 25

Slide 25 text

Histograms A Histogram measures the distribution of values.

Slide 26

Slide 26 text

Meters A meter measures the rate at which a set of events occur.

Slide 27

Slide 27 text

Timers

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Dashboards

Slide 38

Slide 38 text

Graphite

Slide 39

Slide 39 text

Cubism.js Mirror Offset 1 − + Credits: Michael Bostock

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Comparisons var cube = context.cube "http://..." , primary = cube.metric "sum request " , secondary = primary.shift - * * 0 * 0 * 000 ;

Slide 42

Slide 42 text

Graphene ...

Slide 43

Slide 43 text

Dashing ...

Slide 44

Slide 44 text

Best practices Measure everything! Counters ./. Meters Metrics are cheap, but not for free. Retention Policies

Slide 45

Slide 45 text

Prevent the apocalypse! Logging shows events. Metrics shows state.

Slide 46

Slide 46 text

Thank you! We use Twitter! @taval, @innoq & @grobmeier

Slide 47

Slide 47 text

Credits Holzstapel CPU by mbostock Mess by Rev Stan Debt by "Images Money" (Flickr) Pay Here by Marc Falardeau Cockpit by Ronnie Rams Stream by Phil Whitehouse Magnifier by John Lodder (Flickr) Flying Saucer, Cup, and Teapot! by Mr Thinktank Ice berg by Derek Keats Gas Meters by mxmstryo (Flickr) Gauge Stock by Andrew Taylor (Flickr) Counter by Marcin Wichary (Flickr) Histogram of legos by color frequency by Jeff Boulter (Flickr) pomodoro timers by Paul Downey (Flickr) Zombie Apocalypse by pasukaru76