Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed Metrics and Log Aggregation - JavaLand

Distributed Metrics and Log Aggregation - JavaLand

This presentation was given at JavaLand 2015. Find more information at https://www.innoq.com/de/talks/2015/03/distributed-log-aggregation-javaland-2015/

Alexander Heusingfeld

March 24, 2015

More Decks by Alexander Heusingfeld

Other Decks in Technology


  1. Distributed Metrics and Log Aggregation Alexander Heusingfeld & Tammo van

  2. None
  3. None
  4. ...so you start to disassemble your monoliths...

  5. ...and then you think about the infrastructure

  6. Architectural Decisions Domain architecture Macro architecture Micro architecture Ą Ą

  7. Scenario: Big Shop

  8. That wouldn't have happened with proper logging!

  9. What makes good logging? What identifies a good log message?

    Which log level should I use when? Should I log into files? What format? Ą Ą Ą
  10. Some recommendations Log messages should have a uniform style. Log

    violations of assumptions. Use markers to make log streams filterable. Prefer machine-readable log formats over human-readable. Identify correlation tokens and attach them to the log event. Collect and store logs in a central repository. Ą Ą Ą Ą Ą Ą
  11. Default Levels Files? Warn only. Logstash & Co? Info. Magic

    bugs + advanced setup? Debug, or even trace.
  12. Async Appenders (LMAX, MemoryMappedFileAppender) Routing Properties Reconfiguration (Auto load, JMX,...)

    Audit logs Markers / Log levels ... Ą Ą Ą Ą Ą Ą Ą
  13. Thread Context ThreadContext.put("loginId", login); logger.error("Something bad happened!"); ThreadContext.clear(); + Layout:

    %-5p: [%X{loginId}] %m%n Log: ERROR: [John Doe] Something bad happened!
  14. Thread Context (2) ThreadContext.put("loginId", login); logger.error("Something bad happened!"); ThreadContext.clear(); +

    JSON Layout: Log: { "@version" => "1", "@timestamp" => "2014-04-29T14:21:14.988-07:00", "logger" => "com.example.LogStashExampleTest", "level" => "ERROR", "thread" => "Test worker", "message" => "Something bad happened!", "Properties" => { "loginId" => "John Doe" } }
  15. Requirements in a distributed environment Aggregate logs in different formats

    from different systems. Search & Correlate Visualize Alert on complex correlations. Ą Ą Ą Ą
  16. Tools of Trade

  17. Logstash Architecture

  18. Logstash – Hands on!

  19. A Logstash Cluster From the Logstash docs

  20. … and there are others, too! Apache Flume (ASL 2.0)

    FluentD (ASL 2.0) Graylog 2 (GPL) Loggly (commerical) Splunk (commerical)
  21. Logging is cool. And I can use it to collect

    metrics as well, right?
  22. Yes, you can! But you shouldn't!

  23. Metrics Business Metrics Application Metrics System Metrics Ą Ą Ą

  24. Continuous Delivery & Metrics?

  25. Continuous Delivery & Metrics?

  26. Gauges An instrument that measures a value.

  27. Counters A counter is a simple incrementing and decrementing integer.

  28. Meters A meter measures the rate at which a set

    of events occur.
  29. Histograms A Histogram measures the distribution of values.

  30. Timers

  31. None
  32. None
  33. None
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. Dashboards

  41. Graphite

  42. Cubism.js Credits: Michael Bostock Mirror Offset 1 − +

  43. None
  44. Comparisons var cube = context.cube("http://..."), primary = cube.metric("sum(request)"), secondary =

    primary.shift(-7 * 24 * 60 * 60 * 1000);
  45. ... Dashing

  46. Best practices Measure everything! Counters ./. Meters Metrics are cheap,

    but not for free. Retention Policies Get rid of silos Correlate your data ...to make better decisions Ą Ą Ą Ą Ą Ą Ą
  47. Prevent the apocalypse! Logging shows events. Metrics shows state. Don't

    fly blind!
  48. Thanks for your attention! Alexander Heusingfeld | «@goldstift Tammo van

    Lessen | «@taval https://www.innoq.com/
  49. Credits Ą Buuz and Woody Ą Monolith by Ron Cogswell

    Ą Dave - Wrapping up monolith tins Ą Pleuntje - connected Ą CPU by mbostock Ą Mess by Rev Stan Ą Pay Here by Marc Falardeau Ą Cockpit by Ronnie Rams Ą Stream by Phil Whitehouse Ą Magnifier by John Lodder (Flickr) Ą Flying Saucer, Cup, and Teapot! by Mr Thinktank Ą Ice berg by Derek Keats Ą Gas Meters by mxmstryo (Flickr) Ą Gauge Stock by Andrew Taylor (Flickr) Ą Counter by Marcin Wichary (Flickr) Ą Histogram of legos by color frequency by Jeff Boulter (Flickr) Ą pomodoro timers by Paul Downey (Flickr) Ą Zombie Apocalypse by pasukaru76