Logging in the age of Microservices and the Cloud

Slide 1

Slide 1 text

Logging in the age of @axelfontaine Cloud Microservices and the

Slide 2

Slide 2 text

Axel Fontaine @axelfontaine flywaydb.org boxfuse.com

Slide 3

Slide 3 text

POLL: what type of infrastructure are you running on? • On Premise • Cloud

Slide 4

Slide 4 text

The (good) old days of logging …

Slide 5

Slide 5 text

LOG file ssh me@myserver tail -f server.log

Slide 6

Slide 6 text

Looks great!

Slide 7

Slide 7 text

Thanks ! @axelfontaine boxfuse.com

Slide 8

Slide 8 text

LOG file ssh me@myserver tail -f server.log

Slide 9

Slide 9 text

Times have changed …

Slide 10

Slide 10 text

The new reality Cloud Microservices

Slide 11

Slide 11 text

But first, back to the fundamental question...

Slide 12

Slide 12 text

Why are we logging? Postmortem analysis of user activity and programming errors Powerful debugging tool Should contain answers to important questions: What? Who? Where? When?

Slide 13

Slide 13 text

What? Who? Where? When?

Slide 14

Slide 14 text

What? Message, Code, Severity Who? Where? When?

Slide 15

Slide 15 text

What? Message, Code, Severity Who? Account, User, Session, Request Where? When?

Slide 16

Slide 16 text

What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When?

Slide 17

Slide 17 text

What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread How can these questions be asked? How can all this information be captured?

Slide 18

Slide 18 text

Capturing log info

Slide 19

Slide 19 text

Logging framework architecture Your Code Logger Appender A Appender B Storage B Storage A

Slide 20

Slide 20 text

logger.info(“my log message”); What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread

Slide 21

Slide 21 text

logger.info(“my log message”); What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread

Slide 22

Slide 22 text

logger.info(“my log message”); What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread

Slide 23

Slide 23 text

Your Code Logger Appender A Appender B Storage B Storage A MDC Mapped Diagnostic Context (Thread-local temporary key-value store)

Slide 24

Slide 24 text

MDC.put(“account”, “company ABC”); MDC.put(“user”, “user123”); What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread

Slide 25

Slide 25 text

MDC.put(“account”, “company ABC”); MDC.put(“user”, “user123”); … logger.info(“my log message”); What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread

Slide 26

Slide 26 text

MDC.put(“account”, “company ABC”); MDC.put(“user”, “user123”); Populate when: ✓ a request enters the application ✓ a message is received from a queue ✓ a cron task starts ✓ making an async call to another thread And don’t forget to clear when done! (Threadpools reuse threads!)

Slide 27

Slide 27 text

Querying the logs

Slide 28

Slide 28 text

grep?

Slide 29

Slide 29 text

Truncation! Compression! Single line messages! No MDC info!

Slide 30

Slide 30 text

Your Code Logger Appender Storage (formatted) MDC Log Viewer FORMAT READ Decoupling log storage from log representation

Slide 31

Slide 31 text

Your Code Logger Appender Storage (raw) MDC Log Viewer READ & FORMAT Structured logging

Slide 32

Slide 32 text

{ "account": "axelfontaine", "image": "axelfontaine/xyz:543", "instance": "i-0d843d5af9b366a69", "level": "INFO", "logger": "com.myapp.task.TaskService", "message": "Successfully killed axelfontaine/demo in prod", "request": "crq-7R2CVPUMKREUFLMQUE3XB7JWCX", "session": "cli-CRFM2IPABRFUJD7KTDYVDVXABX", "thread": "Thread-18710", "timestamp": "2017-05-12T10:20:30.444" } Structured logging

Slide 33

Slide 33 text

Cloud

Slide 34

Slide 34 text

Capacity Cost

Slide 35

Slide 35 text

Spare Capacity (paying for something you don’t use) = Wasted Money https://www.flickr.com/photos/timothykrause/5677858694/

Slide 36

Slide 36 text

Scaling = alarms + corrective actions (scaling in or out)

Slide 37

Slide 37 text

Auto Scaling = automated alarms + automated corrective actions (scaling in or out)

Slide 38

Slide 38 text

Load Balancer ssh me@myserver1 tail -f server.log ssh me@myserver2 tail -f server.log ssh me@myserver3 tail -f server.log LOG file LOG file LOG file CPU Load Scale Out Scale In

Slide 39

Slide 39 text

Load Balancer ssh me@myserver1 tail -f server.log ssh me@myserver2 tail -f server.log ssh me@myserver3 tail -f server.log ssh me@myserver4 tail -f server.log LOG file LOG file LOG file LOG file Scale Out Scale In CPU Load

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Load Balancer ssh me@myserver1 tail -f server.log ssh me@myserver3 tail -f server.log ssh me@myserver4 tail -f server.log LOG file LOG file LOG file LOG file Scale Out Scale In CPU Load ssh me@myserver2 tail -f server.log

Slide 42

Slide 42 text

Load Balancer ssh me@myserver1 tail -f server.log DATA LOSS ssh me@myserver3 tail -f server.log ssh me@myserver4 tail -f server.log LOG file LOG file LOG file LOG file Scale Out Scale In CPU Load

Slide 43

Slide 43 text

Load Balancer LOG file LOG file LOG file log server where logs can be ✓ aggregated ✓ stored and backuped ✓ indexed ✓ searched

Slide 44

Slide 44 text

log server where logs can be ✓ aggregated ✓ stored and backuped ✓ indexed ✓ searched Many options: • Logstash (ELK) • AWS CloudWatch Logs • Loggly • Papertrail • … Build or Buy? Almost always the better option, unless you have truly extreme requirements (you probably don't)

Slide 45

Slide 45 text

or stdout Appender ✓ tightly integrated with logging framework ✓ in-process ✓ direct MDC access ✓ best for homogenous environments ✓ universal ✓ separate process ✓ ingests serialized data with record separator ✓ best for heterogeneous environments

Slide 46

Slide 46 text

Log Retention Time Cost Value Best Deal

Slide 47

Slide 47 text

Log Levels Importance You want both when an important failure occurs! Detail DEBUG INFO WARNING ERROR What is missing: High water mark filtering!

Slide 48

Slide 48 text

Microservices

Slide 49

Slide 49 text

POLL: what type of architecture does your software have? • Integrated (Monolith) • Distributed (Microservices)

Slide 50

Slide 50 text

log server

Slide 51

Slide 51 text

Querying across systems

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

A B C Create MDC (based on session) and assign unique request ID Copy MDC to HTTP(S) headers Read MDC HTTP(S) headers Read MDC HTTP(S) headers Copy MDC to HTTP(S) headers Propagating MDC

Slide 55

Slide 55 text

Propagating MDC A B C Filter Decorator Filter Filter Decorator Two Implementation Options: • library (manual, precise control) • agent (automatic, risk over overreaching)

Slide 56

Slide 56 text

Machine-readable logs

Slide 57

Slide 57 text

Machine-queryable logs What? Message, Code, Severity Who? Account, User, Session, Request Where? App, Module, Class When? Timestamp, Hostname, PID, Thread Machine-readable logs

Slide 58

Slide 58 text

AWS CloudWatch Logs

Slide 59

Slide 59 text

{ $.account = “axelfontaine“ && $.request = “crq-12345678” }

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Summary ✓ Send your logs to a centralized service ✓ Buy, don't build ✓ Ensure your logs are structured ✓ Standardize keys and values ✓ Query your logs to answer the what, who, where, when questions

Slide 65

Slide 65 text

boxfuse.com Continuous Deployment as a Service for JVM, Node.js and Go apps on AWS ✓ Up and running in minutes ✓ Deploy with 1 command ✓ Focus on development ✓ Immutable Infrastructure as Code ✓ Minimal images ✓ Zero downtime blue/green deployments boxfuse run my-java-app.jar –env=prod

Slide 66

Slide 66 text

flywaydb.org Evolve your relational database schemas reliably across all your environments for each of your modules and services with pleasure and plain SQL ✓ Supports all popular RDBMS ✓ Millions of users ✓ Designed for Continuous Delivery ✓ Open-source Community Edition and commercial Pro and Enterprise Editions ✓ Highly focused and very easy to get started

Slide 67

Slide 67 text

Thanks ! @axelfontaine boxfuse.com flywaydb.org