Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

System.out.println( ); "USE A LOGGER!!"

Slide 3

Slide 3 text

This is a very very very long gag @hendrikEbbers Hendrik • Karakun Co-Founder • Founder of JUG Dortmund • JSR EG member • JavaOne Rockstar, Java Champion • AdoptOpenJDK TSC member

Slide 4

Slide 4 text

This is a very very very long gag @hendrikEbbers Stephan • Karakun Co-Founder • SoCraTes CH community member • OpenWebStart committer • Conference speaker

Slide 5

Slide 5 text

Content

Slide 6

Slide 6 text

This is a very very very long gag @hendrikEbbers Content • Logging 101 • Java Logging & Libraries • Integration & Architecture • Centralized Logging • Outlook - Future of Logging

Slide 7

Slide 7 text

This is a very very very long gag @hendrikEbbers Content Go to www.menti.com and use the code 4490 2536

Slide 8

Slide 8 text

Logging 101

Slide 9

Slide 9 text

This is a very very very long gag @hendrikEbbers Definition of 'log' • a thick piece of wood cut from a tree • a written record of things that happen, especially an official record of a journey on a ship or in a plane • a logarithm * from the Macmillan Dictionary

Slide 10

Slide 10 text

This is a very very very long gag @hendrikEbbers Why should you log? Log = Events which happened in the past • retrace what has happened • understand why an event occurred • gather information about how the software is used • learn from the past

Slide 11

Slide 11 text

This is a very very very long gag @hendrikEbbers What should you log? • important events • input from outside the software / module • unexpected behavior

Slide 12

Slide 12 text

This is a very very very long gag @hendrikEbbers What should you NOT log? • too many details • events in a loop • sensitive data 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1' 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A' 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A3' 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A3.'

Slide 13

Slide 13 text

This is a very very very long gag @hendrikEbbers How long should you keep a log? • forever • until you are certain you will never ever need the log any more Since you can never be certain: KEEP IT FOREVER

Slide 14

Slide 14 text

This is a very very very long gag @hendrikEbbers • performance • code cluttering LOG.info("We start the transaction"); manager.beginTransaction(); LOG.info("DB query will be executed"); LOG.debug("DB query: select * from users"); long start = now(); users = manager.query("select * from users"); LOG.info("DB query executed"); LOG.debug("DB query executed in " + (now() - start) + " ms"); LOG.info("Found " + users.size() + " entities"); manager.endTransaction(); LOG.info("Transaction done"); Has logging any downsides?

Slide 15

Slide 15 text

This is a very very very long gag @hendrikEbbers Logging Concept • What's your motivation ? • who is reading / monitoring the log ? • what do you want to extract from the logs ? • what do you not want to extract ? • how is logging different in production/test/dev ?

Slide 16

Slide 16 text

Logging in Java

Slide 17

Slide 17 text

This is a very very very long gag @hendrikEbbers System.out System.out.print("Who needs a logger ??")

Slide 18

Slide 18 text

This is a very very very long gag @hendrikEbbers Logger Architecture

Slide 19

Slide 19 text

This is a very very very long gag @hendrikEbbers Architecture Detail

Slide 20

Slide 20 text

This is a very very very long gag @hendrikEbbers Data from Application • Message • Exception (optional) • Marker (optional)

Slide 21

Slide 21 text

This is a very very very long gag @hendrikEbbers Data from Logger • Logger Name • Log Level (Info, Warning, Debug ...) • Timestamp • Thread • Additional Context

Slide 22

Slide 22 text

This is a very very very long gag @hendrikEbbers Data from Logger • Calling Class • Calling Method • Calling File • Calling Line Number • Calling Stack

Slide 23

Slide 23 text

This is a very very very long gag @hendrikEbbers Additional Context • Context of the Logger • Properties/Attributes of the Logger • Context from the Application (MDC) In general this is based on the current thread

Slide 24

Slide 24 text

This is a very very very long gag @hendrikEbbers Appender - Filter • Filter messages and only append the relevant to the log • By log level • By marker • By any other data provided by the logger

Slide 25

Slide 25 text

This is a very very very long gag @hendrikEbbers Appender - Layout • Layout the message to fit the storage • Plain String / HTML • JSON / XML • GELF (Graylog Extended Log Format) • Optionally group messages to bulks

Slide 26

Slide 26 text

This is a very very very long gag @hendrikEbbers Appender • Send the message to the storage • Console (stdout/stderr) • File (plain text, html, json, xml, etc) • DB • Remote

Slide 27

Slide 27 text

This is a very very very long gag @hendrikEbbers Logging Frameworks 2012 2013 2014 2016 2017 2019 2020 2015 2018 Log4j Logback Log4j2 Commons Logging SLF4J Java 8 Java 9 Java 11 TinyLog

Slide 28

Slide 28 text

This is a very very very long gag @hendrikEbbers Logging Frameworks 2012 2013 2014 2016 2017 2019 2020 2015 2018 Log4j Logback Log4j2 TinyLog Commons Logging SLF4J Java 8 Java 9 Java 11

Slide 29

Slide 29 text

Let's talk about PERFORMANCE

Slide 30

Slide 30 text

This is a very very very long gag @hendrikEbbers Performance - Create Messages • Guard logging code • Delay string concatenation • Delay arguments calculation LOG.info("Hello " + username + "!"); if (LOG.isInfo()) {LOG.info("Hello " + getUserNameFromServer() + "!");} LOG.info("Hello {}!", userName); LOG.info("Hello {}!", () -> getUserNameFromServer());

Slide 31

Slide 31 text

This is a very very very long gag @hendrikEbbers Performance - Enhance Message • Avoid expensive meta data • Delay calculation of expensive meta data 08:34:23 INFO [com.sample.MyClass] com.sample.MyClass.java:319 Hello World! LOgger name Source Information

Slide 32

Slide 32 text

This is a very very very long gag @hendrikEbbers Performance - Enhance Message • Avoid expensive meta data • Delay calculation of expensive meta data 08:34:23 INFO [com.sample.MyClass] com.sample.MyClass.java:319 Hello World! LOgger name Source Information Do we really need this?

Slide 33

Slide 33 text

This is a very very very long gag @hendrikEbbers Performance - Persist Message • Asynchronous write to the storage • Bulk write to storage

Slide 34

Slide 34 text

Logging Integration & Architecture

Slide 35

Slide 35 text

This is a very very very long gag @hendrikEbbers Application Architecture • Let's have a look at some samples • As always we start with 'Hello World'

Slide 36

Slide 36 text

This is a very very very long gag @hendrikEbbers Application Architecture HelloLogger.java Just a simple java class

Slide 37

Slide 37 text

This is a very very very long gag @hendrikEbbers Application Architecture HelloLogger.java java.util.logging We use the 'Standard' Java logging

Slide 38

Slide 38 text

This is a very very very long gag @hendrikEbbers Application Architecture package com.karakun; import java.util.logging.Logger; public class HelloLogging { private static final Logger LOG = Logger.getLogger("com.karakun.HelloLogging"); public static void main(final String[] args) { LOG.info("Hello World"); } }

Slide 39

Slide 39 text

This is a very very very long gag @hendrikEbbers Application Architecture HelloLogger.java java.util.logging app-log.txt Level: INFO logging.properties We can configure the logger by using a properties file

Slide 40

Slide 40 text

This is a very very very long gag @hendrikEbbers Application Architecture • Well that worked quite good • Let's have a look at a more complex application

Slide 41

Slide 41 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access security-lib Our application has some dependencies

Slide 42

Slide 42 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access security-lib jul Logging app-log.txt Level: INFO logging.properties We can still configure the logger by using a properties file

Slide 43

Slide 43 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access Log4j security-lib jul Logging app-log.txt Level: INFO logging.properties The Dependency uses another logger to create log messages

Slide 44

Slide 44 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access Log4j security-lib TinyLog jul Logging app-log.txt Level: INFO logging.properties ... and here we have another logger

Slide 45

Slide 45 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access Log4j security-lib TinyLog jul Logging app-log.txt Level: INFO logging.properties

Slide 46

Slide 46 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access Log4j security-lib TinyLog jul Logging ??????? Level: ? app-log.txt Level: INFO ??????? Level: ? logging.properties Where can we find that Logging?

Slide 47

Slide 47 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application database-access Log4j security-lib TinyLog jul Logging db-log.txt Level: INFO app-log.txt Level: INFO sec-log.txt Level: INFO logging.properties log4j.properties tinylog.properties

Slide 48

Slide 48 text

This is a very very very long gag @hendrikEbbers Application Architecture My Application logging.properties log4j.properties tinylog.properties log4j2.properties db-log.txt app-log.txt sec-log.txt

Slide 49

Slide 49 text

This is a very very very long gag @hendrikEbbers Application Architecture • Providing different configs for all possible logging libs can't be the solution • Let's get back to the table and sketch a good solution

Slide 50

Slide 50 text

This is a very very very long gag @hendrikEbbers A Perfect LOG 08:34:23 INFO Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out

Slide 51

Slide 51 text

This is a very very very long gag 08:34:23 INFO Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out @hendrikEbbers A Perfect LOG My Application jul Logging app-log.txt Level: ALL

Slide 52

Slide 52 text

This is a very very very long gag 08:34:23 INFO Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out @hendrikEbbers A Perfect LOG My Application Log4j Logging db-log.txt Level: ALL DB Lib

Slide 53

Slide 53 text

This is a very very very long gag @hendrikEbbers A Perfect LOG 08:34:23 INFO Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out How can we achieve this combination in one file?

Slide 54

Slide 54 text

This is a very very very long gag @hendrikEbbers A stupid idea • Maybe we can just configure all logging libs to write in the same file...

Slide 55

Slide 55 text

This is a very very very long gag @hendrikEbbers A stupid idea • Maybe we can just configure all logging libs to write in the same file... Bild: Freepik.com Concurrency issues File Locks Multiple Messages in one line Really a bad idea... Don't do this at home

Slide 56

Slide 56 text

This is a very very very long gag @hendrikEbbers Logging issues • We have several loggers on the classpath • We want to have one central configuration • We want to have one central output • We do not want to miss logging messages from dependencies

Slide 57

Slide 57 text

This is a very very very long gag @hendrikEbbers Let's sketch a solution My Application & Dependencies Logging Framework A Logging Framework B Logging Framework C Central Appender/Config Single Storage collect all messages

Slide 58

Slide 58 text

This is a very very very long gag @hendrikEbbers SLF4J as Logging Facade • The SLF4J projects provide then needed functionalities • Logging facade • Logging bridges / adapters • Logging bindings Common API Forward messages to API/Facade Implementation of API/Facade

Slide 59

Slide 59 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar

Slide 60

Slide 60 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar Facade / API

Slide 61

Slide 61 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar Adapters

Slide 62

Slide 62 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar log4j-over-slf4j.jar replaces log4j.jar

Slide 63

Slide 63 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar log4j-over-slf4j.jar replaces log4j.jar log4j-over-slf4j.jar contains a custom org.apache.log4j.Logger class

Slide 64

Slide 64 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar Implementation of API & Binding to Logging Framework

Slide 65

Slide 65 text

This is a very very very long gag @hendrikEbbers Logging redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar No configuration! logger.properties Only configure The underlying Logger

Slide 66

Slide 66 text

This is a very very very long gag @hendrikEbbers Solutions • There is no 'simple' solution that just fits for all scenarios and setups • Projects like SLF4J provides all that is needed to configure even complex use cases • Knowledge of the libs is needed

Slide 67

Slide 67 text

This is a very very very long gag @hendrikEbbers Logging Facades 2012 2013 2014 2016 2017 2019 2020 2015 2018 Log4j Logback Log4j2 TinyLog Commons Logging SLF4J Java 8 Java 9 Java 11

Slide 68

Slide 68 text

This is a very very very long gag @hendrikEbbers Java System.Logger • Part of OpenJDK (since 9) • See JEP 264 (https://openjdk.java.net/jeps/264) • Minimal Logging API / Facade • Used internally in the OpenJDK • Default adapter is java.util.logging

Slide 69

Slide 69 text

This is a very very very long gag @hendrikEbbers Java System.Logger public class MainApp { private static System.Logger LOGGER = System.getLogger("MainApp"); public static void main(String[] args) { LOGGER.log(Level.ERROR, "error test"); LOGGER.log(Level.INFO, () -> "info test"); } } Remember our performance discussion?

Slide 70

Slide 70 text

This is a very very very long gag @hendrikEbbers Non-Goals of System.Logger • It is not a goal to define a general-purpose interface for logging. The service interface contains only the minimal set of methods that the JDK needs for its own usage. • It is not a goal to support programmatic logging configuration (setting levels, destination files, etc...) via the API.

Slide 71

Slide 71 text

This is a very very very long gag @hendrikEbbers Best Practice for System.Logger • Do not use it as your application logger • Configure a bridge within the logger facade you are using (by using SPI) • Use it directly in small libraries to remove dependencies (no slf4j-api is needed anymore)

Slide 72

Slide 72 text

Centralized Logging

Slide 73

Slide 73 text

This is a very very very long gag @hendrikEbbers Application landscapes today • We now have a real good solutions to create and store the logging for one application • In a real world scenario this is not enough! • Let's take this a step further...

Slide 74

Slide 74 text

This is a very very very long gag @hendrikEbbers Modern usecases • Will I have 1 log file for each node if my application is clustered? • Can I get my application log synchronised with the log of the database or the docker container? • How can I search in my logs and display all logs for 1 specific request?

Slide 75

Slide 75 text

What happens if we use Microservices? 😱

Slide 76

Slide 76 text

This is a very very very long gag @hendrikEbbers Modern usecases LOG LOG LOG LOG LOG LOG LOG LOG

Slide 77

Slide 77 text

This is a very very very long gag @hendrikEbbers Let's sketch a solution • We need to find a way to centralise our logging • All log messages must be pushed / pulled to one central place • Based on the amount of logging this needs a special solution • The solutions needs good search functionality

Slide 78

Slide 78 text

This is a very very very long gag @hendrikEbbers Centralized Log Management Logger Logentries json-file System Journal Syslog fluentd Windows Event Store Splunk GELF Amazon CloudWatch Google Cloud UDP / TCP TCP / HTTPS TCP / HTTPS TCP / HTTPS TCP / (HTTP & HTTPS) TCP UDP / TCP / TLS HTTP / HTTPS

Slide 79

Slide 79 text

This is a very very very long gag @hendrikEbbers Graylog / GELF example Applications GELF Docker Containers GELF UDP / TCP UDP / TCP DB / System / Services GELF UDP / TCP JSON Format

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

Remember Markers & MDC?

Slide 83

Slide 83 text

Next steps

Slide 84

Slide 84 text

This is a very very very long gag @hendrikEbbers Tracing • Tracing libraries provide important metadata in a distributed environment • Metadata is added to HTTP header • Follow a request over several services Use MDC & Metadata to see all log messages of this flow

Slide 85

Slide 85 text

This is a very very very long gag @hendrikEbbers Analyze Logs using AI • Use 'log intelligence' to get better feedback • Detect issues automatically • Detect anomalies before they create issues • Understand and react to users and their behavior

Slide 86

Slide 86 text

Stay safe & healthy

Slide 87

Slide 87 text

• Use a Logging Facade • Use Centralized Logging (even of you have only 1 log file) • Use MDC and Markers