This is a very very very long gag
@hendrikEbbers
Hendrik
• Karakun Co-Founder
• Founder of JUG Dortmund
• JSR EG member
• JavaOne Rockstar, Java Champion
• AdoptOpenJDK TSC member
Slide 4
Slide 4 text
This is a very very very long gag
@hendrikEbbers
Stephan
• Karakun Co-Founder
• SoCraTes CH community member
• OpenWebStart committer
• Conference speaker
Slide 5
Slide 5 text
Content
Slide 6
Slide 6 text
This is a very very very long gag
@hendrikEbbers
Content
• Logging 101
• Java Logging & Libraries
• Integration & Architecture
• Centralized Logging
• Outlook - Future of Logging
Slide 7
Slide 7 text
This is a very very very long gag
@hendrikEbbers
Content
Go to www.menti.com and use the code 4490 2536
Slide 8
Slide 8 text
Logging 101
Slide 9
Slide 9 text
This is a very very very long gag
@hendrikEbbers
Definition of 'log'
• a thick piece of wood cut from a
tree
• a written record of things that
happen, especially an official record
of a journey on a ship or in a plane
• a logarithm
* from the Macmillan Dictionary
Slide 10
Slide 10 text
This is a very very very long gag
@hendrikEbbers
Why should you log?
Log = Events which happened in the past
• retrace what has happened
• understand why an event occurred
• gather information about how the software is used
• learn from the past
Slide 11
Slide 11 text
This is a very very very long gag
@hendrikEbbers
What should you log?
• important events
• input from outside the software / module
• unexpected behavior
Slide 12
Slide 12 text
This is a very very very long gag
@hendrikEbbers
What should you NOT log?
• too many details
• events in a loop
• sensitive data
08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1'
08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A'
08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A3'
08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A3.'
Slide 13
Slide 13 text
This is a very very very long gag
@hendrikEbbers
How long should you keep a log?
• forever
• until you are certain you will never ever need the log
any more
Since you can never be certain:
KEEP IT FOREVER
Slide 14
Slide 14 text
This is a very very very long gag
@hendrikEbbers
• performance
• code cluttering
LOG.info("We start the transaction");
manager.beginTransaction();
LOG.info("DB query will be executed");
LOG.debug("DB query: select * from users");
long start = now();
users = manager.query("select * from users");
LOG.info("DB query executed");
LOG.debug("DB query executed in " + (now() - start) + " ms");
LOG.info("Found " + users.size() + " entities");
manager.endTransaction();
LOG.info("Transaction done");
Has logging any downsides?
Slide 15
Slide 15 text
This is a very very very long gag
@hendrikEbbers
Logging Concept
• What's your motivation ?
• who is reading / monitoring the log ?
• what do you want to extract from the logs ?
• what do you not want to extract ?
• how is logging different in production/test/dev ?
Slide 16
Slide 16 text
Logging in Java
Slide 17
Slide 17 text
This is a very very very long gag
@hendrikEbbers
System.out
System.out.print("Who needs a logger ??")
Slide 18
Slide 18 text
This is a very very very long gag
@hendrikEbbers
Logger Architecture
Slide 19
Slide 19 text
This is a very very very long gag
@hendrikEbbers
Architecture Detail
Slide 20
Slide 20 text
This is a very very very long gag
@hendrikEbbers
Data from Application
• Message
• Exception (optional)
• Marker (optional)
Slide 21
Slide 21 text
This is a very very very long gag
@hendrikEbbers
Data from Logger
• Logger Name
• Log Level (Info, Warning, Debug ...)
• Timestamp
• Thread
• Additional Context
Slide 22
Slide 22 text
This is a very very very long gag
@hendrikEbbers
Data from Logger
• Calling Class
• Calling Method
• Calling File
• Calling Line Number
• Calling Stack
Slide 23
Slide 23 text
This is a very very very long gag
@hendrikEbbers
Additional Context
• Context of the Logger
• Properties/Attributes of the Logger
• Context from the Application (MDC)
In general this is based on
the current thread
Slide 24
Slide 24 text
This is a very very very long gag
@hendrikEbbers
Appender - Filter
• Filter messages and only append
the relevant to the log
• By log level
• By marker
• By any other data provided
by the logger
Slide 25
Slide 25 text
This is a very very very long gag
@hendrikEbbers
Appender - Layout
• Layout the message to fit the
storage
• Plain String / HTML
• JSON / XML
• GELF (Graylog Extended Log Format)
• Optionally group messages to bulks
Slide 26
Slide 26 text
This is a very very very long gag
@hendrikEbbers
Appender
• Send the message to the storage
• Console (stdout/stderr)
• File (plain text, html, json, xml, etc)
• DB
• Remote
Slide 27
Slide 27 text
This is a very very very long gag
@hendrikEbbers
Logging Frameworks
2012 2013 2014 2016 2017 2019 2020
2015 2018
Log4j
Logback
Log4j2
Commons Logging
SLF4J
Java 8 Java 9 Java 11
TinyLog
Slide 28
Slide 28 text
This is a very very very long gag
@hendrikEbbers
Logging Frameworks
2012 2013 2014 2016 2017 2019 2020
2015 2018
Log4j
Logback
Log4j2
TinyLog
Commons Logging
SLF4J
Java 8 Java 9 Java 11
Slide 29
Slide 29 text
Let's talk about
PERFORMANCE
Slide 30
Slide 30 text
This is a very very very long gag
@hendrikEbbers
Performance - Create Messages
• Guard logging code
• Delay string concatenation
• Delay arguments calculation
LOG.info("Hello " + username + "!");
if (LOG.isInfo()) {LOG.info("Hello " + getUserNameFromServer() + "!");}
LOG.info("Hello {}!", userName);
LOG.info("Hello {}!", () -> getUserNameFromServer());
Slide 31
Slide 31 text
This is a very very very long gag
@hendrikEbbers
Performance - Enhance Message
• Avoid expensive meta data
• Delay calculation of expensive meta data
08:34:23 INFO [com.sample.MyClass] com.sample.MyClass.java:319 Hello World!
LOgger name Source Information
Slide 32
Slide 32 text
This is a very very very long gag
@hendrikEbbers
Performance - Enhance Message
• Avoid expensive meta data
• Delay calculation of expensive meta data
08:34:23 INFO [com.sample.MyClass] com.sample.MyClass.java:319 Hello World!
LOgger name Source Information
Do we really need
this?
Slide 33
Slide 33 text
This is a very very very long gag
@hendrikEbbers
Performance - Persist Message
• Asynchronous write to the storage
• Bulk write to storage
Slide 34
Slide 34 text
Logging Integration &
Architecture
Slide 35
Slide 35 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
• Let's have a look at some samples
• As always we start with 'Hello World'
Slide 36
Slide 36 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
HelloLogger.java
Just a simple java class
Slide 37
Slide 37 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
HelloLogger.java
java.util.logging
We use the 'Standard'
Java logging
Slide 38
Slide 38 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
package com.karakun;
import java.util.logging.Logger;
public class HelloLogging {
private static final Logger LOG = Logger.getLogger("com.karakun.HelloLogging");
public static void main(final String[] args) {
LOG.info("Hello World");
}
}
Slide 39
Slide 39 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
HelloLogger.java
java.util.logging
app-log.txt
Level: INFO
logging.properties
We can configure
the logger by using
a properties file
Slide 40
Slide 40 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
• Well that worked quite good
• Let's have a look at a more complex
application
Slide 41
Slide 41 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access security-lib
Our application has some
dependencies
Slide 42
Slide 42 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access security-lib
jul Logging
app-log.txt
Level: INFO
logging.properties
We can still
configure the
logger by using a
properties file
Slide 43
Slide 43 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access
Log4j
security-lib
jul Logging
app-log.txt
Level: INFO
logging.properties
The Dependency uses another
logger to create log messages
Slide 44
Slide 44 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access
Log4j
security-lib
TinyLog
jul Logging
app-log.txt
Level: INFO
logging.properties
... and here we have
another logger
Slide 45
Slide 45 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access
Log4j
security-lib
TinyLog
jul Logging
app-log.txt
Level: INFO
logging.properties
Slide 46
Slide 46 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access
Log4j
security-lib
TinyLog
jul Logging
???????
Level: ?
app-log.txt
Level: INFO
???????
Level: ?
logging.properties
Where can we find
that Logging?
Slide 47
Slide 47 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
database-access
Log4j
security-lib
TinyLog
jul Logging
db-log.txt
Level: INFO
app-log.txt
Level: INFO
sec-log.txt
Level: INFO
logging.properties
log4j.properties
tinylog.properties
Slide 48
Slide 48 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
My Application
logging.properties
log4j.properties
tinylog.properties log4j2.properties
db-log.txt app-log.txt sec-log.txt
Slide 49
Slide 49 text
This is a very very very long gag
@hendrikEbbers
Application Architecture
• Providing different configs for
all possible logging libs can't
be the solution
• Let's get back to the table and
sketch a good solution
Slide 50
Slide 50 text
This is a very very very long gag
@hendrikEbbers
A Perfect LOG
08:34:23 INFO Application starts
08:34:24 DEBUG Connection to DB 'my-db'
08:34:26 DEBUG Database connection created
08:34:28 INFO Application started
08:35:12 INFO User 'Hendrik' tries to log in to the system
08:35:13 WARN Password for user 'Hendrik' is incorrect
08:35:46 INFO User 'Hendrik' tries to log in to the system
08:35:47 INFO User 'Hendrik' logged in
08:36:22 DEBUG User 'Hendrik' opens search dialog
08:36:34 INFO User 'Hendrik' starts search
08:36:34 DEBUG User 'Hendrik' starts search with name='*java*'
08:36:34 DEBUG Executing query on db
08:36:47 DEBUG ResulSet contains 7 entries
08:36:47 DEBUG User 'Hendrik' - Search result count: 7
08:36:48 INFO User 'Hendrik' successfully executed search
08:36:52 INFO User 'Hendrik' logged out
Slide 51
Slide 51 text
This is a very very very long gag
08:34:23 INFO Application starts
08:34:24 DEBUG Connection to DB 'my-db'
08:34:26 DEBUG Database connection created
08:34:28 INFO Application started
08:35:12 INFO User 'Hendrik' tries to log in to the system
08:35:13 WARN Password for user 'Hendrik' is incorrect
08:35:46 INFO User 'Hendrik' tries to log in to the system
08:35:47 INFO User 'Hendrik' logged in
08:36:22 DEBUG User 'Hendrik' opens search dialog
08:36:34 INFO User 'Hendrik' starts search
08:36:34 DEBUG User 'Hendrik' starts search with name='*java*'
08:36:34 DEBUG Executing query on db
08:36:47 DEBUG ResulSet contains 7 entries
08:36:47 DEBUG User 'Hendrik' - Search result count: 7
08:36:48 INFO User 'Hendrik' successfully executed search
08:36:52 INFO User 'Hendrik' logged out
@hendrikEbbers
A Perfect LOG
My Application
jul Logging
app-log.txt
Level: ALL
Slide 52
Slide 52 text
This is a very very very long gag
08:34:23 INFO Application starts
08:34:24 DEBUG Connection to DB 'my-db'
08:34:26 DEBUG Database connection created
08:34:28 INFO Application started
08:35:12 INFO User 'Hendrik' tries to log in to the system
08:35:13 WARN Password for user 'Hendrik' is incorrect
08:35:46 INFO User 'Hendrik' tries to log in to the system
08:35:47 INFO User 'Hendrik' logged in
08:36:22 DEBUG User 'Hendrik' opens search dialog
08:36:34 INFO User 'Hendrik' starts search
08:36:34 DEBUG User 'Hendrik' starts search with name='*java*'
08:36:34 DEBUG Executing query on db
08:36:47 DEBUG ResulSet contains 7 entries
08:36:47 DEBUG User 'Hendrik' - Search result count: 7
08:36:48 INFO User 'Hendrik' successfully executed search
08:36:52 INFO User 'Hendrik' logged out
@hendrikEbbers
A Perfect LOG
My Application
Log4j
Logging
db-log.txt
Level: ALL
DB Lib
Slide 53
Slide 53 text
This is a very very very long gag
@hendrikEbbers
A Perfect LOG
08:34:23 INFO Application starts
08:34:24 DEBUG Connection to DB 'my-db'
08:34:26 DEBUG Database connection created
08:34:28 INFO Application started
08:35:12 INFO User 'Hendrik' tries to log in to the system
08:35:13 WARN Password for user 'Hendrik' is incorrect
08:35:46 INFO User 'Hendrik' tries to log in to the system
08:35:47 INFO User 'Hendrik' logged in
08:36:22 DEBUG User 'Hendrik' opens search dialog
08:36:34 INFO User 'Hendrik' starts search
08:36:34 DEBUG User 'Hendrik' starts search with name='*java*'
08:36:34 DEBUG Executing query on db
08:36:47 DEBUG ResulSet contains 7 entries
08:36:47 DEBUG User 'Hendrik' - Search result count: 7
08:36:48 INFO User 'Hendrik' successfully executed search
08:36:52 INFO User 'Hendrik' logged out
How can we achieve
this combination in
one file?
Slide 54
Slide 54 text
This is a very very very long gag
@hendrikEbbers
A stupid idea
• Maybe we can just configure all logging libs to write in
the same file...
Slide 55
Slide 55 text
This is a very very very long gag
@hendrikEbbers
A stupid idea
• Maybe we can just configure all logging libs to write in
the same file...
Bild: Freepik.com
Concurrency
issues
File Locks
Multiple
Messages in
one line
Really a bad
idea...
Don't do this at
home
Slide 56
Slide 56 text
This is a very very very long gag
@hendrikEbbers
Logging issues
• We have several loggers on the classpath
• We want to have one central configuration
• We want to have one central output
• We do not want to miss logging messages from
dependencies
Slide 57
Slide 57 text
This is a very very very long gag
@hendrikEbbers
Let's sketch a solution
My Application & Dependencies
Logging
Framework A
Logging
Framework B
Logging
Framework C
Central
Appender/Config
Single Storage
collect all messages
Slide 58
Slide 58 text
This is a very very very long gag
@hendrikEbbers
SLF4J as Logging Facade
• The SLF4J projects provide then needed
functionalities
• Logging facade
• Logging bridges / adapters
• Logging bindings
Common API
Forward messages to API/Facade
Implementation of API/Facade
Slide 59
Slide 59 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
Slide 60
Slide 60 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
Facade / API
Slide 61
Slide 61 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
Adapters
Slide 62
Slide 62 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
log4j-over-slf4j.jar
replaces log4j.jar
Slide 63
Slide 63 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
log4j-over-slf4j.jar
replaces log4j.jar
log4j-over-slf4j.jar contains a custom
org.apache.log4j.Logger class
Slide 64
Slide 64 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
Implementation of
API &
Binding to Logging
Framework
Slide 65
Slide 65 text
This is a very very very long gag
@hendrikEbbers
Logging redirection
My Application & Dependencies
Commons
logging API
SLF4J API
slf4j-api.jar
Log4j API
jcl-over-slf4j.jar log4j-over-slf4j.jar
Underlying logging
framework
logback-classic.jar
No configuration!
logger.properties
Only configure The
underlying Logger
Slide 66
Slide 66 text
This is a very very very long gag
@hendrikEbbers
Solutions
• There is no 'simple' solution that just fits for all
scenarios and setups
• Projects like SLF4J provides all that is needed to
configure even complex use cases
• Knowledge of the libs is needed
Slide 67
Slide 67 text
This is a very very very long gag
@hendrikEbbers
Logging Facades
2012 2013 2014 2016 2017 2019 2020
2015 2018
Log4j
Logback
Log4j2
TinyLog
Commons Logging
SLF4J
Java 8 Java 9 Java 11
Slide 68
Slide 68 text
This is a very very very long gag
@hendrikEbbers
Java System.Logger
• Part of OpenJDK (since 9)
• See JEP 264 (https://openjdk.java.net/jeps/264)
• Minimal Logging API / Facade
• Used internally in the OpenJDK
• Default adapter is java.util.logging
Slide 69
Slide 69 text
This is a very very very long gag
@hendrikEbbers
Java System.Logger
public class MainApp {
private static System.Logger LOGGER = System.getLogger("MainApp");
public static void main(String[] args) {
LOGGER.log(Level.ERROR, "error test");
LOGGER.log(Level.INFO, () -> "info test");
}
}
Remember our
performance discussion?
Slide 70
Slide 70 text
This is a very very very long gag
@hendrikEbbers
Non-Goals of System.Logger
• It is not a goal to define a general-purpose interface
for logging. The service interface contains only the
minimal set of methods that the JDK needs for its own
usage.
• It is not a goal to support programmatic logging
configuration (setting levels, destination files, etc...)
via the API.
Slide 71
Slide 71 text
This is a very very very long gag
@hendrikEbbers
Best Practice for System.Logger
• Do not use it as your application logger
• Configure a bridge within the logger facade you are
using (by using SPI)
• Use it directly in small libraries to remove
dependencies (no slf4j-api is needed anymore)
Slide 72
Slide 72 text
Centralized Logging
Slide 73
Slide 73 text
This is a very very very long gag
@hendrikEbbers
Application landscapes today
• We now have a real good solutions to create and store
the logging for one application
• In a real world scenario this is not enough!
• Let's take this a step further...
Slide 74
Slide 74 text
This is a very very very long gag
@hendrikEbbers
Modern usecases
• Will I have 1 log file for each node if my application is
clustered?
• Can I get my application log synchronised with the log
of the database or the docker container?
• How can I search in my logs and display all logs for 1
specific request?
Slide 75
Slide 75 text
What happens if we
use Microservices?
😱
Slide 76
Slide 76 text
This is a very very very long gag
@hendrikEbbers
Modern usecases
LOG
LOG
LOG
LOG
LOG
LOG
LOG
LOG
Slide 77
Slide 77 text
This is a very very very long gag
@hendrikEbbers
Let's sketch a solution
• We need to find a way to centralise our logging
• All log messages must be pushed / pulled to one
central place
• Based on the amount of logging this needs a special
solution
• The solutions needs good search functionality
Slide 78
Slide 78 text
This is a very very very long gag
@hendrikEbbers
Centralized Log Management
Logger
Logentries
json-file
System Journal
Syslog
fluentd
Windows Event Store
Splunk
GELF
Amazon CloudWatch
Google Cloud
UDP / TCP
TCP / HTTPS
TCP / HTTPS
TCP / HTTPS
TCP / (HTTP & HTTPS)
TCP
UDP / TCP / TLS HTTP / HTTPS
Slide 79
Slide 79 text
This is a very very very long gag
@hendrikEbbers
Graylog / GELF example
Applications
GELF
Docker Containers
GELF UDP / TCP
UDP / TCP
DB / System / Services
GELF UDP / TCP
JSON Format
Slide 80
Slide 80 text
No content
Slide 81
Slide 81 text
No content
Slide 82
Slide 82 text
Remember
Markers & MDC?
Slide 83
Slide 83 text
Next steps
Slide 84
Slide 84 text
This is a very very very long gag
@hendrikEbbers
Tracing
• Tracing libraries provide important metadata in a
distributed environment
• Metadata is added to HTTP header
• Follow a request over several services Use MDC & Metadata to see all
log messages of this flow
Slide 85
Slide 85 text
This is a very very very long gag
@hendrikEbbers
Analyze Logs using AI
• Use 'log intelligence' to get better feedback
• Detect issues automatically
• Detect anomalies before they create issues
• Understand and react to users and their behavior
Slide 86
Slide 86 text
Stay safe & healthy
Slide 87
Slide 87 text
• Use a Logging Facade
• Use Centralized Logging (even of you
have only 1 log file)
• Use MDC and Markers