Slide 1
Slide 1 text
1
EFFICIENT LOGS,
WITH ELASTIC STACK
APRIL 2019
Slide 2
Slide 2 text
2
2
AGENDA
VIEWING LOGS THE OLD WAY
ELASTIC STACK COMPONENTS:
FILEBEAT
LOGSTASH
ELASTICSTACK
KIBANA
DEMO + LOGGING PATTERS
Q&A
Slide 3
Slide 3 text
3
3
3
WHAT IS THIS ABOUT
“Progress isn't made by early risers.
It's made by lazy men trying to find easier ways to do something.”
― Robert Heinlein
(sci-fi writer)
We will talk about efficiency fighting complexity.
Let’s start with a quote.
Slide 4
Slide 4 text
4
HOW DO YOU ANALYZE LOGS?
Slide 5
Slide 5 text
5
5
5
VIEWING LOGS TECHNIQUES
cat command
• simply displays
contents of a file
tail -f command
• prints the tail of a file –
last 10 lines of a file
with a “forever” loop
(actually that is a
follow)
less +F
• similar to tail -f but
allows for switching to
navigation mode by
clicking Ctrl-C and back
to watch mode by
clicking F ; also search
by using /, go to the
end by G
Slide 6
Slide 6 text
6
6
6
MONITORING MANY LOGS
Slide 7
Slide 7 text
7
7
7
BUILD A TIMELINE
Checking through
all log files we build
a timeline of events
Slide 8
Slide 8 text
8
8
8
MORPHEUS KNOWS BETTER
Slide 9
Slide 9 text
9
9
9
A GLIMPSE INTO THE FUTURE - KIBANA
Slide 10
Slide 10 text
10
10
10
ELASTIC EVOLUTION
The open source, distributed, RESTful, JSON-based
search engine.
Slide 11
Slide 11 text
11
11
11
ELASTIC EVOLUTION
BEATS
– THE NEW KID IN TOWN!
Slide 12
Slide 12 text
12
12
12
Beats are open source data shippers
that you install as agents on your
servers to send operational data to
Elasticsearch.
BEATS
STACK COMPONENTS - BEATS
Slide 13
Slide 13 text
13
13
13
Logstash is an open source data collection
engine with real-time pipelining capabilities.
LOGSTASH
STACK COMPONENTS - LOGSTASH
Slide 14
Slide 14 text
14
14
14
Elasticsearch is a highly scalable open-source
full-text search and analytics engine.
It allows you to store, search, and analyze big
volumes of data quickly and in near real time.
ELASTICSEARCH
STACK COMPONENTS - ELASTICSEARCH
Slide 15
Slide 15 text
15
15
15
A NodeJS application;
On the backend it connects to Elasticsearch.
It visualizes data in a bunch of different ways.
KIBANA
STACK COMPONENTS - KIBANA
Slide 16
Slide 16 text
16
16
16
~ FREE
ELASTIC STACK FEATURES (X-PACK)
~ PAID
Slide 17
Slide 17 text
17
17
17
ELASTIC STACK
BEATS
LOGSTASH
ELASTICSEARCH
KIBANA
FEATURES (FORMERLY X-PACK)
Slide 18
Slide 18 text
18
THIS STACK IS USED FOR
BIG AMOUNDS OF DATA
Slide 19
Slide 19 text
19
19
ELASTIC STACK USE CASES
Slide 20
Slide 20 text
20
20
20
BEATS TYPES
Slide 21
Slide 21 text
21
21
21
INTERACTION WITH BEATS
Slide 22
Slide 22 text
22
22
22
ELASTIC LOGGING SETUP
Slide 23
Slide 23 text
23
23
23
Kind of a tail –f over the network.
Tails the file and sends its contents
over the network.
FILEBEAT
Filebeat
Slide 24
Slide 24 text
24
24
24
A harvester - reading the content
of a single file. The harvester
reads each file, line by line, and
sends the content to the output.
HARVESTERS
FILEBEAT CONCEPTS
An input - managing harvesters &
finding all sources to read from.
INPUTS REGISTRY FILES
Registry - stores last offset a
harvester was reading from and to
ensure all log lines are sent
Slide 25
Slide 25 text
25
25
25
Perform basic processing of
events before they are sent to the
output.
PROCESSORS
FILEBEAT CONCEPTS
Templates that simplify the
collection, parsing, and
visualization of common log
formats. (ex. Apache, MySQL,
IIS etc.)
MODULES OUTPUTS
One of the many destinations
Filebeat can deliver data to.
Slide 26
Slide 26 text
26
26
26
FILEBEAT CONFIG
Sample Docker config:
Slide 27
Slide 27 text
27
27
27
FILEBEAT INPUTS AND OUTPUTS
Events
Stdin events
Docker
container logs
BSD syslog
events
Slide 28
Slide 28 text
28
LET’S SEE IT LIVE!
Slide 29
Slide 29 text
29
29
29
Aggregate, “ tail -f ” & Search
FILEBEAT
Slide 30
Slide 30 text
30
30
30
LOGSTASH
WHAT IS LOGSTASH?
Logstash is an open source, server-side data
processing pipeline that ingests data from a
multitude of sources simultaneously,
transforms it, and then sends it to your
favorite “stash.”
Slide 31
Slide 31 text
31
WHY DO WE NEED IT?
IF WE HAVE BEATS
Slide 32
Slide 32 text
32
32
32
LOGSTASH INPUTS
INGEST DATA OF ALL SHAPES,
SIZES, AND SOURCES
Logstash supports a variety of inputs that pull
in events from a multitude of common
sources, all at the same time. Easily ingest
from your logs, metrics, web applications,
data stores, and various AWS services, all in
continuous, streaming fashion.
Slide 33
Slide 33 text
33
BUT STILL … WHY?
Slide 34
Slide 34 text
34
34
34
LOGSTASH INPUTS
and more …
Slide 35
Slide 35 text
35
35
35
LOGSTASH FILTERS
PARSE & TRANSFORM
YOUR DATA ON THE FLY
As data travels from source to store, Logstash
filters parse each event, identify named fields
to build structure
Slide 36
Slide 36 text
36
36
36
LOGSTASH FILTERS
SOME INTERESTING ONES:
aggregate - Aggregates information from several events originating with a single task
cidr - Checks IP addresses against a list of network blocks
elapsed - Calculates the elapsed time between a pair of events
grok - Parses unstructured event data into fields
throttle - Throttles the number of events
uuid - Adds a UUID to events
Slide 37
Slide 37 text
37
37
37
LOGSTASH OUTPUTS
CHOOSE YOUR STASH,
TRANSPORT YOUR DATA
Logstash has a variety of outputs that let you
route data where you want
Slide 38
Slide 38 text
38
38
38
LOGSTASH CONFIG
A SAMPLE DOCKER CONFIG
Here we can see al the 3 components of a
configuration file.
Slide 39
Slide 39 text
39
39
39
MAIN PIECE OF ELASTIC STACK
An open-source search and analytics engine,
written in Java; built on Apache Lucene
Distributed: Scales to thousands of nodes
High availability: Multiple replicas for each
shard may be used
RESTful API: CRUD, monitoring etc. by JSON
HTTP calls
Powerful Query DSL: express complex queries
simply
No schema: index data with no explicit schema
(no data types and corresponding fields names
required, before indexing and parsing)
Slide 40
Slide 40 text
40
40
40
An Elasticsearch cluster is a
collection of 1 or more
nodes. Can scale to
thousands of them.
Nodes store & index data,
allow for searching it.
ELASTICSEARCH CLUSTER
ELASTICSEARCH BASIC CONCEPTS
A document is a basic unit of
information that can be
indexed.
Documents are expressed in
JSON format.
DOCUMENT
Slide 41
Slide 41 text
41
41
41
ELASTICSEARCH BASIC CONCEPTS
SHARDS & REPLICAS
Shards (aka primary
shards) are pieces of a
subdivided index.
For failover scenarios
Elasticsearch allows you
to make one or more
copies of your index’s
shards into what are
called replica shards
(aka replicas).
INDEXES & DOCUMENTS
Documents make up an index.
Documents are expressed in JSON.
Separate indexes are used for
different logically related documents.
Indexes number is not limited.
Slide 42
Slide 42 text
43
43
43
LET’S TRY IT!
Slide 43
Slide 43 text
44
44
44
We can use SLF4J as a logging façade. Additionally Logback will be
added BUT it will be enabled only in case we take care to add the
Logback-related binding from SLF4J.
All 3 things may be pulled using a single dependency:
SLF4J + BINDING + LOGBACK
SAMPLE SETUP
ch.qos.logback
logback-classic
1.0.13
Slide 44
Slide 44 text
45
45
45
In the demo 4 different logging flows are presented
DEMO SETUP
LOGGING PATTERS
Slide 45
Slide 45 text
46
46
46
DOCKER COMPOSE
Compose is a tool for defining
and running multi-container
Docker applications.
Slide 46
Slide 46 text
47
47
47
REAL PROJECT
Screenshot from real project
54.589 events logged in 15 mins
Slide 47
Slide 47 text
48
THANK YOU
VITALIE BORDINIUC
[email protected]
SENIOR JAVA DEVELOPER
http://codingideas.blog/
https://github.com/vitalieb