Elastic Stack 5.0 alpha 1-4

by Jun Ohtani

Slide 1

Slide 1 text

‹#› 2016/07 Jun Ohtani @johtani Elastic Stack 5.0 alpha1 - 4

Slide 2

Slide 2 text

Slide 3

Slide 3 text

ElasticελοΫ 3 Elastic Cloud X-Pack Kibana Elasticsearch ! " Logstash Beats +

Slide 4

Slide 4 text

4 Product Logo

Slide 5

Slide 5 text

ELK Stack

Slide 6

Slide 6 text

ELK Stack

Slide 7

Slide 7 text

Along Came Beats ELKB

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

The Elastic Stack

Slide 10

Slide 10 text

Versions

Slide 11

Slide 11 text

Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats

Slide 12

Slide 12 text

es kibana ls beats Oct 28th Nov 21st Feb 2nd 2.0 4.2 2.0 2.1 4.3 2.1 1.0 2.2 4.4 2.2 1.1 Release Bonanza

Slide 13

Slide 13 text

v 5. 0

Slide 14

Slide 14 text

THIS IS ALPHA RELEASE! DO NOT USE IN PRODUCTION

Slide 15

Slide 15 text

elasticsearch

Slide 16

Slide 16 text

elasticsearch I N G E S T Ingest Node

Slide 17

Slide 17 text

elasticsearch Painless Scripting Dynamic/ Static

Slide 18

Slide 18 text

elasticsearch Kuromoji • N-best cost / example • Not only A best token stream • Number filter

Slide 19

Slide 19 text

elasticsearch Other changes - Alpha 1 • Plugin command • bin/elasticsearch-plugin • Text/Keyword field type instead of string field • text or keyword field types / deprecated string field type

Slide 20

Slide 20 text

elasticsearch Alpha 2 / Alpha 3 • Lucene 6 • Dimensional Points / Multi-dimensional points / IPv6 • Percolate Query • Improve Indexing performance • dots in Field name

Slide 21

Slide 21 text

elasticsearch • Low-level API • Can use Java 7 • Minimal dependencies • Only accept JSON-in and JSON-out Java REST Client

Slide 22

Slide 22 text

elasticsearch Alpha 4 • Rollover / Shrink API • Wait for refresh • Search profiler support aggs • half_float field data type, only uses 16 bits • _timestamp and _ttl can not create

Slide 23

Slide 23 text

kibana

Slide 24

Slide 24 text

kibana Brings a new focus on your data New design

Slide 25

Slide 25 text

kibana Appear in the main navigation Applicaton Framework

Slide 26

Slide 26 text

kibana • Plugin command • bin/kibana-plugin • "Sense" will be "Console" (not yet available) • Sense plugin will be built into Kibana Other changes - Alpha1

Slide 27

Slide 27 text

kibana Console - Alpha2 Sense is now 'Console'

Slide 28

Slide 28 text

kibana Import from CSV - Alpha4 Import data from CSV

Slide 29

Slide 29 text

kibana Monitoring Kibana instance - Alpha4 With X-Pack

Slide 30

Slide 30 text

kibana • Disable visualization buckets and drag to sort • Fixed small bugs Other changes - Alpha4

Slide 31

Slide 31 text

logstash

Slide 32

Slide 32 text

logstash Monitor all the things Monitoring API

Slide 33

Slide 33 text

logstash Monitor all the things hot threads API

Slide 34

Slide 34 text

logstash Other changes - Alpha1 • Plugin command • bin/logstash-plugin • Kafka 0.9 support • support SSL encryption and client auth

Slide 35

Slide 35 text

logstash Alpha2 / Alpha3 • Support GeoIP2 • Support IPv6 and more information • Introduce logstash.yml • Allow to set common settings • Improve Release Packages • Support systemd • Plugin Generator

Slide 36

Slide 36 text

logstash Moar Metrics, Moar Visibility per-plugin metrics - Alpha 4

Slide 37

Slide 37 text

logstash • Improve Elasticsearch Output • Kafka 0.10 Support • Allow to set common settings • Created a new LS_HOME/data directory to store plugin states • Added hostname, http_address, version for all Monitoring APIs Alpha 4

Slide 38

Slide 38 text

beats

Slide 39

Slide 39 text

decode JSON from log lines JSON support in Filebeat

Slide 40

Slide 40 text

Filtering the exported data Custom Fields and generic ﬁltering

Slide 41

Slide 41 text

Other changes - Alpha1 • Kafka output • output to Kafka directly • Integration with Ingest Node • set "pipeline parameter" in the Elasticsearch output config • Support IP/TCP flows • report statistics like packet/byte counts

Slide 42

Slide 42 text

Alpha2 / Alpha3 • Kibana dashboards per beat • New directory layout • https://www.elastic.co/guide/en/beats/filebeat/master/directory-layout.html • Metricbeat • Collect many type of metrics include Topbeat • Add conditions to filtering • And introduce 'drop_event'

Slide 43

Slide 43 text

• Monitor MongoDB with Metricbeat • Support for gzip compression in Elasticsearch output • Ignore Symlinks log files • Kibana Dashboards for Apache Metricbeat Module Alpha4

Slide 44

Slide 44 text

Bundled set of features: Security, Alerting, Monitoring, Graph, and more … x-pack