Elastic Stack 5.0 alpha1 - alpha5

by Jun Ohtani

Slide 1

Slide 1 text

‹#› 2016/09 Jun Ohtani @johtani Elastic Stack 5.0 alpha1 - 5

Slide 2

Slide 2 text

Slide 3

Slide 3 text

ElasticελοΫ 3 Elastic Cloud X-Pack Kibana Elasticsearch ! " Logstash Beats +

Slide 4

Slide 4 text

Versions

Slide 5

Slide 5 text

Jun 9, 2015 1.6 Jul 16, 2015 1.7 Feb 19, 2015 4.0 Jun 10, 2015 4.1 May 14th, 2015 1.5 May 27th, 2015 1.0 Beta 1 July 13th, 2015 1.0 Beta 2 Sept 4 th, 2015 1.0 Beta 3 May 23, 2015 1.5 Nov 5, 2014 1.4 It’s complicated es kibana ls beats

Slide 6

Slide 6 text

es kibana ls beats Oct 28th Nov 21st Feb 2nd 2.0 4.2 2.0 2.1 4.3 2.1 1.0 2.2 4.4 2.2 1.1 Release Bonanza

Slide 7

Slide 7 text

v 5. 0

Slide 8

Slide 8 text

THIS IS ALPHA RELEASE! DO NOT USE IN PRODUCTION

Slide 9

Slide 9 text

elasticsearch

Slide 10

Slide 10 text

elasticsearch I N G E S T Ingest Node

Slide 11

Slide 11 text

elasticsearch Painless Scripting Dynamic/ Static

Slide 12

Slide 12 text

elasticsearch Kuromoji • N-best cost / example • Not only A best token stream • Number filter

Slide 13

Slide 13 text

elasticsearch Other changes - Alpha 1 • Plugin command • bin/elasticsearch-plugin • Text/Keyword field type instead of string field • text or keyword field types / deprecated string field type

Slide 14

Slide 14 text

elasticsearch Alpha 2 / Alpha 3 • Lucene 6 • Dimensional Points / Multi-dimensional points / IPv6 • Percolate Query • Improve Indexing performance • dots in Field name

Slide 15

Slide 15 text

elasticsearch • Low-level API • Can use Java 7 • Minimal dependencies • Only accept JSON-in and JSON-out Java REST Client

Slide 16

Slide 16 text

elasticsearch Alpha 4 • Rollover / Shrink API • Wait for refresh • Search profiler support aggs • half_float field data type, only uses 16 bits • _timestamp and _ttl can not create

Slide 17

Slide 17 text

elasticsearch Alpha 5 • Netty 4 • Aggregation improvements • Easy to try custom tokenizers/filters in the _analyze API

Slide 18

Slide 18 text

elasticsearch Custom analyzer in _analyze API Try custom analyzer without creating index

Slide 19

Slide 19 text

elasticsearch ES-Hadoop Alpha 5 • Support Spark 2.0!

Slide 20

Slide 20 text

kibana

Slide 21

Slide 21 text

kibana Brings a new focus on your data New design

Slide 22

Slide 22 text

kibana Appear in the main navigation Applicaton Framework

Slide 23

Slide 23 text

kibana • Plugin command • bin/kibana-plugin • "Sense" will be "Console" (not yet available) • Sense plugin will be built into Kibana Other changes - Alpha1

Slide 24

Slide 24 text

kibana Console - Alpha2 Sense is now 'Console'

Slide 25

Slide 25 text

kibana Import from CSV - Alpha4 Import data from CSV

Slide 26

Slide 26 text

kibana Monitoring Kibana instance - Alpha4 With X-Pack

Slide 27

Slide 27 text

kibana • Disable visualization buckets and drag to sort • Fixed small bugs • Tile map uses Elastic Tile Service Other changes - Alpha4 - Alpha5

Slide 28

Slide 28 text

logstash

Slide 29

Slide 29 text

logstash Monitor all the things Monitoring API

Slide 30

Slide 30 text

logstash Monitor all the things hot threads API

Slide 31

Slide 31 text

logstash Other changes - Alpha1 • Plugin command • bin/logstash-plugin • Kafka 0.9 support • support SSL encryption and client auth

Slide 32

Slide 32 text

logstash Alpha2 / Alpha3 • Support GeoIP2 • Support IPv6 and more information • Introduce logstash.yml • Allow to set common settings • Improve Release Packages • Support systemd • Plugin Generator

Slide 33

Slide 33 text

logstash Moar Metrics, Moar Visibility per-plugin metrics - Alpha 4

Slide 34

Slide 34 text

logstash • Improve Elasticsearch Output • Kafka 0.10 Support • Allow to set common settings • Created a new LS_HOME/data directory to store plugin states • Added hostname, http_address, version for all Monitoring APIs • Improve Beats input • Per column encoding support for JDBC input Alpha 4 & Alpha 5

Slide 35

Slide 35 text

beats

Slide 36

Slide 36 text

decode JSON from log lines JSON support in Filebeat

Slide 37

Slide 37 text

Filtering the exported data Custom Fields and generic ﬁltering

Slide 38

Slide 38 text

Other changes - Alpha1 • Kafka output • output to Kafka directly • Integration with Ingest Node • set "pipeline parameter" in the Elasticsearch output config • Support IP/TCP flows • report statistics like packet/byte counts

Slide 39

Slide 39 text

Alpha2 / Alpha3 • Kibana dashboards per beat • New directory layout • https://www.elastic.co/guide/en/beats/filebeat/master/directory-layout.html • Metricbeat • Collect many type of metrics include Topbeat • Add conditions to filtering • And introduce 'drop_event'

Slide 40

Slide 40 text

• Monitor MongoDB with Metricbeat • Support for gzip compression in Elasticsearch output • Ignore Symlinks log files • Kibana Dashboards for Apache Metricbeat Module • Filtering flexibility based on conditions Alpha 4 & Alpha 5

Slide 41

Slide 41 text

Bundled set of features: Security, Alerting, Monitoring, Graph, and more … x-pack