Introduction to Elasticsearch

Slide 1

Slide 1 text

Introduction to Elasticsearch Igor Motov Software Engineer 03/25/2015

Slide 2

Slide 2 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 2 Elastic • Founded in 2012 • By the people behind the Elasticsearch and Apache Lucene • Renamed to Elastic in 2015 • Headquarters: Amsterdam and Mountain View, CA • We provide • Training (public & onsite) • Development support • Production support • Hosted elasticsearch

Slide 3

Slide 3 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 Enough about me, let’s talk about you for a bit!

Slide 4

Slide 4 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 Have you used Elasticsearch recently?

Slide 5

Slide 5 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 Have you used elasticsearch recently?

Slide 6

Slide 6 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Some Elasticsearch users

Slide 7

Slide 7 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 What’s Elasticsearch • Real time search and analytics engine • JSON-oriented, Apache Lucene-based • Automatic Schema Detection • With full control of schema when needed • Distributed • Scales Up+Out, Highly Available • Multi-tenancy • Dynamically create/delete indices • API centric • Most functionality is exposed through an API

Slide 8

Slide 8 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 Elasticsearch Documents • Identified by index/type/id • JSON Document: { "author": "imotov", "title": "Introduction to Elasticsearch", "post_date": "2015-03-25", "size": 30 }

Slide 9

Slide 9 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 Elasticsearch Download

Slide 10

Slide 10 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 Elasticsearch distribution . ├── LICENSE.txt ├── NOTICE.txt ├── README.textile ├── bin │ ├── elasticsearch │ ├── elasticsearch.in.sh │ └── plugin ├── config │ ├── elasticsearch.yml │ └── logging.yml ├── data │ └── elasticsearch ├── lib │ ├── elasticsearch-x.y.z.jar │ ├── ... │ └── └── logs ├── elasticsearch.log └── elasticsearch_index_search_slowlog.log

Slide 11

Slide 11 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 DEMO

Slide 12

Slide 12 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 Communicating with Elasticsearch • REST Protocol • Binary Protocol

Slide 13

Slide 13 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 13 Communicating with Elasticsearch • REST API • JavaScript • Perl • PHP • Python • Ruby • .NET • Binary Protocol • Java • Groovy

Slide 14

Slide 14 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 It’s not just search

Slide 15

Slide 15 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 Aggregations

Slide 16

Slide 16 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 Aggregations

Slide 17

Slide 17 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 Aggregations • Group by • values • ranges • histogram • time intervals (years/months/days…) • IP ranges • geo location • …

Slide 18

Slide 18 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 • Calculate • counts • max • min • avg • sum • standard deviation (with bounds) • variance • sum of squares • percentile rank

Slide 19

Slide 19 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 DEMO

Slide 20

Slide 20 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 Aggregations

Slide 21

Slide 21 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 21 On a large scale!

Slide 22

Slide 22 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 Let’s talk about scale and high availability.

Slide 23

Slide 23 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 23 Scale and High Availability node1 0 1 2 3 4 hash(id)%5 M

Slide 24

Slide 24 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 High Availability node1 0 1 2 3 4 node2 M

Slide 25

Slide 25 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 25 High Availability node1 0 1 2 3 4 node2 0 1 2 3 4 M

Slide 26

Slide 26 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 26 High Availability node1 0 1 2 3 4 hash(id%5) node1 0 1 2 3 4 M

Slide 27

Slide 27 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 27 Scale node1 0 2 4 node2 1 2 3 node3 0 1 3 4 M

Slide 28

Slide 28 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 28 Scale node1 0 2 4 node2 1 2 3 node3 0 1 3 4 M

Slide 29

Slide 29 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 29 Scale node1 0 2 node2 1 2 3 node3 1 3 4 0 4 M

Slide 30

Slide 30 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 30 Scale node1 0 2 node2 1 2 3 node3 1 3 4 0 4 M

Slide 31

Slide 31 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 31 Master is gone node2 1 2 3 node3 3 4 0 4

Slide 32

Slide 32 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 32 Master is elected node2 1 2 3 node3 3 4 0 4 M

Slide 33

Slide 33 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 33 Primary are promoted node2 1 2 3 node3 3 4 0 4 M

Slide 34

Slide 34 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 34 High Availability node2 1 2 3 node3 3 4 0 4 M

Slide 35

Slide 35 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 35 High Availability is restored node2 1 2 3 node3 3 4 0 4 M 1 2 0

Slide 36

Slide 36 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 36 Elasticsearch ecosystem

Slide 37

Slide 37 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 37 Elasticsearch Ecosystem

Slide 38

Slide 38 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 38 Elasticsearch Ecosystem logstash

Slide 39

Slide 39 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 39 Elasticsearch Ecosystem logstash

Slide 40

Slide 40 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Shield 40 Elasticsearch Ecosystem logstash

Slide 41

Slide 41 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 41 Elasticsearch Resources

Slide 42

Slide 42 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 42 Elasticsearch - The Definitive Guide http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html

Slide 43

Slide 43 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 43 Elasticsearch Meetups

Slide 44

Slide 44 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 44 http://www.meetup.com/Elasticsearch-Boston/ Boston Elasticsearch Meetup

Slide 45

Slide 45 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 45

Slide 46

Slide 46 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 46 Elasticsearch Training http://purchases.elastic.co/

Slide 47

Slide 47 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 47 Questions?

Slide 48

Slide 48 text

Thank you! www.elastic.co [email protected] @imotov github.com/imotov