Introduction to Elasticsearch

New England Search Technologies (NEST) Group Meetup

Transcript

1. Introduction to Elasticsearch Igor Motov Software Engineer 03/25/2015
2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 2 Elastic • Founded in 2012 • By the people behind the Elasticsearch and Apache Lucene • Renamed to Elastic in 2015 • Headquarters: Amsterdam and Mountain View, CA • We provide • Training (public & onsite) • Development support • Production support • Hosted elasticsearch
3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 Enough about me, let’s talk about you for a bit!
4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 Have you used Elasticsearch recently?
5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 Have you used elasticsearch recently?
6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Some Elasticsearch users
7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 What’s Elasticsearch • Real time search and analytics engine • JSON-oriented, Apache Lucene-based • Automatic Schema Detection • With full control of schema when needed • Distributed • Scales Up+Out, Highly Available • Multi-tenancy • Dynamically create/delete indices • API centric • Most functionality is exposed through an API
8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 Elasticsearch Documents • Identified by index/type/id • JSON Document: { "author": "imotov", "title": "Introduction to Elasticsearch", "post_date": "2015-03-25", "size": 30 }
9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 Elasticsearch Download
10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 Elasticsearch distribution . ├── LICENSE.txt ├── NOTICE.txt ├── README.textile ├── bin │ ├── elasticsearch │ ├── elasticsearch.in.sh │ └── plugin ├── config │ ├── elasticsearch.yml │ └── logging.yml ├── data │ └── elasticsearch ├── lib │ ├── elasticsearch-x.y.z.jar │ ├── ... │ └── └── logs ├── elasticsearch.log └── elasticsearch_index_search_slowlog.log
11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 DEMO
12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 Communicating with Elasticsearch • REST Protocol • Binary Protocol
13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 13 Communicating with Elasticsearch • REST API • JavaScript • Perl • PHP • Python • Ruby • .NET • Binary Protocol • Java • Groovy
14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 It’s not just search
15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 Aggregations
16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 Aggregations
17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 Aggregations • Group by • values • ranges • histogram • time intervals (years/months/days…) • IP ranges • geo location • …
18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 • Calculate • counts • max • min • avg • sum • standard deviation (with bounds) • variance • sum of squares • percentile rank
19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 DEMO
20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 Aggregations
21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 21 On a large scale!
22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 Let’s talk about scale and high availability.
23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 23 Scale and High Availability node1 0 1 2 3 4 hash(id)%5 M
24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 High Availability node1 0 1 2 3 4 node2 M
25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 25 High Availability node1 0 1 2 3 4 node2 0 1 2 3 4 M
26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 26 High Availability node1 0 1 2 3 4 hash(id%5) node1 0 1 2 3 4 M
27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 27 Scale node1 0 2 4 node2 1 2 3 node3 0 1 3 4 M
28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 28 Scale node1 0 2 4 node2 1 2 3 node3 0 1 3 4 M
29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 29 Scale node1 0 2 node2 1 2 3 node3 1 3 4 0 4 M
30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 30 Scale node1 0 2 node2 1 2 3 node3 1 3 4 0 4 M
31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 31 Master is gone node2 1 2 3 node3 3 4 0 4
32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 32 Master is elected node2 1 2 3 node3 3 4 0 4 M
33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 33 Primary are promoted node2 1 2 3 node3 3 4 0 4 M
34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 34 High Availability node2 1 2 3 node3 3 4 0 4 M
35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 35 High Availability is restored node2 1 2 3 node3 3 4 0 4 M 1 2 0
36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 36 Elasticsearch ecosystem
37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 37 Elasticsearch Ecosystem
38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 38 Elasticsearch Ecosystem logstash
39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 39 Elasticsearch Ecosystem logstash
40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Shield 40 Elasticsearch Ecosystem logstash
41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 41 Elasticsearch Resources
42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 42 Elasticsearch - The Definitive Guide http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 43 Elasticsearch Meetups
44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 44 http://www.meetup.com/Elasticsearch-Boston/ Boston Elasticsearch Meetup
45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 45
46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 46 Elasticsearch Training http://purchases.elastic.co/
47. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 47 Questions?
48. Thank you! www.elastic.co igor@elastic.co @imotov github.com/imotov