Introduction to Elasticsearch

New England Search Technologies (NEST) Group Meetup

Transcript

1. Introduction to
   Elasticsearch
   Igor Motov
   Software Engineer
   03/25/2015
   

   View Slide

2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   2
   Elastic
   • Founded in 2012
   • By the people behind the Elasticsearch and Apache
   Lucene
   • Renamed to Elastic in 2015
   • Headquarters: Amsterdam and Mountain View, CA
   • We provide
   • Training (public & onsite)
   • Development support
   • Production support
   • Hosted elasticsearch
   

   View Slide

3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   3
   Enough about me, let’s talk
   about you for a bit!
   

   View Slide

4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   4
   Have you used Elasticsearch
   recently?
   

   View Slide

5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   5
   Have you used elasticsearch recently?
   

   View Slide

6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   6
   Some Elasticsearch users
   

   View Slide

7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   7
   What’s Elasticsearch
   • Real time search and analytics engine
   • JSON-oriented, Apache Lucene-based
   • Automatic Schema Detection
   • With full control of schema when needed
   • Distributed
   • Scales Up+Out, Highly Available
   • Multi-tenancy
   • Dynamically create/delete indices
   • API centric
   • Most functionality is exposed through an API
   

   View Slide

8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   8
   Elasticsearch Documents
   • Identified by index/type/id
   • JSON Document:
   {
   "author": "imotov",
   "title": "Introduction to Elasticsearch",
   "post_date": "2015-03-25",
   "size": 30
   }
   

   View Slide

9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   9
   Elasticsearch Download
   

   View Slide

10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    10
    Elasticsearch distribution
    .
    ├── LICENSE.txt
    ├── NOTICE.txt
    ├── README.textile
    ├── bin
    │ ├── elasticsearch
    │ ├── elasticsearch.in.sh
    │ └── plugin
    ├── config
    │ ├── elasticsearch.yml
    │ └── logging.yml
    ├── data
    │ └── elasticsearch
    ├── lib
    │ ├── elasticsearch-x.y.z.jar
    │ ├── ...
    │ └──
    └── logs
    ├── elasticsearch.log
    └── elasticsearch_index_search_slowlog.log
    

    View Slide

11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    11
    DEMO
    

    View Slide

12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    12
    Communicating with Elasticsearch
    • REST Protocol • Binary Protocol
    

    View Slide

13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    13
    Communicating with Elasticsearch
    • REST API
    • JavaScript
    • Perl
    • PHP
    • Python
    • Ruby
    • .NET
    • Binary Protocol
    • Java
    • Groovy
    

    View Slide

14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    14
    It’s not just search
    

    View Slide

15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    15
    Aggregations
    

    View Slide

16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    16
    Aggregations
    

    View Slide

17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    17
    Aggregations
    • Group by
    • values
    • ranges
    • histogram
    • time intervals (years/months/days…)
    • IP ranges
    • geo location
    • …
    

    View Slide

18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    18
    • Calculate
    • counts
    • max
    • min
    • avg
    • sum
    • standard deviation (with bounds)
    • variance
    • sum of squares
    • percentile rank
    

    View Slide

19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    19
    DEMO
    

    View Slide

20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    20
    Aggregations
    

    View Slide

21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    21
    On a large scale!
    

    View Slide

22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    22
    Let’s talk about scale
    and high availability.
    

    View Slide

23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    23
    Scale and High Availability
    node1
    0 1 2 3 4
    hash(id)%5
    M
    

    View Slide

24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    24
    High Availability
    node1
    0 1 2 3 4
    node2
    M
    

    View Slide

25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    25
    High Availability
    node1
    0 1 2 3 4
    node2
    0 1 2 3 4
    M
    

    View Slide

26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    26
    High Availability
    node1
    0 1 2 3 4
    hash(id%5)
    node1
    0 1 2 3 4
    M
    

    View Slide

27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    27
    Scale
    node1
    0 2 4
    node2
    1 2 3
    node3
    0
    1 3 4
    M
    

    View Slide

28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    28
    Scale
    node1
    0 2 4
    node2
    1 2 3
    node3
    0
    1
    3 4
    M
    

    View Slide

29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    29
    Scale
    node1
    0 2
    node2
    1 2 3
    node3
    1
    3 4
    0
    4
    M
    

    View Slide

30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    30
    Scale
    node1
    0 2
    node2
    1 2 3
    node3
    1
    3 4 0 4
    M
    

    View Slide

31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    31
    Master is gone
    node2
    1 2 3
    node3
    3 4 0 4
    

    View Slide

32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    32
    Master is elected
    node2
    1 2 3
    node3
    3 4 0 4
    M
    

    View Slide

33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    33
    Primary are promoted
    node2
    1 2 3
    node3
    3 4 0 4
    M
    

    View Slide

34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    34
    High Availability
    node2
    1 2 3
    node3
    3 4 0 4
    M
    

    View Slide

35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    node4
    35
    High Availability is restored
    node2
    1 2 3
    node3
    3 4 0 4
    M
    1 2
    0
    

    View Slide

36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    36
    Elasticsearch
    ecosystem
    

    View Slide

37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    37
    Elasticsearch Ecosystem
    

    View Slide

38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    38
    Elasticsearch Ecosystem
    logstash
    

    View Slide

39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    39
    Elasticsearch Ecosystem
    logstash
    

    View Slide

40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    Shield
    40
    Elasticsearch Ecosystem
    logstash
    

    View Slide

41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    41
    Elasticsearch
    Resources
    

    View Slide

42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    42
    Elasticsearch - The Definitive Guide
    http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
    

    View Slide

43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    43
    Elasticsearch Meetups
    

    View Slide

44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    44
    http://www.meetup.com/Elasticsearch-Boston/
    Boston Elasticsearch Meetup
    

    View Slide

45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    45
    

    View Slide

46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    46
    Elasticsearch Training
    http://purchases.elastic.co/
    

    View Slide

47. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    47
    Questions?
    

    View Slide

48. Thank you!
    www.elastic.co
    igor@elastic.co
    @imotov
    github.com/imotov
    

    View Slide