Elasticsearch (R)Evolution — You Know, for Search...

Elasticsearch is a distributed, RESTful search and analytics engine built on top of Apache Lucene. After the initial release in 2010 it has become the most widely used full-text search engine, but it is not stopping there.

The revolution happened and now it is time for evolution. We dive into the following questions:
* How did leniency help the initial adoption, but why and how do we lean more on strictness today?
* How can upgrades be improved to avoid any downtime even when changing major versions?
* How can new resiliency features improve recovery scenarios and add totally new features?
* Why are types finally disappearing and how are we are trying to avoid the upgrade pain as much as possible?
* What are examples for some clever performance improvements?
* How can you shrink and (finally) split shards in a highly efficient way?

Attendees learn both about new and upcoming features as well as the motivation and engineering challenges behind them.

Transcript

1. (R)Evolution
   Philipp Krenn̴̴̴̴̴̴̴̴@xeraa
   

   View Slide

2. Developer
   

   View Slide

3. Revolution
   

   View Slide

4. https://db-engines.com/en/ranking
   

   View Slide

5. View Slide

6. Who uses Elasticsearch?
   

   View Slide

7. View Slide

8. Terms
   Cluster, Node, Index, Shard, Document, ID
   

   View Slide

9. $ curl http://localhost:9200
   {
   "name": "elasticsearch1",
   "cluster_name": "docker-cluster",
   "cluster_uuid": "NflLphonSp2QPKx2llxkgQ",
   "version": {
   "number": "5.6.14",
   "build_hash": "f310fe9",
   "build_date": "2018-12-05T21:20:16.416Z",
   "build_snapshot": false,
   "lucene_version": "6.6.1"
   },
   "tagline": "You Know, for Search"
   }
   

   View Slide

10. $ curl http://localhost:9200
    {
    "name": "elasticsearch1",
    "cluster_name": "docker-cluster",
    "cluster_uuid": "NflLphonSp2QPKx2llxkgQ",
    "version": {
    "number": "5.6.14",
    "build_hash": "f310fe9",
    "build_date": "2018-12-05T21:20:16.416Z",
    "build_snapshot": false,
    "lucene_version": "6.6.1"
    },
    "tagline": "You Know, for Search"
    }
    

    View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. Evolution
    

    View Slide

19. Rolling Upgrades*
    6.0
    * Demo
    

    View Slide

20. View Slide

21. PS: Kibana 6.6
    elasticsearch.url
    elasticsearch.hosts
    

    View Slide

22. Floodstage Watermark*
    6.0
    * Demo
    

    View Slide

23. View Slide

24. Low 85%
    High 90%
    Floodstage 95%
    

    View Slide

25. Sequence Numbers*
    6.0
    * Demo
    

    View Slide

26. View Slide

27. View Slide

28. 63 bits ought to be
    enough for anyone.
    

    View Slide

29. Tradeoff
    index.translog.retention.size: 512MB
    index.translog.retention.age: 12h
    

    View Slide

30. Cross Datacenter Replication
    6.5
    

    View Slide

31. Types*
    5.6 to 8.0
    * Demo
    

    View Slide

32. View Slide

33. Why
    Data types
    Sparsity
    Scoring
    

    View Slide

34. How
    5.6 opt-in for single type
    6.x single type for new indices
    7.x types optional
    fixed API PUT {index}/_doc/{id}
    include_type_name: true for GET|PUT _mapping
    8.x no more types
    https://www.elastic.co/guide/en/elasticsearch/reference/current/
    removal-of-types.html#_schedule_for_removal_of_mapping_types
    

    View Slide

35. Shrink & Split*
    ̴5.0̴̴̴6.1
    * Demo
    

    View Slide

36. View Slide

37. Shrink
    Combine shards by a factor
    

    View Slide

38. View Slide

39. Split
    Split into a factor of number_of_routing_shards
    Not required in 7.0+
    

    View Slide

40. Shards
    7.0
    

    View Slide

41. View Slide

42. Default: 1 Shard per Index
    Oversharding
    

    View Slide

43. Simon Says
    Use a single shard until it
    blows up
    

    View Slide

44. Conclusion
    

    View Slide

45. Don't be too Shay
    to improve
    

    View Slide

46. Rolling Upgrades
    Floodstage Watermark
    Sequence Numbers
    

    View Slide

47. Types
    Shrink & Split
    Default Shard Number
    

    View Slide

48. Questions?
    Philipp Krenn̴̴̴̴̴̴̴@xeraa
    

    View Slide