What's coming in 2.x

An overview of the themes and features that we are planning to deliver in the 2.x branch

Transcript

1. Elasticsearch 2.0 and beyond Simon Willnauer & Clinton Gormley
2. { } CC-BY-ND 4.0 Feb 8, 2010 2
3. { } CC-BY-ND 4.0 Feb 8, 2010 3
4. { } CC-BY-ND 4.0 Feb 8, 2010 4 Elasticsearch v0.04 released
5. { } CC-BY-ND 4.0 Feb 9, 2010 5
6. { } CC-BY-ND 4.0 Feb 10, 2010 6
7. { } CC-BY-ND 4.0 Feb 11, 2010 7 First user ticket on GitHub
8. { } CC-BY-ND 4.0 Feb 12, 2010 8
9. { } CC-BY-ND 4.0 Feb 13, 2010 9
10. { } CC-BY-ND 4.0 Feb 14, 2010 10 First Elasticsearch REST API
11. { } CC-BY-ND 4.0 0.05 … 0.20
 Mar 2010 … Dec 2012 11 stats APIs, index aliases, highlighting, local gateway, translog, facets, geo point, geo shape, scripting, date math, dynamic settings, bulk APIs, delete/open/close indices, index templates, dynamic mapping, parent/ child, nested, versioning, percolator, realtime GET, min_master_nodes, cluster reroute, phrase suggester
12. { } CC-BY-ND 4.0 0.90 … 1.3
 Apr 2013 … Jul 2014 12 Lucene 4, Doc values, Aggregations, Distributed percolator, Function score, Snapshot/Restore, Tribe node, Circuit breakers, Streaming aggs, Significant terms, Completion suggester, Cross fields matching
13. { } CC-BY-ND 4.0 1.4 … 1.5
 Nov 2014 … Mar 2015 13
14. { } CC-BY-ND 4.0 14 Safety first! 1.4 … 1.5
 Nov 2014 … Mar 2015
15. { } CC-BY-ND 4.0 15 Prevent Elasticsearch from hurting you 1.4 … 1.5
 Nov 2014 … Mar 2015
16. { } CC-BY-ND 4.0 16 Checksums, checksums, checksums… Refactoring & simplification Randomised testing 1.4 … 1.5
 Nov 2014 … Mar 2015
17. { } CC-BY-ND 4.0 17 So what's coming in 2.x?

18. { } CC-BY-ND 4.0 18 So what's coming in 2.x?
 Safety first!
19. { } CC-BY-ND 4.0 19 So what's coming in 2.x?
 Prevent You from hurting Yourself
20. { } CC-BY-ND 4.0 20 Stricter field/type mappings
21. { } CC-BY-ND 4.0 21 Stricter field/type mappings Best practice
22. { } CC-BY-ND 4.0 22 Stricter field/type mappings Best practice → Enforced rules
23. { } CC-BY-ND 4.0 23 Stricter field/type mappings same field name   same index different types
24. { } CC-BY-ND 4.0 24 Stricter field/type mappings } same field name   same index different types same mapping
25. { } CC-BY-ND 4.0 25 Stricter everywhere
26. { } CC-BY-ND 4.0 26 Stricter everywhere Don't silently ignore errors
27. { } CC-BY-ND 4.0 27 Stricter everywhere If something is wrong, complain about it!
28. { } CC-BY-ND 4.0 28 Stricter everywhere If something is deprecated, complain about it!
29. { } CC-BY-ND 4.0 29 Stricter everywhere And complain legibly…
30. { } CC-BY-ND 4.0 30 {        "error":  "SearchPhaseExecutionException[Failed  to  execute  phase  [query],  all  shards   failed;  shardFailures  {[iFZT3lrSRy-­‐PGybC15R4Cw][t][0]:  SearchParseException[[t][0]:   from[-­‐1],size[-­‐1]:  Parse  Failure  [Failed  to  parse  source  [{\n    \"query\":  {\n        \"mathc\":   {\n            \"foo\":  \"bar\"\n        }\n    }\n}\n]]];  nested:  QueryParsingException[[t]  No  query   registered  for  [mathc]];  }{[iFZT3lrSRy-­‐PGybC15R4Cw][t][1]:  SearchParseException[[t][1]:   from[-­‐1],size[-­‐1]:  Parse  Failure  [Failed  to  parse  source  [{\n    \"query\":  {\n        \"mathc\":   {\n            \"foo\":  \"bar\"\n        }\n    }\n}\n]]];  nested:  QueryParsingException[[t]  No  query   registered  for  [mathc]];  }{[iFZT3lrSRy-­‐PGybC15R4Cw][t][2]:  SearchParseException[[t][2]:   from[-­‐1],size[-­‐1]:  Parse  Failure  [Failed  to  parse  source  [{\n    \"query\":  {\n        \"mathc\":   {\n            \"foo\":  \"bar\"\n        }\n    }\n}\n]]];  nested:  QueryParsingException[[t]  No  query   registered  for  [mathc]];  }{[iFZT3lrSRy-­‐PGybC15R4Cw][t][3]:  SearchParseException[[t][3]:   from[-­‐1],size[-­‐1]:  Parse  Failure  [Failed  to  parse  source  [{\n    \"query\":  {\n        \"mathc\":   {\n            \"foo\":  \"bar\"\n        }\n    }\n}\n]]];  nested:  QueryParsingException[[t]  No  query   registered  for  [mathc]];  }{[iFZT3lrSRy-­‐PGybC15R4Cw][t][4]:  SearchParseException[[t][4]:   from[-­‐1],size[-­‐1]:  Parse  Failure  [Failed  to  parse  source  [{\n    \"query\":  {\n        \"mathc\":   {\n            \"foo\":  \"bar\"\n        }\n    }\n}\n]]];  nested:  QueryParsingException[[t]  No  query   registered  for  [mathc]];  }]",        "status":  400   }
31. { } CC-BY-ND 4.0 31 {      "error":  {          "type":      "QueryParsingException",          "reason":  "No  query  registered  for  [mathc]",          "offset":  20,          "source":  "{\n    \"query\":  {\n        \"mathc\":  {\n            \"foo\":  \"bar\"\n        }\n    }\n}\n"      }   }  
32. { } CC-BY-ND 4.0 32 Heap is limited… …use it wisely
33. { } CC-BY-ND 4.0 33 Biggest heap user: Fielddata At query time, loads field values into heap, for aggregations, sorting, parent/child
34. { } CC-BY-ND 4.0 34 Disk-based Fielddata Written at index time Loaded into file system cache Managed by operating system
35. { } CC-BY-ND 4.0 35 Disk-based Fielddata Currently, opt in
36. { } CC-BY-ND 4.0 36 2.x: Enabled by default Disk-based Fielddata
37. { } CC-BY-ND 4.0 37 Make things simpler
38. { } CC-BY-ND 4.0 38 What should merge throttling be set to? Make things simpler
39. { } CC-BY-ND 4.0 39 What should merge throttling be set to? How many merge threads should we enable? Make things simpler
40. { } CC-BY-ND 4.0 40 Don't know? Make things simpler
41. { } CC-BY-ND 4.0 41 You shouldn't care! Make things simpler
42. { } CC-BY-ND 4.0 42 Replace settings with auto-regulating feedback loops Make things simpler
43. { } CC-BY-ND 4.0 43 Writing a query?
44. { } CC-BY-ND 4.0 44 Should I use a query or a filter here?
 Writing a query?
45. { } CC-BY-ND 4.0 45 Should I use a query or a filter here?
 Which filters should be cached? Or not? Writing a query?
46. { } CC-BY-ND 4.0 46 Should I use a query or a filter here?
 Which filters should be cached? Or not? Bool? And? Or? Not? Writing a query?
47. { } CC-BY-ND 4.0 47 Should I use a query or a filter here?
 Which filters should be cached? Or not? Bool? And? Or? Not? Optimal order of execution? Writing a query?
48. { } CC-BY-ND 4.0 48 Don't know? Make things simpler
49. { } CC-BY-ND 4.0 49 You shouldn't care! Make things simpler
50. { } CC-BY-ND 4.0 50 Queries and filters are merging Make query DSL simpler
51. { } CC-BY-ND 4.0 51 Queries and filters are merging query  =  filter  +  scoring Make query DSL simpler
52. { } CC-BY-ND 4.0 52 Any clause can be cached as filter Make query DSL simpler
53. { } CC-BY-ND 4.0 53 Any clause can be cached as filter Auto-cached when makes sense Make query DSL simpler
54. { } CC-BY-ND 4.0 54 Make things more powerful
55. { } CC-BY-ND 4.0 55 simple facets → hierarchical aggregations → approximate streaming → significant terms Make things more powerful
56. { } CC-BY-ND 4.0 56 Reducers
57. { } CC-BY-ND 4.0 57 Computation on aggregation results Reducers
58. { } CC-BY-ND 4.0 58
59. { } CC-BY-ND 4.0 59
60. { } CC-BY-ND 4.0 60
61. { } CC-BY-ND 4.0 61
62. { } CC-BY-ND 4.0 62
63. { } CC-BY-ND 4.0 63
64. { } CC-BY-ND 4.0 64
65. { } CC-BY-ND 4.0 65
66. { } CC-BY-ND 4.0 66 Even more powerful with scripting… Reducers
67. { } CC-BY-ND 4.0 67 Scripting
68. { } CC-BY-ND 4.0 68 Losing Groovy was a blow Scripting
69. { } CC-BY-ND 4.0 69 Lucene expressions fast and safe, but too simple Scripting
70. { } CC-BY-ND 4.0 70 We're expanding expressions! Safe, fast mini-language Scripting
71. { } CC-BY-ND 4.0 71 Core changes
72. { } CC-BY-ND 4.0 72 Data structure on every node Cluster state
73. { } CC-BY-ND 4.0 73 Data structure on every node Indices, Settings, Mappings, Nodes, Shard allocations, etc Cluster state
74. { } CC-BY-ND 4.0 74 Only master can publish new cluster state Cluster state
75. { } CC-BY-ND 4.0 PUT  /test [CS1] → 5 primary shards assigned to 5 nodes [CS2] → Primary shard 1 started on Node 3 [CS3] → Primary shard 2 started on Node 4 [CS4] → Primary shard 3 started on Node 1 [CS5] → Primary shard 0 started on Node 2 [CS6] → Primary shard 4 started on Node 5 [CS7] → Field "title" added to mapping …
76. { } CC-BY-ND 4.0 76 Cluster state more nodes more shards more fields }more, bigger updates
77. { } CC-BY-ND 4.0 77 Cluster state diffs
78. { } CC-BY-ND 4.0 78 Cluster state diffs Lightweight updates Faster publishing Faster cluster restarts
79. { } CC-BY-ND 4.0 79 Cluster state diffs Lays groundwork for…
80. { } CC-BY-ND 4.0 80 Task management API
81. { } CC-BY-ND 4.0 81 Control long-running tasks Task management API
82. { } CC-BY-ND 4.0 82 Control long-running tasks Status, Pause, Restart, Abort Task management API
83. { } CC-BY-ND 4.0 83 Reindex API
84. { } CC-BY-ND 4.0 84 Reindex API old index to new → update mappings
85. { } CC-BY-ND 4.0 85 Reindex API old index to new into same index → update mappings → update by query
86. { } CC-BY-ND 4.0 86 old index to new into same index transform data Reindex API → update mappings → update by query → extract new fields
87. { } CC-BY-ND 4.0 87 old index to new into same index transform data local or remote cluster Reindex API → update mappings → update by query → extract new fields
88. { } CC-BY-ND 4.0 88 Changes API
89. { } CC-BY-ND 4.0 89 like scan/scroll, but… Changes API
90. { } CC-BY-ND 4.0 90 like scan/scroll, but… can "follow" ongoing changes Changes API
91. { } CC-BY-ND 4.0 91 combined with reindex API Changes API
92. { } CC-BY-ND 4.0 92 combined with reindex API live copy to remote cluster Changes API
93. { } CC-BY-ND 4.0 93 combined with reindex API "rollup" related data, e.g. clickstream → user sessions Changes API
94. { } CC-BY-ND 4.0 94 Faster recovery
95. { } CC-BY-ND 4.0 95 Recovery is heavy often copy entire shards
96. { } CC-BY-ND 4.0 96 Recovery is heavy node failure/reboot triggers shard reallocation storm
97. { } CC-BY-ND 4.0 97 Recovery is heavy upgrading is painful
98. { } CC-BY-ND 4.0 98 Faster recovery almost-instant recovery of existing replica shards
99. { } CC-BY-ND 4.0 99 Faster recovery more lenient reallocation avoid reshuffling
100. { } CC-BY-ND 4.0 100 Faster recovery rolling upgrades should be easy
101. { } CC-BY-ND 4.0 101 Elastic is a family
102. { } CC-BY-ND 4.0 102 ./bin/plugin  install  kibana   Elastic is a family
103. { } CC-BY-ND 4.0 103 ./bin/plugin  install  kibana   >    Installed  elastic-­‐ui   >    Installed  kibana   Elastic is a family
104. { } CC-BY-ND 4.0 104 ./bin/elastic  start   Elastic is a family
105. { } CC-BY-ND 4.0 105 ./bin/elastic  start   >    Started  elasticsearch   >    Started  elastic-­‐ui   Elastic is a family
106. { } CC-BY-ND 4.0 106 curl  localhost:9200  #  Elasticsearch   open  localhost:5601  #  Kibana   Elastic is a family
107. { } Thank you          @s1m0nw      @clintongormley
108. { } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0