What's Evolving in Elasticsearch

Elastic's Clinton Gormley and Simon Willnauer present the latest happenings in Elasticsearch at Elastic{ON}16, February 17, 2016, in San Francisco.

1. ‹#› Clinton Gormley & Simon Willnauer @clintongormley @s1m0nw What's evolving in Elasticsearch
2. 2 0.x 1.x 2.x 5.x
3. 3 CRUD, Mapping, Search, Query DSL, Highlighting, Aliases, Cluster / Nodes / Indices Stats, Facets, Scripting, Index Templates, Dynamic Mapping, Parent/Child, Nested Objects, Realtime GET, Versioning, Routing, Percolator, Geo-points, Geo-shapes, Suggesters 0.x 1.x 2.x 5.x
4. 4 0.x 1.x 2.x 5.x Aggregations, CAT API, Distributed Percolator, Doc Values, Snapshot/Restore, Tribe Node, Circuit Breakers, Search Templates, Completion Suggester, Lucene Scripting, Top Hits
5. 5 0.x 1.x 2.x 5.x Aggregations, CAT API, Distributed Percolator, Doc Values, Snapshot/Restore, Tribe Node, Circuit Breakers, Search Templates, Completion Suggester, Lucene Scripting, Top Hits OOM Corruption Split Brain Security Exploits
6. 6 0.x 1.x 2.x 5.x Aggregations, CAT API, Distributed Percolator, Doc Values, Snapshot/Restore, Tribe Node, Circuit Breakers, Search Templates, Completion Suggester, Lucene Scripting, Top Hits
7. 7 Cluster Resiliency, Checksums everywhere, Lock down scripting, Make doc values fast 0.x 1.x 1.4 5.x 2.x
8. 8 Make rolling restarts easy: Synced flush, Delayed allocation, Cancellable allocation 0.x 1.x 1.4 5.x 2.x
9. ‹#› A rolling restart of an #elasticsearch cluster reduced from 8 days to 30 minutes! @elastic win! #elasticon Ash Kapow - @ashkapow
10. 10 Fix All the Things! 0.x 1.x 1.4 5.x 2.x
11. ‹#› Flexible 11 Great Mappings Refactoring Image by Abraham Ortelius - The Library of Congress, Public Domain, https://commons.wikimedia.org/w/ index.php?curid=6872417
12. ‹#› Too Flexible… 12 Great Mappings Refactoring Image by Abraham Ortelius - The Library of Congress, Public Domain, https://commons.wikimedia.org/w/ index.php?curid=6872417
13. ‹#› Too Flexible Ambiguous Silent failures Index corruption 13 Great Mappings Refactoring Image by Abraham Ortelius - The Library of Congress, Public Domain, https://commons.wikimedia.org/w/ index.php?curid=6872417
14. ‹#› Now Consistent Reliable Solid foundation 14 Great Mappings Refactoring Image by Abraham Ortelius - The Library of Congress, Public Domain, https://commons.wikimedia.org/w/ index.php?curid=6872417
15. ‹#› 15 Translog reliability Image licensed under CC BY-SA 3.0, https:// commons.wikimedia.org/w/index.php?curid=157542
16. ‹#› 16 Translog reliability • Silent corruption • Data loss • Confusing semantics Image licensed under CC BY-SA 3.0, https:// commons.wikimedia.org/w/index.php?curid=157542
17. ‹#› 17 Translog reliability • Durable by default • Atomic writes • Fsync after each request Image licensed under CC BY-SA 3.0, https:// commons.wikimedia.org/w/index.php?curid=157542
18. ‹#› 18 Usability
19. ‹#› Query/filter merging 19 Usability
20. ‹#› Query/filter merging Query autocaching 20 Usability
21. ‹#› Query/filter merging Query autocaching Merge autothrottling 21 Usability
22. ‹#› Query/filter merging Query autocaching Merge autothrottling Structured exceptions 22 Usability
23. ‹#› Query/filter merging Query autocaching Merge autothrottling Structured exceptions Admin safeguards 23 Usability
24. ‹#› 24 Heap usage Better data structures Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
25. ‹#› 25 Fast Doc-values Columnar store File-system cache On by default Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Heap usage Better data structures
26. ‹#› 26 Field length norms Relevance statistic File-system cache Sparse data structure Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Heap usage Better data structures
27. ‹#› 27 Parent/Child v2 Doc values Faster joins Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Heap usage Better data structures
28. ‹#› 28 Geo-points v2 Doc values 50% index size Much faster queries Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Heap usage Better data structures
29. ‹#› 29 Security Image by User:Nino Barbieri - Own work, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php? curid=1766599
30. ‹#› Image by User:Nino Barbieri - Own work, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php? curid=1766599 JarHell check 30 Security
31. ‹#› Image by User:Nino Barbieri - Own work, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php? curid=1766599 JarHell check Java Security Manager 31 Security
32. ‹#› Image by User:Nino Barbieri - Own work, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php? curid=1766599 JarHell check Java Security Manager Minimal privileges 32 Security
33. ‹#› Image by User:Nino Barbieri - Own work, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php? curid=1766599 JarHell check Java Security Manager Minimal privileges Modularisation of core 33 Security
34. ‹#› Image by User:Nino Barbieri - Own work, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php? curid=1766599 JarHell check Java Security Manager Minimal privileges Modularisation of core No Java serialisation 34 Security
35. ‹#› Image by CSIRO, CC BY 3.0, https:// commons.wikimedia.org/w/index.php?curid=35458192 35 Tests Tests Tests
36. ‹#› Image by CSIRO, CC BY 3.0, https:// commons.wikimedia.org/w/index.php?curid=35458192 36 Tests Tests Tests We can't claim to support it… …unless we actually test it. Robert Muir
37. ‹#› Image by CSIRO, CC BY 3.0, https:// commons.wikimedia.org/w/index.php?curid=35458192 37 Tests Tests Tests Unit testable code
38. ‹#› Image by CSIRO, CC BY 3.0, https:// commons.wikimedia.org/w/index.php?curid=35458192 38 Tests Tests Tests Unit testable code Core plugins with tests
39. ‹#› Image by CSIRO, CC BY 3.0, https:// commons.wikimedia.org/w/index.php?curid=35458192 39 Tests Tests Tests Unit testable code Core plugins with tests Real integration tests
40. ‹#› Image by CSIRO, CC BY 3.0, https:// commons.wikimedia.org/w/index.php?curid=35458192 40 Tests Tests Tests Unit testable code Core plugins with tests Real integration tests Leniency
41. 41 STABILITY
42. STABILITY FEATURES 42
43. ‹#› 43 Pipeline Aggs Advanced analytics
44. ‹#› 44 Pipeline Aggs Advanced analytics Post-process the results of other aggs
45. ‹#› 45 Pipeline Aggs Distance
46. ‹#› 46 Pipeline Aggs Δ Distance == Velocity
47. ‹#› 47 Pipeline Aggs Anomaly detection
48. ‹#› 48 Geo-centroid Agg
49. ‹#› 49 Geo-centroid Agg
50. ‹#› 50 Query Profiler
51. ‹#› 51 Query Profiler
52. 52 Users want Developers want Release schedule
53. 53 • Quick bug fixes • Access to new features • Stability • Easy upgrades Users want Developers want Release schedule
54. Release schedule 54 • Quick bug fixes • Access to new features • Stability • Easy upgrades Users want • To see their code being used • Reduced complexity • To move forward fast Developers want
55. 55 Bugfix Releases Minor releases Major releases Release schedule
56. 56 • Current minor version • Last minor of previous major version • Bug fixes, small enhancements only Bugfix Releases Minor releases Major releases Release schedule
57. 57 • Current minor version • Last minor of previous major version • Bug fixes, small enhancements only Bugfix Releases • Frequent • Smaller • No breaking changes, unless required Minor releases Major releases Release schedule
58. 58 • Current minor version • Last minor of previous major version • Bug fixes, small enhancements only Bugfix Releases • Frequent • Smaller • No breaking changes, unless required Minor releases • More frequent • Smaller • Upgrade from any minor version of the previous major version Major releases Release schedule
59. 59 • Current minor version • Last minor of previous major version • Bug fixes, small enhancements only Bugfix Releases • Frequent • Smaller • No breaking changes, unless required Minor releases • More frequent • Smaller • Upgrade from any minor version of the previous major version Major releases Release schedule What about old data?
60. ‹#› 60 Reindex API POST _reindex { "src": { "index": "old_index" }, "dest": { "index": "new_index" } }
61. ‹#› 61 Reindex API POST _reindex { "src": { "index": "old_index", "query": { "match": { "user": "twitter" } } }, "dest": { "index": "new_index" } }
62. ‹#› 62 Update by Query POST users/_update_by_query { "query": { "match": { "user": "twitter" } }, "script": "...." } 62
63. ‹#› 63 Long running tasks Status Cancellation Throttling Task Manager
64. 64 Lucene 6 0.x 1.x 1.4 5.x 2.x
65. ‹#› 65 Ingest Node
66. ‹#› Power of Logstash filters… …available natively in Elasticsearch 66 Ingest Node
67. ‹#› Data transformation Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/ cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log) 67 Ingest Node
68. ‹#› Data transformation { "@timestamp": "2013-12-23T22:30:01.000Z", "syslog_timestamp": "Dec 23 14:30:01", "syslog_hostname": "louis", "syslog_program": "CRON", "syslog_pid": "619", "received_at": "2013-12-23 22:49:22 UTC", "received_from": "0:0:0:0:0:0:0:1:52617", "syslog_message": "(www-data) CMD (php / usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)", 68 Ingest Node
69. ‹#› Data enrichment { "ip": "45.24.12.8" } 69 Ingest Node
70. ‹#› Data enrichment { "ip": "45.24.12.8", "geoip": { "continent_name": "North America", "country_iso_code": "US", "region_name": "Michigan", "city_name": "Westland", "location": { "lon": -83.3686, "lat": 42.289 } 70 Ingest Node
71. ‹#› Wraps index/bulk API POST logs/apache?pipeline=apache_logs { … document body … } POST logs/apache/_bulk { "index": { "pipeline": "apache_logs" }} { … document body … } … … 71 Ingest Node
72. ‹#› 72 Scripting
73. ‹#› 73 Painless Scripting
74. ‹#› 74 Painless Scripting Fast Secure Infinite Loop Detection Dynamic/Static
75. ‹#› 75 Painless Scripting Dynamic/Static def first = input.doc.first_name.0; def last = input.doc.last_name.0; return first + " " + last;
76. ‹#› 76 Painless Scripting Dynamic/Static String first = (String)((List) ((Map)input.get("doc")) .get("first_name")).get(0); String last = (String)((List) ((Map)input.get("doc")) .get("last_name")).get(0); return first + " " + last;
77. ‹#› 77 Painless Scripting Dynamic/Static String first = (String)((List) ((Map)input.get("doc")) .get("first_name")).get(0); String last = (String)((List) ((Map)input.get("doc")) .get("last_name")).get(0); return first + " " + last; 10x faster!
78. ‹#› 78 Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
79. ‹#› 79 String Mappings { "type": "string", "index: "not_analyzed" } Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
80. ‹#› 80 String Mappings { "type": "text", "index: true } { "type": "keyword", "index: true } Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
81. ‹#› 81 String Mappings { "type": "text"} Full text field Full analysis chain Full text relevance Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
82. ‹#› 82 String Mappings { "type": "keyword"} Concrete string value Limited keyword analysis Doc values Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
83. ‹#› 83 Point field encoding Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
84. ‹#› 84 Point field encoding k-dimensional tree Replace encoding for Numeric/Date/IPv4 Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
85. NumericField vs PointField 85 0% 25% 50% 75% 100% Index Size Index Time Search Time Search Time Heap Usage 15% 76% 49% 49% 100% 100% 100% 100% NumericField PointField
86. ‹#› 86 Point field encoding k-dimensional tree Allows support for • BigInteger • BigDecimal • IPv6 Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
87. ‹#› 87 Point field encoding k-dimensional tree Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
88. ‹#› 88 Point field encoding k-dimensional tree where 1 =< k <= 8 Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
89. ‹#› 89 Point field encoding k-dimensional tree k=2: Lat/Lon k=3: 3D Geo Points k=?: …. Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
90. ‹#› 90 Geo Point Fields vs Geo Shape Fields Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
91. ‹#› 91 Geo Fields Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
92. ‹#› 92 Geo Fields Experimental! Share encoding Combined functionality Image by Laserlicht (Own work) [CC BY-SA 3.0 (http:// creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons Data structures
93. ‹#› 93 Search
94. ‹#› Completion suggester Document oriented Respects deletes Boost by: • prefix length • context • geolocation 94 Search
95. ‹#› Search After Deep paging Linear performance 95 Search
96. ‹#› Search After "size": 10, "sort": [ { "age": "asc" }, { "_uid": "desc" } ], "search_after": [ 18, "tweet#654323" ] 96 Search
97. ‹#› Search Refactoring Stricter parsing Improved caching Flexible search phases Extractable schema 97 Search
98. ‹#› 98 Java HTTP Client
99. ‹#› 99 Java HTTP Client • Decouple server/client • core becomes server • Single system boundary
100. ‹#› 100 Java HTTP Client • Aligns clients across languages • Eating our own dog food • Minimal dependencies
101. ‹#› 101 Resiliency
102. ‹#› 102 Resiliency Settings • Strict validation • Applied atomically • Reset to default • Return all defaults
103. ‹#› • Reliable cluster state publishing • Improved primary election/relocation • Improved index isolation 103 Resiliency
104. 104 elasticsearch-5.0.0-alpha1 Out Soon!
105. ‹#› Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 105