Elasticsearch: what's cooking

This talk was given at the Elastic meetup in Milan on February 5th 2016. It is about the improvements released with Elasticsearch 2.0, 2.1 and 2.2, as well as the improvements coming with the next major version.

1. ‹#› Luca Cavanna Software Developer @javanna Elasticsearch: what’s cooking
2. Agenda 2 About Elastic Elasticsearch 2.0 Elasticsearch 2.1 Elasticsearch 2.2 Elasticsearch 3.0 1 2 3 4 5
3. About Elastic • Elastic the company gets founded • Elasticsearch 0.19 • Series A investment • Commercial offerings: support and trainings 3 2012
4. About Elastic • Series B investment • Kibana joins Elastic • Elasticsearch for Apache Hadoop gets released • Logstash joins Elastic • Language clients 4 2012 2013
5. About Elastic • Series C investment • Elasticsearch 1.0 released • Marvel gets released 5 2012 2013 2014
6. About Elastic • Shield gets released • Elastic{ON}15: first user conference • Found gets acquired • Packetbeat gets acquired • Watcher gets released • Elasticsearch 2.0 gets released 6 2012 2013 2014 2015
7. About Elastic • > 300 employees worldwide • We are hiring: https://www.elastic.co/about/careers • Elastic{ON}16: February 17-19, 2016 | San Francisco • Elasticsearch 3.0? 7 2012 2013 2014 2015 2016
8. About me • Joined in July 2013 • Working on Elasticsearch • Development, trainings, support 8 2012 2013 2014 2015 2016
9. ‹#› Elasticsearch 2.0 October 2015
10. Features • Pipeline aggregations • Better query execution • Structured, readable exceptions 10
11. Better query execution https://www.elastic.co/blog/better-query-execution-coming-elasticsearch-2-0 11 { “query”: { “filtered”: { “query”: { //query definition }, “filter”: { //filter definition } } } } { “query”: { “bool”: { “must”: { //query definition }, “filter”: { //filter definition } } } }
12. ‹#› Leniency is the root of all evil. Robert Muir (@Muir_Says)
13. The Great Mapping Refactoring 13 Fields Types • Names without dots • Full name references only • No type name prefix • Mappings at index level • Cannot be deleted • Names cannot start with a dot https://www.elastic.co/blog/great-mapping-refactoring
14. Resiliency • Cluster state diffs • Doc values by default for non analyzed fields • Durable writes by default • More checksums to detect corruption • Java classpath validation (aka JarHell) 14
15. Security • Java security manager enabled by default • Unicast discovery • Bind to local interfaces only by default 15
16. Cleanups 16 Removed Moved to plugins • Facets • Rivers • Shutdown api • MVEL scripting language • Delete by query • Multicast discovery • _size metadata field • murmur3 type
17. Migration plugin 17
18. Sense is back 18
19. Commercial plugins 19 Shield Watcher Marvel • Document level security • Field level security • Realms api • Hipchat/slack actions • Activate/deactivate api • Less but more useful metrics • Free in production too!
20. ‹#› Elasticsearch 2.1 November 2015
21. Main features • Additional pipeline aggregations • Geo-centroid aggregation • Faster recovery and shard allocation • As always, many bugs fixed! 21
22. ‹#› Elasticsearch 2.2 February 2016
23. Main improvements 23 Profile api Geo point rewrite Scripting security • Detailed query profiling • More to come • 50% faster indexing • 20-50% faster geo queries • Half disk and heap usage • Permissions lock-down • Whitelist of classes to load
24. ‹#› Elasticsearch 3.0 ? 2016
25. Highlights 25 Ingest node Scripting language Task management • Processing pipeline execution within a node • Simulate api • Dynamic • More than lucene expressions • Safe • Will unblock new features • Reindex api
26. More highlights 26 Settings Percolator • Strict settings validation • Ability to unset a setting • Search request parsing on the coordinating node • Unblocks request caching for time based indices • Query terms get indexed • Percolate api becomes a query Search internals
27. ‹#› Thank you! Q & A