Elasticsearch: what's cooking

Presentation given at the Milan Elasticsearch meetup on February 5th 2016

Transcript

1. ‹#› Luca Cavanna Software Developer @javanna Elasticsearch: what’s cooking
2. Agenda 2 About Elastic Elasticsearch 2.0 Elasticsearch 2.1 Elasticsearch 2.2 Elasticsearch 3.0 1 2 3 4 5
3. About Elastic • Elastic the company gets founded • Elasticsearch 0.19 • Series A investment • Commercial offerings: support and trainings 3 2012
4. About Elastic • Series B investment • Kibana joins Elastic • Elasticsearch for Apache Hadoop gets released • Logstash joins Elastic • Language clients 4 2012 2013
5. About Elastic • Series C investment • Elasticsearch 1.0 released • Marvel gets released 5 2012 2013 2014
6. About Elastic • Shield gets released • Elastic{ON}15: first user conference • Found gets acquired • Packetbeat gets acquired • Watcher gets released • Elasticsearch 2.0 gets released 6 2012 2013 2014 2015
7. About Elastic • > 300 employees worldwide • We are hiring: https://www.elastic.co/about/careers • Elastic{ON}16: February 17-19, 2016 | San Francisco • Elasticsearch 3.0? 7 2012 2013 2014 2015 2016
8. About me • Joined in July 2013 • Working on Elasticsearch • Development, trainings, support 8 2012 2013 2014 2015 2016
9. ‹#› Elasticsearch 2.0 October 2015
10. Features • Pipeline aggregations • Better query execution • Structured, readable exceptions 10
11. Better query execution https://www.elastic.co/blog/better-query-execution-coming-elasticsearch-2-0 11 { “query”: { “filtered”: { “query”: { //query definition }, “filter”: { //filter definition } } } } { “query”: { “bool”: { “must”: { //query definition }, “filter”: { //filter definition } } } }
12. ‹#› Leniency is the root of all evil. Robert Muir (@Muir_Says)
13. The Great Mapping Refactoring 13 Fields Types • Names without dots • Full name references only • No type name prefix • Mappings at index level • Cannot be deleted • Names cannot start with a dot https://www.elastic.co/blog/great-mapping-refactoring
14. Resiliency • Cluster state diffs • Doc values by default for non analyzed fields • Durable writes by default • More checksums to detect corruption • Java classpath validation (aka JarHell) 14
15. Security • Java security manager enabled by default • Unicast discovery • Bind to local interfaces only by default 15
16. Cleanups 16 Removed Moved to plugins • Facets • Rivers • Shutdown api • MVEL scripting language • Delete by query • Multicast discovery • _size metadata field • murmur3 type
17. Migration plugin 17
18. Sense is back 18
19. Commercial plugins 19 Shield Watcher Marvel • Document level security • Field level security • Realms api • Hipchat/slack actions • Activate/deactivate api • Less but more useful metrics • Free in production too!
20. ‹#› Elasticsearch 2.1 November 2015
21. Main features • Additional pipeline aggregations • Geo-centroid aggregation • Faster recovery and shard allocation • As always, many bugs fixed! 21
22. ‹#› Elasticsearch 2.2 February 2016
23. Main improvements 23 Profile api Geo point rewrite Scripting security • Detailed query profiling • More to come • 50% faster indexing • 20-50% faster geo queries • Half disk and heap usage • Permissions lock-down • Whitelist of classes to load
24. ‹#› Elasticsearch 3.0 ? 2016
25. Highlights 25 Ingest node Scripting language Task management • Processing pipeline execution within a node • Simulate api • Dynamic • More than lucene expressions • Safe • Will unblock new features • Reindex api
26. More highlights 26 Settings Percolator • Strict settings validation • Ability to unset a setting • Search request parsing on the coordinating node • Unblocks request caching for time based indices • Query terms get indexed • Percolate api becomes a query Search internals
27. ‹#› Thank you! Q & A