Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch: what's cooking

Elasticsearch: what's cooking

Presentation given at the Milan Elasticsearch meetup on February 5th 2016

Luca Cavanna

February 05, 2016
Tweet

More Decks by Luca Cavanna

Other Decks in Programming

Transcript

  1. About Elastic • Elastic the company gets founded • Elasticsearch

    0.19 • Series A investment • Commercial offerings: support and trainings 3 2012
  2. About Elastic • Series B investment • Kibana joins Elastic

    • Elasticsearch for Apache Hadoop gets released • Logstash joins Elastic • Language clients 4 2012 2013
  3. About Elastic • Shield gets released • Elastic{ON}15: first user

    conference • Found gets acquired • Packetbeat gets acquired • Watcher gets released • Elasticsearch 2.0 gets released 6 2012 2013 2014 2015
  4. About Elastic • > 300 employees worldwide • We are

    hiring: https://www.elastic.co/about/careers • Elastic{ON}16: February 17-19, 2016 | San Francisco • Elasticsearch 3.0? 7 2012 2013 2014 2015 2016
  5. About me • Joined in July 2013 • Working on

    Elasticsearch • Development, trainings, support 8 2012 2013 2014 2015 2016
  6. Better query execution https://www.elastic.co/blog/better-query-execution-coming-elasticsearch-2-0 11 { “query”: { “filtered”: {

    “query”: { //query definition }, “filter”: { //filter definition } } } } { “query”: { “bool”: { “must”: { //query definition }, “filter”: { //filter definition } } } }
  7. The Great Mapping Refactoring 13 Fields Types • Names without

    dots • Full name references only • No type name prefix • Mappings at index level • Cannot be deleted • Names cannot start with a dot https://www.elastic.co/blog/great-mapping-refactoring
  8. Resiliency • Cluster state diffs • Doc values by default

    for non analyzed fields • Durable writes by default • More checksums to detect corruption • Java classpath validation (aka JarHell) 14
  9. Security • Java security manager enabled by default • Unicast

    discovery • Bind to local interfaces only by default 15
  10. Cleanups 16 Removed Moved to plugins • Facets • Rivers

    • Shutdown api • MVEL scripting language • Delete by query • Multicast discovery • _size metadata field • murmur3 type
  11. Commercial plugins 19 Shield Watcher Marvel • Document level security

    • Field level security • Realms api • Hipchat/slack actions • Activate/deactivate api • Less but more useful metrics • Free in production too!
  12. Main features • Additional pipeline aggregations • Geo-centroid aggregation •

    Faster recovery and shard allocation • As always, many bugs fixed! 21
  13. Main improvements 23 Profile api Geo point rewrite Scripting security

    • Detailed query profiling • More to come • 50% faster indexing • 20-50% faster geo queries • Half disk and heap usage • Permissions lock-down • Whitelist of classes to load
  14. Highlights 25 Ingest node Scripting language Task management • Processing

    pipeline execution within a node • Simulate api • Dynamic • More than lucene expressions • Safe • Will unblock new features • Reindex api
  15. More highlights 26 Settings Percolator • Strict settings validation •

    Ability to unset a setting • Search request parsing on the coordinating node • Unblocks request caching for time based indices • Query terms get indexed • Percolate api becomes a query Search internals