What's New in Elastic 2?

A presentation given at multiple meetups and training sessions highlighting some of the new features of Elasticsearch 2.0, Logstash 2.0, Kibana 4.x, and Beats 1.0.

Transcript

1. What's New for You in Elastic 2? Kurt Hurtado Elastic Engineering (w/ help from Tanya Bragin & Igor Motov!)
2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 2 About Elastic • Founded: July 2012 • Renamed Elasticsearch → Elastic: March 2015 • Headquarters: Amsterdam and Mountain View, CA • Develops Elasticsearch, Logstash, Kibana, Beats • Provides: • Training (public and onsite) • Development and production support • Hosted Elasticsearch (Found) • Commercial plugins: Marvel, Shield, Watcher
3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 Elastic Product Family Kibana Visualize and explore data Elasticsearch Store, search, analyze Logstash Beats ES-Hadoop Collect, parse and enrich data Marvel Monitor and manage Shield Secure and protect Found Elasticsearch as a Service Open Source Products Commercial Products Training Professional Services Support Subscriptions BUILT FOR TODAY’S SCALABLE, DISTRIBUTED SYSTEMS Watcher Monitor and Notify
4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 Agenda • Elasticsearch 2.0 • Four main themes • Logstash 2.0 • Released in conjunction with ES2.0 • Kibana 4.0 - 4.2 • Beats • Filebeat • Topbeat • Packetbeat
5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 Four Main Themes in 2.0 • Simplification • Removing, deprecating features • Query DSL / Doc improvements • Security • Always high on customer wish lists • Resiliency • Started in 1.x, but ongoing • Features • pipeline aggs • Compression
6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Elasticsearch 2.0! • Very large release • >2,500 Pull Requests • 469 committers • Four themes
7. Theme 1: Simplification
8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 Removed Entirely • Rivers - use logstash or create your own ingestion layer • Facets - replaced by aggregations • _shutdown API - use platform specific services • Support for Thrift and Memcached protocols • Bulk UDP - use the standard bulk API, or use UDP to send documents to Logstash first.
9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 Moved to Plugins • Delete by query • Problematic, not a "core" feature • Types: • murmur3 • _size • Multicast discovery • Unicast was always recommended in production
10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 Mappings • Conflicting field mappings • Fields cannot be referenced by short name • Type name prefix removed • Field names cannot contain dots • Type names cannot start with a dot • Type may no longer be deleted • index_analyzer is removed • _analyzer field is removed • date format changes • ... and more ...
11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 Conflicting Mappings PUT my_index { "mappings": { "type_one": { "properties": { "name": { "type": "string" } } }, "type_two": { "properties": { "name": { "type": "string", "analyzer": "english" } } } } } What is the mapping for name? Unexpected results. This is not allowed in Elasticsearch 2.0.
12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 Ambiguous Mappings in < 2.0 PUT my_index { "mappings": { "name": { "properties": { "title": { "type": "string" }, "name": { "properties": { "title": { "type": "string" } } } } } } } What does name refer to? name.title? name.name.title?
13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 13 Refactored Mappings in 2.0 PUT my_index { "mappings": { "name": { "properties": { "title": { "type": "string" }, "name": { "properties": { "title": { "type": "string" } } } } } } } name.name.title is not a thing. title name.title
14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 Analyzer Mappings PUT my_index { "mappings": { "my_type": { "properties": { "title": { "type": "string", "analyzer": "my_analyzer } } } } } There are some changes in how field-specific analyzers are now set. This format, which sets both search and index analyzers, is still acceptable in 2.0.
15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 Analyzer Mappings • Before 2.0: • analyzer - sets index and search analyzer • search_analyzer - sets search analyzer • index_analyzer - sets index analyzer • Starting with 2.0: • analyzer - sets index and search analyzers • search_analyzer - overrides search analyzer
16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 Query and Filter Execution Changes • Before 2.0 • Queries: • Typically contribute to scoring • No caching • Filters: • Don't contribute to scoring • Can be cached
17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 Query and Filter Execution Changes { "filtered" : { "query": { query definition }, "filter": { filter definition } } } Before 2.0: { "bool" : { "must": { query definition }, "must_not": { query definition }, "should": { query definition }, "filter": { filter definition } } } After 2.0:
18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 Query and Filter Execution Changes • Approximation phase • quickly iterates over a superset of the matching documents • Verification phase • check if a document in this superset actually matches the query Two-Phase Query Execution
19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 Analyzer Mappings { "bool" : { "must": [{ "match_phrase": { "body": "quick fox" }, { "match_phrase": { "body": "brown dog" } }] } } Two-Phase Query Execution Example • Approximation phase • all docs with "quick", "fox", "brown", and "dog" • Verification phase • actual phrase matching
20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 Query and Filter Execution Changes • Fully automatic • Keeps track of 256 most recently used queries • Only caches those that appear 5 times or more • Does not cache segments which have less than 10000 documents or 3% of the documents of the index • More efficient query cache (roaring bitmaps) • Non-scoring components are cache-able Query Caching
21. Theme 2: Security
22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 Security Enhancements • Elasticsearch now binds to local interfaces ONLY • Unicast discovery is now the default • Makes Elasticsearch more secure by default • Protects Elasticsearch in the wild (don't do that!) • Security Manager • Prevents outside access outside of Elasticsearch even if Elasticsearch process is compromised • All resources that Elasticsearch can access are defined on node startup
23. Theme 3: Resiliency
24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 Durability of Transaction Log • Before 2.0 transaction log was fsynced every 5 sec • Transaction log is now fsynced after each operation • Configurable • On SSDs indexing is about 7% - 10% slower with bulk indexing compared to async translog flushes Index operations are now durable by default!
25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 25 Multiple data path striping Take advantage of striping in path.data configuration:
26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 26 Multiple data path striping Before Elasticsearch 2.0:
27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 27 Multiple data path striping PIC Now safer in Elasticsearch 2.0!
28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 28 Cluster State Diffs • Before 2.0, the entire cluster state was shipped on every change to every node • Starting with 2.0 only changes are sent • This can be a massive improvement on clusters with large cluster states! • Thanks, Igor!
29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 29 Non-Ambiguous Setting Units curl -XPUT "localhost:9200/test/_settings" -d '{ "index" : { "refresh_interval" : "5" } }' Settings now require units (when appropriate) 5 what??
30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 30 Doc Values by Default • Fielddata was a common culprit in OOMs • Doc Values: Lucene data structure (disk-based) • Dramatic heap memory reduction by default • Values for sorting, aggs, etc are moved onto disk • Let the OS deal with it! • Indexed, not_analyzed fields now use doc values • Only for indices created with 2.0 • Reindex required for older data
31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 31 Previous Resiliency Improvements • Sync-flush (1.6) • Async shard allocation (1.6) • Delayed Allocation (1.7) • Better handling of nodes leaving/rejoining • Resiliency page contains latest information: • https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html
32. Theme 4: Features
33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 33 Pipeline Aggregations • Derivatives • Moving average • Holt Winters (prediction / anomaly detection) • Stats: Min/Max/avg • Time-series math
34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 34 Index Compression • 10-30% reduction in index size • Some indexing/merging impact • Dynamic setting - could be set before optimization for time-based indices
35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 35 Upgrading to Elasticsearch 2.0 • Major Version Upgrade!!! • No rolling upgrades • One way - no way to downgrade back to 1.x • Take Snapshot (and test restore) before proceeding • Test! Test! Test! • Use the Migration plugin • Site plugin for 1.x that checks for potential issues • https://github.com/elastic/elasticsearch-migration
36. Logstash
37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 37 Logstash Recent Features • Logstash 1.5 • Plugin management improvements • Grok performance improvements (2-3x) • Heartbeat plugin for monitoring of LS health • Logstash 2.0 • Elasticsearch 2.0 compatibility • HTTP as default transport protocol • Better shutdown semantics • New Logstash plugins • Kafka input/output • JDBC input • HTTP input • WebHDFS output • Salesforce input
38. Kibana
39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 39 Kibana 4.0 • Complete re-write of Kibana • Separate Tasks • Drag and Drop • Re-usable components • Using D3 Javascript visualization library • NodeJS backend server (rather than client-side) • Leverages Elasticsearch aggregations • Multi-dimensional visualizations • Plug-in architecture for "Kibana apps" • Export to CSV • Many more!
40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 40 Kibana 4.0 / 4.1: Customizability Dark theme Field formatters Customizable maps
41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 41 Kibana 4.0 / 4.1: Customizability Dark theme Field formatters Customizable maps
42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 42 Kibana 4.0 / 4.1: Customizability Dark theme Field formatters Offline/customizable maps
43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 43 Kibana 4: Management Functionality Kibana server status page Configurable log levels Saved object export
44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 44 Kibana 4: Enhanced Analytics Bubble charts Geo heatmap New aggregations (e.g. IP range)
45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 45 Kibana 4: Usability Map filters Pinned filters Clickable legends
46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 46 Kibana 3 -> Kibana 4 Migration Tips • Kibana 3 to Kibana 4 (Kibana 3 EOL is Nov 2015) • Embedded web server • Platform-specific installation packages • Performance improvements • Dashboards will not be migrated • Some panel types not available, yet • Kibana 4 to Kibana 4.2 • Support for Elasticsearch 2.x • Not backward-compatible with Elasticsearch 1.x • Dashboards are automatically migrated
47. Beats Family
48. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 48 Beats Update • More Packetbeat protocols • MongoDB • DNS • Memcache • More Beats • Topbeat: Shipper for CPU, memory, process resource metrics • Improved platform support • Windows support (e.g. self-contained installer) • Developer guides • Building Beats • Building Packetbeat protocol modules
49. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 49 Documentation and Help • Discussion Forums - https://discuss.elastic.co • Meetups - https://elasticsearch.meetup.com • Docs - https://elastic.co/docs • Community - https://elastic.co/community • More Resources - https://www.elastic.co/learn