Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic v6.0 - complete presentation about this new release

Elastic v6.0 - complete presentation about this new release

complete presentation about this new release v6.0. Elastic Stack (Elasticsearch, Kibana & logshtash, Watcher, Beats and more)

Camilo Sierra

October 30, 2017
Tweet

More Decks by Camilo Sierra

Other Decks in Programming

Transcript

  1. 2 ALONE WE CAN DO SO LITTLE; TOGETHER WE CAN

    DO SO MUCH. Helen Keller ‘‘
  2. 5 Nouvelle expérience de mise à jour Mise à niveau

    progressive et plus simple Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x
  3. 6 Brand new upgrade experience • New Upgrade Assistant (UI

    & API) • Zero downtime upgrades ‒Rolling restarts from latest 5.x to 6.x ‒Cross-cluster search across major version Upgrades just got oh so simpler X-Pack feature (Basic, free)
  4. 7 Space-saving columnar store • Better for storing sparse fields

    • Save on disk space & file system cache Tapping into Lucene 7 goodness (sparse doc value) user first middle last age phone johns Alex Smith jrice Jill Amy Rice 508.567.121 1 mt123 Jeff Twain 56 sadams Sue Adams adoe Amy Doe 31 lp12 Liz Potter
  5. 8 age 56 31 5.X 32 bit 32 bit 32

    bit 32 bit 32 bit 32 bit age 56 31 6.0 32 bit 32 bit Space-saving columnar store Large improvements to how sparse data is stored • More efficient data structure in Elasticsearch 6.0 for storing sparse fields • In real life, calculate space on disk saved may not be that simple in most cases.
  6. 9 Much speedier sorted queries Tapping into Lucene 7 goodness

    (index sorting) Player 1 Score: 600 5.x Query for top 3 player scores Player 2 Score: 0 Player 3 Score: 200 Player 4 Score: 700 Player 5 Score: 300 Player 1907 Score: 800 ... Query for top 3 player scores ... Player 1907 Score: 800 Player 4 Score: 700 Player 1 Score: 600 Player 5 Score: 300 Player 3 Score: 200 Player 2 Score: 0 6.x Sort at index time vs. query time Optimize on-disk format for some use cases Improve query performance at the cost of index performance
  7. 10 Game Points Joust 1000 Joust 990 Joust 980 Dragon’s

    Lair 1000 Dragon’s Lair 950 Dragon’s Lair 940 Index Sorting: Define a Sorted Index
  8. 12 Large Improvements to Replication • Limit syncs to only

    changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd) New operation-based approach to recovery (sequence numbers)
  9. 15 Simpler data models with type removal • Breaking change

    • Gradual migration path ‒ 6.0 indices can be created with only one type ‒ Existing 5.x indices using _type will continue to function • Introducing new APIs for type-less operations Say goodbye to _type confusion
  10. 17 Join datatype !!! Only one join field mapping is

    allowed per index. PUT my_index { "mappings": { "doc": { "properties": { "my_join_field": { "type": "join", "relations": { "question": "answer" } } } } } } PUT my_index/doc/1 { "text": "a question", "my_join_field": { "name": "question" } } PUT my_index/doc/3?routing=1 { "text": "an answer", "my_join_field": { "name": "answer", "parent": "1" } }
  11. 18 Distributed watch execution • Watches are no longer executed

    on only the master node • They are executed on nodes which hold shards of the .watches index • Configure all or specific nodes dedicated to watch execution • To improve scalability, you need to add more replicas to .watches index X-Pack feature (Gold)
  12. 20 • Run multiple, distinct workloads on a single Logstash

    JVM • Simplify dataflow logic by managing per data source logic independently • Monitor each pipeline separately with the new Pipeline Viewer Multiple Pipelines, One Logstash Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline
  13. 21 • Visualize pipeline topologies as graphs
 • Reveal bottlenecks

    at the plugin level
 • Optimize dataflow with better metrics
 • Integrated with Monitoring UI Zoom in on your Pipelines Pipeline Viewer X-Pack feature (Basic, free)
  14. 22 Runtime Stats for Your Plugins • Filters and Outputs

    display : • CPU usage • Performance • Throughput
 • Highlights areas of high CPU usage and execution time • Provide an easy way to diagnose bottlenecks Pipeline Viewer X-Pack feature (Basic, free)
  15. 23 • Manage multiple pipelines from multiple nodes in a

    single UI
 • Logstash nodes can poll and dynamically reload pipelines on configuration change
 • Secure access to configuration management with X-Pack Centrally Manage Logstash Pipelines Configuration Management X-Pack feature (Gold) Elasticsearch Kibana Logstash Apache Logstash Logstash Config Mgmt UI DevOps / Admins Auto-Update Pipelines JDBC Netflow
  16. 26 Export saved searches to CSV with a single click

    Highly requested feature Trigger export via Watcher X-Pack feature (Basic, free)
  17. 28 Lock down edits with Dashboard Only mode Share dashboards

    without worrying about accidental changes X-Pack feature (Gold)
  18. 30

  19. 31 6.0 starts Kibana on the accessibility path • High

    contrast color scheme • Keyboard accessibility • Screen reader support • More improvements on the way Accessibility improvements
  20. 33 Kibana now supports multiple query languages • Lucene Query

    Language (default) • Kuery (off by default, experimental in 6.0) • ... perhaps others in the future We want your feedback! • Enable Kuery from Advanced Settings More ways to query with Kuery Consistent syntax and simple to get started
  21. 34 Kuery syntax • Function("field", value) We want your feedback!

    • Kuery: is("response", 200) • Lucene: response:200 • Kuery: not(is("response", 404)) • Lucene: !response:404 • Kuery: range("bytes", gt=1000, lt=8000) • Lucene: bytes:[1000 to 8000] • Kuery: geoPolygon("geo.coordinates", "40.97, -127.26", "24.20, -84.375", "40.44, -66.09") • Lucene: not supported More ways to query with Kuery Consistent syntax and simple to get started
  22. 35 Get e-mail alerts on Cluster Alerts • Cluster Alerts

    are built-in Watches for cluster issues • Get e-mails when Cluster Alerts get triggered and resolved • Add admin e-mail in Kibana Advanced Settings
 X-Pack feature (Gold)
  23. 36 Easily create simple threshold alerts New form based UI

    for threshold alerts X-Pack feature (Gold)
  24. 37 Actions in threshold alerts UI / API Actions in

    UI for threshold alerts / Actions supported by API watcher Watcher API supports the following types of actions: • email, • webhook, • index, • logging, • hipchat, • Slack, • and pagerduty.
  25. 39 • New Kubernetes module in Metricbeat ‒ CPU, memory,

    bytes on network and more. • New processor to add_docker_metadata ‒ Container ID, name, image, labels • New processor to add_kubernetes_metadata ‒ Pod name, pod namespace, container name, pod labels Beats <3 containerization Monitor your Docker and Kubernetes deployments with ease
  26. 41 • Improved dashboards for Metricbeat system module • Filebeat

    NGINX module ships with Machine Learning jobs ‒ We want your feedback Better Modules
  27. 42 • Skip the hassle of parsing auditd logs ‒

    Auditbeat subscribes to the kernel directly • Reuse auditd rule formats (no need to learn new rule formats) • Plus, file integrity checks on Linux, macOS, and Windows ‒ Watch files or directories (non-recursively) for changes ‒ Report file metadata and MD5, SHA1, SHA256 hashes on changes Auditbeat - a simpler way to track audit logs An alternative to auditd on Linux
  28. 43 Elasticsearch • Rolling upgrades (upgrade with no downtime), •

    Top queries are super-fast (sort at index time), • Save disk space (save a lot of both disk space and file-system cache), • Say goodbye to multiple _types within index (parent/child become Join datatype) • Fast replica recovery (Sequence numbers improve replicas synchronization, recovery if primary shard fails... much more to come)
  29. 44 Logstash • Run multiple, distinct workloads on a single

    Logstash (Simplify dataflow logic by managing per data source logic independently) • Runtime Stats for Your Plugins (indicators on the components spend extra time, easy way to diagnose bottlenecks) • Manage multiple pipelines from multiple nodes in a single UI (change pipelines configuration in running services)
  30. 45 Kibana • CSV Export (Highly requested. Most commented issue

    in the Kibana repo), • Dashboard only mode (Hides all other applications, Dashboard's presented are in read only mode), • Full screen mode (Perfect for operations use case and "command centers"), • Compliance & Accessibility (High contrast colors for the color blind, Keyboard accessible, Improved support for screen readers), • Watcher UI (UI for creating and editing alerts based on thresholds)
  31. 46 Become an Elastic Pioneer Download 6.0 preview release Provide

    feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3
  32. 47 Elastic Pioneer Program Download 6.0 preview release (Beta, RC,

    etc) Provide feedback via GitHub or Discuss forum Get limited edition Pioneer swag 1 2 3 We want your feedback!