Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's new in Elastic Stack 6.1?

Jun Ohtani
January 30, 2018

What's new in Elastic Stack 6.1?

LINE Dev Meetupで発表した資料になります。

Jun Ohtani

January 30, 2018

More Decks by Jun Ohtani

Other Decks in Technology


  1. about • Me, Jun Ohtani / Developer Advocate ‒ lucene-gosenίϛολʔ

    ‒ σʔλ෼ੳج൫ߏஙೖ໳ͷஶऀͷ1ਓ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS 2
  2. 5 Brand new upgrade experience Upgrades just got oh so

    simpler Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x
  3. 6 Brand new upgrade experience • New Upgrade Assistant (UI

    & API) • Zero downtime upgrades ‒Rolling restarts from latest 5.x to 6.x ‒Cross-cluster search across major version Upgrades just got oh so simpler
  4. 7 Space-saving columnar store • Better for storing sparse fields

    • Save on disk space & file system cache Tapping into Lucene 7 goodness (sparse doc value) user first middle last age phone johns Alex Smith jrice Jill Amy Rice 508.567.121 1 mt123 Jeff Twain 56 sadams Sue Adams adoe Amy Doe 31 lp12 Liz Potter
  5. 8 Much speedier sorted queries Tapping into Lucene 7 goodness

    (index sorting) Player 1 Score: 600 5.x Query for top 3 player scores Player 2 Score: 0 Player 3 Score: 200 Player 4 Score: 700 Player 5 Score: 300 Player 1907 Score: 800 ... Query for top 3 player scores ... Player 1907 Score: 800 Player 4 Score: 700 Player 1 Score: 600 Player 5 Score: 300 Player 3 Score: 200 Player 2 Score: 0 6.x Sort at index time vs. query time Optimize on-disk format for some use cases Improve query performance at the cost of index performance
  6. 9 Large Improvements to Replication • Limit syncs to only

    changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd) New operation-based approach to recovery (sequence numbers)
  7. 10 Breaking changes • Improved tools to handle breaking changes

    ‒Deprecation logging ‒Upgrade Assistant (UI & APIs) • Refer to Release Notes for complete list • Test, test, test Because major releases is time for major cleanup
  8. 11 Simpler data models with type removal • Breaking change

    • Gradual migration path ‒ 6.0 indices can be created with only one type ‒ Existing 5.x indices using _type will continue to function • Introducing new APIs for type-less operations Say goodbye to _type confusion
  9. 12 Some interesting changes • Rename template to index_patterns in

    _template • Content-Type detection disabled • Set explicit Content-Type in request header • Deprecation of _all • _all can no longer be configured for indices in 6.0 • Use all_fields in query
  10. 13 Some interesting changes • <= 2.x indices need to

    be reindexed • Re-index into 5.x or 6.0 cluster • Deprecate Groovy, Python, Javascript lang plugin • Rewrite scripts in plainless • Java High Level REST Client • Starting from version 5.6.0 a new Java client has been released.
  11. 14 Distributed watch execution • Watches are no longer executed

    on only the master node • They are executed on nodes which hold shards of the .watches index • Configure all or specific nodes dedicated to watch execution X-Pack feature (Gold)
  12. 15 Secure all the things Default security No default passwords

    Mandatory TLS between nodes changeme X-Pack feature (Gold)
  13. 16 What's new in 6.1? • Index Splitting • Original

    primary shard is split into some primary shard in new index • Composite Aggregation • Designed to return ALL terms and sorted in ‘natural order’ • Improve indexing throughput • Simple change of _fields metafield • Scripted Similarity • Custom similarity has become much easier
  14. 18 Export saved searches to CSV with a single click

    Highly requested feature Trigger export via Watcher X-Pack feature (Basic, free)
  15. 19 Lock down edits with Dashboard Only mode Share dashboards

    without worrying about accidental changes X-Pack feature (Gold)
  16. 21

  17. 22 6.0 starts Kibana on the accessibility path • High

    contrast color scheme • Keyboard accessibility • Screen reader support • More improvements on the way Accessibility improvements
  18. 24 Kibana now supports multiple query languages • Lucene Query

    Language (default) • Kuery (off by default, experimental in 6.0) • ... perhaps others in the future We want your feedback! • Enable Kuery from Advanced Settings More ways to query with Kuery Consistent syntax and simple to get started
  19. 25 Get e-mail alerts on Cluster Alerts • Cluster Alerts

    are built-in Watches for cluster issues • Get e-mails when Cluster Alerts get triggered and resolved • Add admin e-mail in Kibana Advanced Settings
 X-Pack feature (Gold)
  20. 26 Easily create simple threshold alerts New form based UI

    for threshold alerts X-Pack feature (Gold)
  21. 31 • Run multiple, distinct workloads on a single Logstash

    JVM • Simplify dataflow logic by managing per data source logic independently • Monitor each pipeline separately with the new Pipeline Viewer Multiple Pipelines, One Logstash Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline
  22. 32 • Visualize pipeline topologies as graphs
 • Reveal bottlenecks

    at the plugin level
 • Optimize dataflow with better metrics
 • Integrated with Monitoring UI Zoom in on your Pipelines Pipeline Viewer X-Pack feature (Basic, free)
  23. 33 • Manage multiple pipelines from multiple nodes in a

    single UI
 • Logstash nodes can poll and dynamically reload pipelines on configuration change
 • Secure access to configuration management with X-Pack Centrally Manage Logstash Pipelines Configuration Management X-Pack feature (Gold) Elasticsearch Kibana Logstash Apache Logstash Logstash Config Mgmt UI DevOps / Admins Auto-Update Pipelines JDBC Netflow
  24. 35 Convert ingest node to Logstash pipelines with a CLI

    tool Why Logstash? • More input sources • Multiple outputs • Richer transformations • Buffering, persistent queues Easily Migrate from Ingest Node Pipelines Ingest Node Converter $LS_HOME/bin/ingest-convert.sh --input file:///path/to/ ingest_pipeline.json --output file:///path/to/ logstash_pipeline.conf
  25. 36 File Based Ruby Scripting Support - 6.1.0 filter {

    ruby { # Cancel 90% of events path => "/etc/logstash/drop_percentage.rb" script_params => { "percentage" => 0.9 } } } def register(params) @should_reject = params["reject"] end def filter(event) return [] if event.get("message") == @should_reject event.set("field_test", rand(10)) extra_processing(event) [event] end ...
  26. 38 • New Kubernetes module in Metricbeat ‒ CPU, memory,

    bytes on network and more. • New processor to add_docker_metadata ‒ Container ID, name, image, labels • New processor to add_kubernetes_metadata ‒ Pod name, pod namespace, container name, pod labels Beats <3 containerization Monitor your Docker and Kubernetes deployments with ease
  27. 40 • Improved dashboards for Metricbeat system module • Filebeat

    NGINX module ships with Machine Learning jobs ‒ We want your feedback Better Modules
  28. 41 • Skip the hassle of parsing auditd logs ‒

    Auditbeat subscribes to the kernel directly • Reuse auditd rule formats (no need to learn new rule formats) • Plus, file integrity checks on Linux, macOS, and Windows ‒ Watch files or directories (non-recursively) for changes ‒ Report file metadata and MD5, SHA1, SHA256 hashes on changes Auditbeat - a simpler way to track audit logs An alternative to auditd on Linux
  29. 42 • Index pattern versions • Simpler configuration ‒ Module

    commands and configuration files ‒ module.d directory ‒ ./metricbeat module enable system ‒ Dashboards are easier to load and packaged with the Beat And moar awesome all around
  30. 43 In 6.1? • Docker Autodiscovery • New Metricbeat and

    Filebeat modules • Metricbeat: Graphite, HTTP server metricset, Etcd, Logstash, System uptime, Windows service, OSD tree, RabbitMQ queue metricset • Filebeat: Logstash, Postgres, Kafka • TLS support in Packetbeat
  31. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co