Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's new in Elastic Stack 6.1?

Jun Ohtani
January 30, 2018

What's new in Elastic Stack 6.1?

LINE Dev Meetupで発表した資料になります。
https://line.connpass.com/event/76226/

Jun Ohtani

January 30, 2018
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. 1
    Jun Ohtani
    2018/01/30 at LINE Dev Meetup
    @johtani
    What's new in
    Elastic Stack 6.1?

    View full-size slide

  2. about
    • Me, Jun Ohtani / Developer Advocate
    ‒ lucene-gosenίϛολʔ
    ‒ σʔλ෼ੳج൫ߏஙೖ໳ͷஶऀͷ1ਓ
    ‒ http://blog.johtani.info

    • Elasticsearch, founded in 2012
    ‒ Products: Elasticsearch, Logstash, Kibana, Beats 

    X-Pack, Elastic Cloud

    Professional services: Support & development subscriptions
    ‒ Trainings, Consulting, SaaS
    2

    View full-size slide

  3. 3
    ར༻͍ͯ͠Δόʔδϣϯ͕1ܥͷํʁ
    ར༻͍ͯ͠Δόʔδϣϯ͕2ܥͷํʁ
    ར༻͍ͯ͠Δόʔδϣϯ͕5ܥͷํʁ
    ར༻͍ͯ͠Δόʔδϣϯ͕6ܥͷํʁ

    View full-size slide

  4. 4
    Elasticsearch

    View full-size slide

  5. 5
    Brand new upgrade experience
    Upgrades just got oh so simpler
    Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x

    View full-size slide

  6. 6
    Brand new upgrade experience
    • New Upgrade Assistant (UI & API)
    • Zero downtime upgrades
    ‒Rolling restarts from latest 5.x to 6.x
    ‒Cross-cluster search across major version
    Upgrades just got oh so simpler

    View full-size slide

  7. 7
    Space-saving columnar store
    • Better for storing
    sparse fields
    • Save on disk space &
    file system cache
    Tapping into Lucene 7 goodness (sparse doc value)
    user first middle last age phone
    johns Alex Smith
    jrice Jill Amy Rice 508.567.121
    1
    mt123 Jeff Twain 56
    sadams Sue Adams
    adoe Amy Doe 31
    lp12 Liz Potter

    View full-size slide

  8. 8
    Much speedier sorted queries
    Tapping into Lucene 7 goodness (index sorting)
    Player 1 Score: 600
    5.x
    Query for top 3 player scores
    Player 2 Score: 0
    Player 3 Score: 200
    Player 4 Score: 700
    Player 5 Score: 300
    Player 1907 Score: 800
    ...
    Query for top 3 player scores
    ...
    Player 1907 Score: 800
    Player 4 Score: 700
    Player 1 Score: 600
    Player 5 Score: 300
    Player 3 Score: 200
    Player 2 Score: 0
    6.x
    Sort at index time vs. query time
    Optimize on-disk format for some
    use cases
    Improve query performance at the
    cost of index performance

    View full-size slide

  9. 9
    Large Improvements to Replication
    • Limit syncs to only changed documents (instead of file-based recovery)
    • Fast replica recovery after temporary unavailability (network issues, etc.)
    • Re-sync on primary failure
    • Laying foundation for future big league features
    ‒Cross-datacenter replication
    ‒Changes API (tbd)
    New operation-based approach to recovery (sequence numbers)

    View full-size slide

  10. 10
    Breaking changes
    • Improved tools to handle breaking changes
    ‒Deprecation logging
    ‒Upgrade Assistant (UI & APIs)
    • Refer to Release Notes for complete list
    • Test, test, test
    Because major releases is time for major cleanup

    View full-size slide

  11. 11
    Simpler data models with type removal
    • Breaking change
    • Gradual migration path
    ‒ 6.0 indices can be created with only one type
    ‒ Existing 5.x indices using _type will continue to function
    • Introducing new APIs for type-less operations
    Say goodbye to _type confusion

    View full-size slide

  12. 12
    Some interesting changes
    • Rename template to index_patterns in _template
    • Content-Type detection disabled
    • Set explicit Content-Type in request header
    • Deprecation of _all
    • _all can no longer be configured for indices in 6.0
    • Use all_fields in query

    View full-size slide

  13. 13
    Some interesting changes
    • <= 2.x indices need to be reindexed
    • Re-index into 5.x or 6.0 cluster
    • Deprecate Groovy, Python, Javascript lang plugin
    • Rewrite scripts in plainless
    • Java High Level REST Client
    • Starting from version 5.6.0 a new Java client has been released.

    View full-size slide

  14. 14
    Distributed watch execution
    • Watches are no longer executed on only the
    master node
    • They are executed on nodes which hold
    shards of the .watches index
    • Configure all or specific nodes dedicated to
    watch execution
    X-Pack feature (Gold)

    View full-size slide

  15. 15
    Secure all the things
    Default security
    No default passwords
    Mandatory TLS
    between nodes
    changeme
    X-Pack feature (Gold)

    View full-size slide

  16. 16
    What's new in 6.1?
    • Index Splitting
    • Original primary shard is split into some primary shard in new index
    • Composite Aggregation
    • Designed to return ALL terms and sorted in ‘natural order’
    • Improve indexing throughput
    • Simple change of _fields metafield
    • Scripted Similarity
    • Custom similarity has become much easier

    View full-size slide

  17. 18
    Export saved searches to CSV with a single click
    Highly requested feature
    Trigger export via Watcher
    X-Pack feature (Basic, free)

    View full-size slide

  18. 19
    Lock down edits with Dashboard Only mode
    Share dashboards without worrying about accidental changes
    X-Pack feature (Gold)

    View full-size slide

  19. 20
    Maximize screen space with Full Screen mode
    Optimized viewing for your NOCs & SOCs

    View full-size slide

  20. 22
    6.0 starts Kibana on the accessibility path
    • High contrast color scheme
    • Keyboard accessibility
    • Screen reader support
    • More improvements on the way
    Accessibility improvements

    View full-size slide

  21. 23
    6.0 starts Kibana on the accessibility path
    Accessibility improvements

    View full-size slide

  22. 24
    Kibana now supports multiple query languages
    • Lucene Query Language (default)
    • Kuery (off by default, experimental in 6.0)
    • ... perhaps others in the future
    We want your feedback!
    • Enable Kuery from Advanced Settings
    More ways to query with Kuery
    Consistent syntax and simple to get started

    View full-size slide

  23. 25
    Get e-mail alerts on Cluster Alerts
    • Cluster Alerts are built-in
    Watches for cluster issues
    • Get e-mails when Cluster
    Alerts get triggered and
    resolved
    • Add admin e-mail in Kibana
    Advanced Settings

    X-Pack feature (Gold)

    View full-size slide

  24. 26
    Easily create simple threshold alerts
    New form based UI for threshold alerts
    X-Pack feature (Gold)

    View full-size slide

  25. 27
    Kibana Homepage - 6.1.0

    View full-size slide

  26. 28
    Lab Visualizations & Pie Chart Data Label - 6.1.0

    View full-size slide

  27. 29
    Preserve Dashboard Layout in Reporting - 6.1.0

    View full-size slide

  28. 31
    • Run multiple, distinct workloads on a
    single Logstash JVM
    • Simplify dataflow logic by managing
    per data source logic independently
    • Monitor each pipeline separately with
    the new Pipeline Viewer
    Multiple Pipelines, One Logstash
    Logstash
    JDBC Pipeline
    Netflow Pipeline
    Apache Pipeline

    View full-size slide

  29. 32
    • Visualize pipeline topologies as graphs

    • Reveal bottlenecks at the plugin level

    • Optimize dataflow with better metrics

    • Integrated with Monitoring UI
    Zoom in on your Pipelines
    Pipeline Viewer
    X-Pack feature (Basic, free)

    View full-size slide

  30. 33
    • Manage multiple pipelines from
    multiple nodes in a single UI

    • Logstash nodes can poll and
    dynamically reload pipelines on
    configuration change

    • Secure access to configuration
    management with X-Pack
    Centrally Manage Logstash Pipelines
    Configuration Management
    X-Pack feature (Gold)
    Elasticsearch
    Kibana
    Logstash
    Apache
    Logstash
    Logstash
    Config Mgmt UI
    DevOps / Admins
    Auto-Update Pipelines
    JDBC
    Netflow

    View full-size slide

  31. 34
    Centrally Manage Logstash Pipelines
    X-Pack feature (Gold)

    View full-size slide

  32. 35
    Convert ingest node to Logstash pipelines with a CLI tool
    Why Logstash?
    • More input sources
    • Multiple outputs
    • Richer transformations
    • Buffering, persistent queues
    Easily Migrate from Ingest Node Pipelines
    Ingest Node Converter
    $LS_HOME/bin/ingest-convert.sh
    --input file:///path/to/
    ingest_pipeline.json
    --output file:///path/to/
    logstash_pipeline.conf

    View full-size slide

  33. 36
    File Based Ruby Scripting Support - 6.1.0
    filter {
    ruby {
    # Cancel 90% of events
    path => "/etc/logstash/drop_percentage.rb"
    script_params => { "percentage" => 0.9 }
    }
    }
    def register(params)
    @should_reject = params["reject"]
    end
    def filter(event)
    return [] if event.get("message") == @should_reject
    event.set("field_test", rand(10))
    extra_processing(event)
    [event]
    end
    ...

    View full-size slide

  34. 38
    • New Kubernetes module in Metricbeat
    ‒ CPU, memory, bytes on network and more.
    • New processor to add_docker_metadata
    ‒ Container ID, name, image, labels
    • New processor to add_kubernetes_metadata
    ‒ Pod name, pod namespace, container name, pod labels
    Beats <3 containerization
    Monitor your Docker and Kubernetes deployments with ease

    View full-size slide

  35. 39
    More modules for more data sources
    New Filebeat modules
    New Metricbeat Modules

    View full-size slide

  36. 40
    • Improved dashboards for Metricbeat
    system module
    • Filebeat NGINX module ships with
    Machine Learning jobs
    ‒ We want your feedback
    Better Modules

    View full-size slide

  37. 41
    • Skip the hassle of parsing auditd logs
    ‒ Auditbeat subscribes to the kernel directly
    • Reuse auditd rule formats (no need to learn new rule formats)
    • Plus, file integrity checks on Linux, macOS, and Windows
    ‒ Watch files or directories (non-recursively) for changes
    ‒ Report file metadata and MD5, SHA1, SHA256 hashes on changes
    Auditbeat - a simpler way to track audit logs
    An alternative to auditd on Linux

    View full-size slide

  38. 42
    • Index pattern versions
    • Simpler configuration
    ‒ Module commands and configuration files
    ‒ module.d directory
    ‒ ./metricbeat module enable system
    ‒ Dashboards are easier to load and packaged with the Beat
    And moar awesome all around

    View full-size slide

  39. 43
    In 6.1?
    • Docker Autodiscovery
    • New Metricbeat and Filebeat modules
    • Metricbeat: Graphite, HTTP server metricset, Etcd, Logstash, System
    uptime, Windows service, OSD tree, RabbitMQ queue metricset
    • Filebeat: Logstash, Postgres, Kafka
    • TLS support in Packetbeat

    View full-size slide

  40. 45
    Elasticsearchษڧձ - meetup.com

    View full-size slide

  41. Thanks for listening!
    Q & A
    We’re hiring!
    https://www.elastic.co/about/careers/
    We’re helping!
    https://www.elastic.co/subscriptions
    http://training.elastic.co

    View full-size slide