$30 off During Our Annual Pro Sale. View Details »

Elastic 2.0

Elastic 2.0

Slide deck from a series of meetups in Denmark and Sweden in November 2015.

Elasticsearch Inc

November 26, 2015
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Elastic 2.0
    Everything you need to know
    Christoph Wurm – Solutions Architect @ Elastic
    Nov 26, 2015
    www.elastic.co
    Twitter: @elastic

    View Slide

  2. www.elastic.co
    1
    Agenda
    1. Elastic and Elastic Stack
    2. Elastic 2.0
    3. Use Cases from the field
    4. Stuff
    5. Q&A

    View Slide

  3. www.elastic.co
    2
    Community
    40,000
    Community
    members
    35,000
    Commits

    View Slide

  4. www.elastic.co
    3
    Downloads
    Mar’15
    Oct’12 Apr’13 Apr’14
    Oct’13
    20.
    Millions of Downloads
    10.
    40+ Million Downloads
    Cumulative across Elastic
    products to date
    Jun’15
    40.
    Sept’14

    View Slide

  5. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    4
    Elastic stack
    Logstash
    Collect, parse
    and enrich data
    Elasticsearch
    Store, search, analyze
    Kibana
    Visualize and explore data
    Hadoop Ecosystem
    Hadoop connector Beats
    Tap into your
    wire data
    Shield
    Security
    Watcher
    Scheduler
    Marvel - Monitoring
    Found
    Scale in the cloud

    View Slide

  6. elasticsearch 2.0

    View Slide

  7. www.elastic.co
    6
    elasticsearch 2.0
    • Networking
    • multicast removed (available as plugin)
    • will only bind to localhost by default, both ipv4 and ipv6
    • new unicast node discovery
    § will contact hosts listed in discovery.zen.ping.unicast.hosts (should contain all master
    nodes)
    § using first 5 ports in transport.tcp.port (default 9300-9400)

    View Slide

  8. www.elastic.co
    7
    elasticsearch 2.0
    • New feature: Pipeline aggregations
    • “Aggregations on the results of other aggregations”
    • Derivatives
    • Moving average
    • Holt Winters (prediction / anomaly detection)
    • Stats: Min/Max/avg
    • Custom

    View Slide

  9. www.elastic.co
    8
    elasticsearch 2.0
    • Moving average

    View Slide

  10. www.elastic.co
    9
    elasticsearch 2.0
    • Linear trends

    View Slide

  11. www.elastic.co
    10
    elasticsearch 2.0
    • Cyclic trends (Holt-Winters)

    View Slide

  12. www.elastic.co
    11
    elasticsearch 2.0
    • New Features
    • Pipeline Aggregations
    • Query DSL/Doc Improvement
    • Index Compression – 10-30%
    • Performance & resilience
    • Lucene 5.2
    • Update Cluster State with diffs
    • Doc_values by default
    • Sync-flush (1.6+)
    • Better handling for node-leave/rejoin (1.7+)
    • Durability-by-default
    • Async shard allocation (1.6+)
    • Breaking Backward Compatibility
    • Facets, Rivers – removed
    • Zen discovery improvements – FULL
    CLUSTER RESTART
    • Type mappings are now strict
    • Index segments created before ES .90.0
    must be upgraded
    • Migration Assistant
    • Units are required in settings

    View Slide

  13. kibana 4.2

    View Slide

  14. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    Dark theme
    Customizable maps
    Apps
    kibana 4.2: customizability

    View Slide

  15. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    Dark theme
    Offline/customizable maps
    Apps
    kibana 4.2: custom maps

    View Slide

  16. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    Dark theme
    Offline/customizable maps
    Apps
    kibana 4.2: apps

    View Slide

  17. www.elastic.co
    16
    kibana migration
    • Kibana 4 to Kibana 4.2
    • Support for Elasticsearch 2.x
    • Not backward-compatible with Elasticsearch 1.x
    • Dashboards are automatically migrated
    • Kibana 3 EOL is Nov 2015

    View Slide

  18. logstash 2.0

    View Slide

  19. www.elastic.co
    18
    logstash
    • logstash 2.0
    • compatible with Elasticsearch 2.0
    • HTTP as default transport protocol
    • Better shutdown process
    § all input plugins need to be adapted

    View Slide

  20. beats 1.0

    View Slide

  21. www.elastic.co
    20
    beats 1.0

    View Slide

  22. www.elastic.co
    21
    demo
    demo.elastic.co

    View Slide

  23. use cases

    View Slide

  24. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    23
    Elasticsearch @ Stagemonitor https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store

    View Slide

  25. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    24
    Mapping
    {
    "template": "stagemonitor-metrics-*”,
    "settings": {
    "index": {
    "refresh_interval": "5s”
    }
    },
    "mappings": {
    "_default_": {
    "dynamic_templates": [
    {
    "strings": {
    "match": "*”,
    "match_mapping_type": "string”,
    "mapping": { "type": "string", "doc_values": true,
    "index": "not_analyzed" }
    }
    }
    ],
    "_all": { "enabled": false },
    "_source": { "enabled": false },

    View Slide

  26. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    25
    Mapping (cont.)
    "properties": {
    "@timestamp": { "type": "date", "doc_values": true },
    "count": { "type": "integer", "doc_values": true, "index": "no" },
    "m1_rate": { "type": "float", "doc_values": true, "index": "no" },
    "m5_rate": { "type": "float", "doc_values": true, "index": "no" },
    "m15_rate": { "type": "float", "doc_values": true, "index": "no" },
    "max": { "type": "integer", "doc_values": true, "index": "no" },
    "mean": { "type": "integer", "doc_values": true, "index": "no" },
    "mean_rate": { "type": "float", "doc_values": true, "index": "no" },
    "median": { "type": "float", "doc_values": true, "index": "no" },
    "min": { "type": "float", "doc_values": true, "index": "no" },
    "p25": { "type": "float", "doc_values": true, "index": "no" },
    "p75": { "type": "float", "doc_values": true, "index": "no" },
    "p95": { "type": "float", "doc_values": true, "index": "no" },
    "p98": { "type": "float", "doc_values": true, "index": "no" },
    "p99": { "type": "float", "doc_values": true, "index": "no" },
    "p999": { "type": "float", "doc_values": true, "index": "no" },
    "std": { "type": "float", "doc_values": true, "index": "no" },
    "value": { "type": "float", "doc_values": true, "index": "no" },
    "value_boolean": { "type": "boolean", "doc_values": true, "index": "no" },
    "value_string": { "type": "string", "doc_values": true, "index": "no" }
    }

    View Slide

  27. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    26
    Force merge
    $ curator optimize --delay 2 --max_num_segments 1 indices --older-
    than 1 --time-unit days --timestring %Y.%m.%d --prefix stagemonitor-
    metrics-
    • Force merge
    – optimize for long-term storage by merging Lucene segments on disk
    – happens automatically on writes
    – run manually after 24-48 hours
    – CPU intensive operation, run during off-peak hours
    Post Optimize: 2.2 GB => ~510 MB

    View Slide

  28. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    27
    Elasticsearch @ MozDef SIEM https://www.elastic.co/elasticon/2015/sf/tackling-security-logs-
    with-the-elk-stack?q=mozilla
    Defender’s version of Metasploit and
    others
    Used in production at Mozilla, processes
    over 300M events per day
    Geolocate attackers
    Alerts
    Handle incidents in real time, collaborate
    with other administrators
    Integrates with other systems, e.g. to ban
    IP address ranges

    View Slide

  29. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    28
    Dashboard

    View Slide

  30. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    29
    Incident Handling

    View Slide

  31. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    30
    Winter is Coming!

    View Slide

  32. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    31
    MozDef Architecture

    View Slide

  33. stuff

    View Slide

  34. www.elastic.co
    33
    Training schedule
    Operations, Hands-On Workshop AMSTERDAM, Netherlands December 1-4, 2015
    Development, Operations MADRID, Spain January 19-21, 2016
    Development, Operations, Kibana BERLIN, Germany January 25-28, 2016
    Development, Operations, Hands-
    On Workshop
    COPENHAGEN, Denmark January 26-29, 2016
    PARIS, France February 2016
    LONDON, United Kingdom February 2016
    AMSTERDAM, Netherlands February 2016
    LONDON, United Kingdom February 2016
    training.elastic.co

    View Slide

  35. www.elastic.co
    34
    Elastic{ON} 2016
    $995 Early Bird Price until next Wednesday, Dec 2 (afterwards $1495)

    View Slide

  36. www.elastic.co
    35
    demo
    timelion

    View Slide

  37. q&a

    View Slide