Elastic 2.0

Elastic 2.0

Slide deck from a series of meetups in Denmark and Sweden in November 2015.

098332e9d988080a9057816f84d668f7?s=128

Elasticsearch Inc

November 26, 2015
Tweet

Transcript

  1. Elastic 2.0 Everything you need to know Christoph Wurm –

    Solutions Architect @ Elastic Nov 26, 2015 www.elastic.co Twitter: @elastic
  2. www.elastic.co 1 Agenda 1. Elastic and Elastic Stack 2. Elastic

    2.0 3. Use Cases from the field 4. Stuff 5. Q&A
  3. www.elastic.co 2 Community 40,000 Community members 35,000 Commits

  4. www.elastic.co 3 Downloads Mar’15 Oct’12 Apr’13 Apr’14 Oct’13 20. Millions

    of Downloads 10. 40+ Million Downloads Cumulative across Elastic products to date Jun’15 40. Sept’14
  5. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 4 Elastic stack Logstash Collect, parse and enrich data Elasticsearch Store, search, analyze Kibana Visualize and explore data Hadoop Ecosystem Hadoop connector Beats Tap into your wire data Shield Security Watcher Scheduler Marvel - Monitoring Found Scale in the cloud
  6. elasticsearch 2.0

  7. www.elastic.co 6 elasticsearch 2.0 • Networking • multicast removed (available

    as plugin) • will only bind to localhost by default, both ipv4 and ipv6 • new unicast node discovery § will contact hosts listed in discovery.zen.ping.unicast.hosts (should contain all master nodes) § using first 5 ports in transport.tcp.port (default 9300-9400)
  8. www.elastic.co 7 elasticsearch 2.0 • New feature: Pipeline aggregations •

    “Aggregations on the results of other aggregations” • Derivatives • Moving average • Holt Winters (prediction / anomaly detection) • Stats: Min/Max/avg • Custom
  9. www.elastic.co 8 elasticsearch 2.0 • Moving average

  10. www.elastic.co 9 elasticsearch 2.0 • Linear trends

  11. www.elastic.co 10 elasticsearch 2.0 • Cyclic trends (Holt-Winters)

  12. www.elastic.co 11 elasticsearch 2.0 • New Features • Pipeline Aggregations

    • Query DSL/Doc Improvement • Index Compression – 10-30% • Performance & resilience • Lucene 5.2 • Update Cluster State with diffs • Doc_values by default • Sync-flush (1.6+) • Better handling for node-leave/rejoin (1.7+) • Durability-by-default • Async shard allocation (1.6+) • Breaking Backward Compatibility • Facets, Rivers – removed • Zen discovery improvements – FULL CLUSTER RESTART • Type mappings are now strict • Index segments created before ES .90.0 must be upgraded • Migration Assistant • Units are required in settings
  13. kibana 4.2

  14. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. Dark theme Customizable maps Apps kibana 4.2: customizability
  15. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. Dark theme Offline/customizable maps Apps kibana 4.2: custom maps
  16. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. Dark theme Offline/customizable maps Apps kibana 4.2: apps
  17. www.elastic.co 16 kibana migration • Kibana 4 to Kibana 4.2

    • Support for Elasticsearch 2.x • Not backward-compatible with Elasticsearch 1.x • Dashboards are automatically migrated • Kibana 3 EOL is Nov 2015
  18. logstash 2.0

  19. www.elastic.co 18 logstash • logstash 2.0 • compatible with Elasticsearch

    2.0 • HTTP as default transport protocol • Better shutdown process § all input plugins need to be adapted
  20. beats 1.0

  21. www.elastic.co 20 beats 1.0

  22. www.elastic.co 21 demo demo.elastic.co

  23. use cases

  24. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 23 Elasticsearch @ Stagemonitor https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store
  25. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 24 Mapping { "template": "stagemonitor-metrics-*”, "settings": { "index": { "refresh_interval": "5s” } }, "mappings": { "_default_": { "dynamic_templates": [ { "strings": { "match": "*”, "match_mapping_type": "string”, "mapping": { "type": "string", "doc_values": true, "index": "not_analyzed" } } } ], "_all": { "enabled": false }, "_source": { "enabled": false },
  26. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 25 Mapping (cont.) "properties": { "@timestamp": { "type": "date", "doc_values": true }, "count": { "type": "integer", "doc_values": true, "index": "no" }, "m1_rate": { "type": "float", "doc_values": true, "index": "no" }, "m5_rate": { "type": "float", "doc_values": true, "index": "no" }, "m15_rate": { "type": "float", "doc_values": true, "index": "no" }, "max": { "type": "integer", "doc_values": true, "index": "no" }, "mean": { "type": "integer", "doc_values": true, "index": "no" }, "mean_rate": { "type": "float", "doc_values": true, "index": "no" }, "median": { "type": "float", "doc_values": true, "index": "no" }, "min": { "type": "float", "doc_values": true, "index": "no" }, "p25": { "type": "float", "doc_values": true, "index": "no" }, "p75": { "type": "float", "doc_values": true, "index": "no" }, "p95": { "type": "float", "doc_values": true, "index": "no" }, "p98": { "type": "float", "doc_values": true, "index": "no" }, "p99": { "type": "float", "doc_values": true, "index": "no" }, "p999": { "type": "float", "doc_values": true, "index": "no" }, "std": { "type": "float", "doc_values": true, "index": "no" }, "value": { "type": "float", "doc_values": true, "index": "no" }, "value_boolean": { "type": "boolean", "doc_values": true, "index": "no" }, "value_string": { "type": "string", "doc_values": true, "index": "no" } }
  27. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 26 Force merge $ curator optimize --delay 2 --max_num_segments 1 indices --older- than 1 --time-unit days --timestring %Y.%m.%d --prefix stagemonitor- metrics- • Force merge – optimize for long-term storage by merging Lucene segments on disk – happens automatically on writes – run manually after 24-48 hours – CPU intensive operation, run during off-peak hours Post Optimize: 2.2 GB => ~510 MB
  28. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 27 Elasticsearch @ MozDef SIEM https://www.elastic.co/elasticon/2015/sf/tackling-security-logs- with-the-elk-stack?q=mozilla Defender’s version of Metasploit and others Used in production at Mozilla, processes over 300M events per day Geolocate attackers Alerts Handle incidents in real time, collaborate with other administrators Integrates with other systems, e.g. to ban IP address ranges
  29. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 28 Dashboard
  30. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 29 Incident Handling
  31. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 30 Winter is Coming!
  32. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 31 MozDef Architecture
  33. stuff

  34. www.elastic.co 33 Training schedule Operations, Hands-On Workshop AMSTERDAM, Netherlands December

    1-4, 2015 Development, Operations MADRID, Spain January 19-21, 2016 Development, Operations, Kibana BERLIN, Germany January 25-28, 2016 Development, Operations, Hands- On Workshop COPENHAGEN, Denmark January 26-29, 2016 PARIS, France February 2016 LONDON, United Kingdom February 2016 AMSTERDAM, Netherlands February 2016 LONDON, United Kingdom February 2016 training.elastic.co
  35. www.elastic.co 34 Elastic{ON} 2016 $995 Early Bird Price until next

    Wednesday, Dec 2 (afterwards $1495)
  36. www.elastic.co 35 demo timelion

  37. q&a