Elastic 2.0 Everything you need to know Christoph Wurm – Solutions Architect @ Elastic Nov 26, 2015 www.elastic.co Twitter: @elastic

www.elastic.co 1 Agenda 1. Elastic and Elastic Stack 2. Elastic 2.0 3. Use Cases from the field 4. Stuff 5. Q&A

www.elastic.co 2 Community 40,000 Community members 35,000 Commits

www.elastic.co 3 Downloads Mar’15 Oct’12 Apr’13 Apr’14 Oct’13 20. Millions of Downloads 10. 40+ Million Downloads Cumulative across Elastic products to date Jun’15 40. Sept’14

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 4 Elastic stack Logstash Collect, parse and enrich data Elasticsearch Store, search, analyze Kibana Visualize and explore data Hadoop Ecosystem Hadoop connector Beats Tap into your wire data Shield Security Watcher Scheduler Marvel - Monitoring Found Scale in the cloud

elasticsearch 2.0

www.elastic.co 6 elasticsearch 2.0 • Networking • multicast removed (available as plugin) • will only bind to localhost by default, both ipv4 and ipv6 • new unicast node discovery § will contact hosts listed in discovery.zen.ping.unicast.hosts (should contain all master nodes) § using first 5 ports in transport.tcp.port (default 9300-9400)

www.elastic.co 7 elasticsearch 2.0 • New feature: Pipeline aggregations • “Aggregations on the results of other aggregations” • Derivatives • Moving average • Holt Winters (prediction / anomaly detection) • Stats: Min/Max/avg • Custom

www.elastic.co 8 elasticsearch 2.0 • Moving average

www.elastic.co 9 elasticsearch 2.0 • Linear trends

www.elastic.co 10 elasticsearch 2.0 • Cyclic trends (Holt-Winters)

www.elastic.co 11 elasticsearch 2.0 • New Features • Pipeline Aggregations • Query DSL/Doc Improvement • Index Compression – 10-30% • Performance & resilience • Lucene 5.2 • Update Cluster State with diffs • Doc_values by default • Sync-flush (1.6+) • Better handling for node-leave/rejoin (1.7+) • Durability-by-default • Async shard allocation (1.6+) • Breaking Backward Compatibility • Facets, Rivers – removed • Zen discovery improvements – FULL CLUSTER RESTART • Type mappings are now strict • Index segments created before ES .90.0 must be upgraded • Migration Assistant • Units are required in settings

kibana 4.2

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. Dark theme Customizable maps Apps kibana 4.2: customizability

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. Dark theme Offline/customizable maps Apps kibana 4.2: custom maps

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. Dark theme Offline/customizable maps Apps kibana 4.2: apps

www.elastic.co 16 kibana migration • Kibana 4 to Kibana 4.2 • Support for Elasticsearch 2.x • Not backward-compatible with Elasticsearch 1.x • Dashboards are automatically migrated • Kibana 3 EOL is Nov 2015

logstash 2.0

www.elastic.co 18 logstash • logstash 2.0 • compatible with Elasticsearch 2.0 • HTTP as default transport protocol • Better shutdown process § all input plugins need to be adapted

beats 1.0

www.elastic.co 20 beats 1.0

www.elastic.co 21 demo demo.elastic.co

use cases

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 23 Elasticsearch @ Stagemonitor https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 24 Mapping { "template": "stagemonitor-metrics-*”, "settings": { "index": { "refresh_interval": "5s” } }, "mappings": { "_default_": { "dynamic_templates": [ { "strings": { "match": "*”, "match_mapping_type": "string”, "mapping": { "type": "string", "doc_values": true, "index": "not_analyzed" } } } ], "_all": { "enabled": false }, "_source": { "enabled": false },

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 25 Mapping (cont.) "properties": { "@timestamp": { "type": "date", "doc_values": true }, "count": { "type": "integer", "doc_values": true, "index": "no" }, "m1_rate": { "type": "float", "doc_values": true, "index": "no" }, "m5_rate": { "type": "float", "doc_values": true, "index": "no" }, "m15_rate": { "type": "float", "doc_values": true, "index": "no" }, "max": { "type": "integer", "doc_values": true, "index": "no" }, "mean": { "type": "integer", "doc_values": true, "index": "no" }, "mean_rate": { "type": "float", "doc_values": true, "index": "no" }, "median": { "type": "float", "doc_values": true, "index": "no" }, "min": { "type": "float", "doc_values": true, "index": "no" }, "p25": { "type": "float", "doc_values": true, "index": "no" }, "p75": { "type": "float", "doc_values": true, "index": "no" }, "p95": { "type": "float", "doc_values": true, "index": "no" }, "p98": { "type": "float", "doc_values": true, "index": "no" }, "p99": { "type": "float", "doc_values": true, "index": "no" }, "p999": { "type": "float", "doc_values": true, "index": "no" }, "std": { "type": "float", "doc_values": true, "index": "no" }, "value": { "type": "float", "doc_values": true, "index": "no" }, "value_boolean": { "type": "boolean", "doc_values": true, "index": "no" }, "value_string": { "type": "string", "doc_values": true, "index": "no" } }

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 26 Force merge $ curator optimize --delay 2 --max_num_segments 1 indices --older- than 1 --time-unit days --timestring %Y.%m.%d --prefix stagemonitor- metrics- • Force merge – optimize for long-term storage by merging Lucene segments on disk – happens automatically on writes – run manually after 24-48 hours – CPU intensive operation, run during off-peak hours Post Optimize: 2.2 GB => ~510 MB

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 27 Elasticsearch @ MozDef SIEM https://www.elastic.co/elasticon/2015/sf/tackling-security-logs- with-the-elk-stack?q=mozilla Defender’s version of Metasploit and others Used in production at Mozilla, processes over 300M events per day Geolocate attackers Alerts Handle incidents in real time, collaborate with other administrators Integrates with other systems, e.g. to ban IP address ranges

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 28 Dashboard

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 29 Incident Handling

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 30 Winter is Coming!

www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 31 MozDef Architecture

stuff

www.elastic.co 33 Training schedule Operations, Hands-On Workshop AMSTERDAM, Netherlands December 1-4, 2015 Development, Operations MADRID, Spain January 19-21, 2016 Development, Operations, Kibana BERLIN, Germany January 25-28, 2016 Development, Operations, Hands- On Workshop COPENHAGEN, Denmark January 26-29, 2016 PARIS, France February 2016 LONDON, United Kingdom February 2016 AMSTERDAM, Netherlands February 2016 LONDON, United Kingdom February 2016 training.elastic.co

www.elastic.co 34 Elastic{ON} 2016 $995 Early Bird Price until next Wednesday, Dec 2 (afterwards $1495)

www.elastic.co 35 demo timelion

q&a