Elastic 2.0

Slide deck from a series of meetups in Denmark and Sweden in November 2015.

Transcript

1. Elastic 2.0 Everything you need to know Christoph Wurm – Solutions Architect @ Elastic Nov 26, 2015 www.elastic.co Twitter: @elastic
2. www.elastic.co 1 Agenda 1. Elastic and Elastic Stack 2. Elastic 2.0 3. Use Cases from the field 4. Stuff 5. Q&A
3. www.elastic.co 2 Community 40,000 Community members 35,000 Commits
4. www.elastic.co 3 Downloads Mar’15 Oct’12 Apr’13 Apr’14 Oct’13 20. Millions of Downloads 10. 40+ Million Downloads Cumulative across Elastic products to date Jun’15 40. Sept’14
5. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 4 Elastic stack Logstash Collect, parse and enrich data Elasticsearch Store, search, analyze Kibana Visualize and explore data Hadoop Ecosystem Hadoop connector Beats Tap into your wire data Shield Security Watcher Scheduler Marvel - Monitoring Found Scale in the cloud
6. elasticsearch 2.0
7. www.elastic.co 6 elasticsearch 2.0 • Networking • multicast removed (available as plugin) • will only bind to localhost by default, both ipv4 and ipv6 • new unicast node discovery § will contact hosts listed in discovery.zen.ping.unicast.hosts (should contain all master nodes) § using first 5 ports in transport.tcp.port (default 9300-9400)
8. www.elastic.co 7 elasticsearch 2.0 • New feature: Pipeline aggregations • “Aggregations on the results of other aggregations” • Derivatives • Moving average • Holt Winters (prediction / anomaly detection) • Stats: Min/Max/avg • Custom
9. www.elastic.co 8 elasticsearch 2.0 • Moving average
10. www.elastic.co 9 elasticsearch 2.0 • Linear trends
11. www.elastic.co 10 elasticsearch 2.0 • Cyclic trends (Holt-Winters)
12. www.elastic.co 11 elasticsearch 2.0 • New Features • Pipeline Aggregations • Query DSL/Doc Improvement • Index Compression – 10-30% • Performance & resilience • Lucene 5.2 • Update Cluster State with diffs • Doc_values by default • Sync-flush (1.6+) • Better handling for node-leave/rejoin (1.7+) • Durability-by-default • Async shard allocation (1.6+) • Breaking Backward Compatibility • Facets, Rivers – removed • Zen discovery improvements – FULL CLUSTER RESTART • Type mappings are now strict • Index segments created before ES .90.0 must be upgraded • Migration Assistant • Units are required in settings
13. kibana 4.2
14. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. Dark theme Customizable maps Apps kibana 4.2: customizability
15. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. Dark theme Offline/customizable maps Apps kibana 4.2: custom maps
16. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. Dark theme Offline/customizable maps Apps kibana 4.2: apps
17. www.elastic.co 16 kibana migration • Kibana 4 to Kibana 4.2 • Support for Elasticsearch 2.x • Not backward-compatible with Elasticsearch 1.x • Dashboards are automatically migrated • Kibana 3 EOL is Nov 2015
18. logstash 2.0
19. www.elastic.co 18 logstash • logstash 2.0 • compatible with Elasticsearch 2.0 • HTTP as default transport protocol • Better shutdown process § all input plugins need to be adapted
20. beats 1.0
21. www.elastic.co 20 beats 1.0
22. www.elastic.co 21 demo demo.elastic.co
23. use cases
24. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 23 Elasticsearch @ Stagemonitor https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store
25. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 24 Mapping { "template": "stagemonitor-metrics-*”, "settings": { "index": { "refresh_interval": "5s” } }, "mappings": { "_default_": { "dynamic_templates": [ { "strings": { "match": "*”, "match_mapping_type": "string”, "mapping": { "type": "string", "doc_values": true, "index": "not_analyzed" } } } ], "_all": { "enabled": false }, "_source": { "enabled": false },
26. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 25 Mapping (cont.) "properties": { "@timestamp": { "type": "date", "doc_values": true }, "count": { "type": "integer", "doc_values": true, "index": "no" }, "m1_rate": { "type": "float", "doc_values": true, "index": "no" }, "m5_rate": { "type": "float", "doc_values": true, "index": "no" }, "m15_rate": { "type": "float", "doc_values": true, "index": "no" }, "max": { "type": "integer", "doc_values": true, "index": "no" }, "mean": { "type": "integer", "doc_values": true, "index": "no" }, "mean_rate": { "type": "float", "doc_values": true, "index": "no" }, "median": { "type": "float", "doc_values": true, "index": "no" }, "min": { "type": "float", "doc_values": true, "index": "no" }, "p25": { "type": "float", "doc_values": true, "index": "no" }, "p75": { "type": "float", "doc_values": true, "index": "no" }, "p95": { "type": "float", "doc_values": true, "index": "no" }, "p98": { "type": "float", "doc_values": true, "index": "no" }, "p99": { "type": "float", "doc_values": true, "index": "no" }, "p999": { "type": "float", "doc_values": true, "index": "no" }, "std": { "type": "float", "doc_values": true, "index": "no" }, "value": { "type": "float", "doc_values": true, "index": "no" }, "value_boolean": { "type": "boolean", "doc_values": true, "index": "no" }, "value_string": { "type": "string", "doc_values": true, "index": "no" } }
27. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 26 Force merge $ curator optimize --delay 2 --max_num_segments 1 indices --older- than 1 --time-unit days --timestring %Y.%m.%d --prefix stagemonitor- metrics- • Force merge – optimize for long-term storage by merging Lucene segments on disk – happens automatically on writes – run manually after 24-48 hours – CPU intensive operation, run during off-peak hours Post Optimize: 2.2 GB => ~510 MB
28. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 27 Elasticsearch @ MozDef SIEM https://www.elastic.co/elasticon/2015/sf/tackling-security-logs- with-the-elk-stack?q=mozilla Defender’s version of Metasploit and others Used in production at Mozilla, processes over 300M events per day Geolocate attackers Alerts Handle incidents in real time, collaborate with other administrators Integrates with other systems, e.g. to ban IP address ranges
29. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 28 Dashboard
30. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 29 Incident Handling
31. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 30 Winter is Coming!
32. www.elastic.co Elastic proprietary and confidential. Please do not share or forward. 31 MozDef Architecture
33. stuff
34. www.elastic.co 33 Training schedule Operations, Hands-On Workshop AMSTERDAM, Netherlands December 1-4, 2015 Development, Operations MADRID, Spain January 19-21, 2016 Development, Operations, Kibana BERLIN, Germany January 25-28, 2016 Development, Operations, Hands- On Workshop COPENHAGEN, Denmark January 26-29, 2016 PARIS, France February 2016 LONDON, United Kingdom February 2016 AMSTERDAM, Netherlands February 2016 LONDON, United Kingdom February 2016 training.elastic.co
35. www.elastic.co 34 Elastic{ON} 2016 $995 Early Bird Price until next Wednesday, Dec 2 (afterwards $1495)
36. www.elastic.co 35 demo timelion
37. q&a