Elastic 2.0

Slide deck from a series of meetups in Denmark and Sweden in November 2015.

Transcript

1. Elastic 2.0
   Everything you need to know
   Christoph Wurm – Solutions Architect @ Elastic
   Nov 26, 2015
   www.elastic.co
   Twitter: @elastic
   

   View Slide

2. www.elastic.co
   1
   Agenda
   1. Elastic and Elastic Stack
   2. Elastic 2.0
   3. Use Cases from the field
   4. Stuff
   5. Q&A
   

   View Slide

3. www.elastic.co
   2
   Community
   40,000
   Community
   members
   35,000
   Commits
   

   View Slide

4. www.elastic.co
   3
   Downloads
   Mar’15
   Oct’12 Apr’13 Apr’14
   Oct’13
   20.
   Millions of Downloads
   10.
   40+ Million Downloads
   Cumulative across Elastic
   products to date
   Jun’15
   40.
   Sept’14
   

   View Slide

5. www.elastic.co
   Elastic proprietary and confidential.
   Please do not share or forward.
   4
   Elastic stack
   Logstash
   Collect, parse
   and enrich data
   Elasticsearch
   Store, search, analyze
   Kibana
   Visualize and explore data
   Hadoop Ecosystem
   Hadoop connector Beats
   Tap into your
   wire data
   Shield
   Security
   Watcher
   Scheduler
   Marvel - Monitoring
   Found
   Scale in the cloud
   

   View Slide

6. elasticsearch 2.0
   

   View Slide

7. www.elastic.co
   6
   elasticsearch 2.0
   • Networking
   • multicast removed (available as plugin)
   • will only bind to localhost by default, both ipv4 and ipv6
   • new unicast node discovery
   § will contact hosts listed in discovery.zen.ping.unicast.hosts (should contain all master
   nodes)
   § using first 5 ports in transport.tcp.port (default 9300-9400)
   

   View Slide

8. www.elastic.co
   7
   elasticsearch 2.0
   • New feature: Pipeline aggregations
   • “Aggregations on the results of other aggregations”
   • Derivatives
   • Moving average
   • Holt Winters (prediction / anomaly detection)
   • Stats: Min/Max/avg
   • Custom
   

   View Slide

9. www.elastic.co
   8
   elasticsearch 2.0
   • Moving average
   

   View Slide

10. www.elastic.co
    9
    elasticsearch 2.0
    • Linear trends
    

    View Slide

11. www.elastic.co
    10
    elasticsearch 2.0
    • Cyclic trends (Holt-Winters)
    

    View Slide

12. www.elastic.co
    11
    elasticsearch 2.0
    • New Features
    • Pipeline Aggregations
    • Query DSL/Doc Improvement
    • Index Compression – 10-30%
    • Performance & resilience
    • Lucene 5.2
    • Update Cluster State with diffs
    • Doc_values by default
    • Sync-flush (1.6+)
    • Better handling for node-leave/rejoin (1.7+)
    • Durability-by-default
    • Async shard allocation (1.6+)
    • Breaking Backward Compatibility
    • Facets, Rivers – removed
    • Zen discovery improvements – FULL
    CLUSTER RESTART
    • Type mappings are now strict
    • Index segments created before ES .90.0
    must be upgraded
    • Migration Assistant
    • Units are required in settings
    

    View Slide

13. kibana 4.2
    

    View Slide

14. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    Dark theme
    Customizable maps
    Apps
    kibana 4.2: customizability
    

    View Slide

15. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    Dark theme
    Offline/customizable maps
    Apps
    kibana 4.2: custom maps
    

    View Slide

16. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    Dark theme
    Offline/customizable maps
    Apps
    kibana 4.2: apps
    

    View Slide

17. www.elastic.co
    16
    kibana migration
    • Kibana 4 to Kibana 4.2
    • Support for Elasticsearch 2.x
    • Not backward-compatible with Elasticsearch 1.x
    • Dashboards are automatically migrated
    • Kibana 3 EOL is Nov 2015
    

    View Slide

18. logstash 2.0
    

    View Slide

19. www.elastic.co
    18
    logstash
    • logstash 2.0
    • compatible with Elasticsearch 2.0
    • HTTP as default transport protocol
    • Better shutdown process
    § all input plugins need to be adapted
    

    View Slide

20. beats 1.0
    

    View Slide

21. www.elastic.co
    20
    beats 1.0
    

    View Slide

22. www.elastic.co
    21
    demo
    demo.elastic.co
    

    View Slide

23. use cases
    

    View Slide

24. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    23
    Elasticsearch @ Stagemonitor https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store
    

    View Slide

25. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    24
    Mapping
    {
    "template": "stagemonitor-metrics-*”,
    "settings": {
    "index": {
    "refresh_interval": "5s”
    }
    },
    "mappings": {
    "_default_": {
    "dynamic_templates": [
    {
    "strings": {
    "match": "*”,
    "match_mapping_type": "string”,
    "mapping": { "type": "string", "doc_values": true,
    "index": "not_analyzed" }
    }
    }
    ],
    "_all": { "enabled": false },
    "_source": { "enabled": false },
    

    View Slide

26. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    25
    Mapping (cont.)
    "properties": {
    "@timestamp": { "type": "date", "doc_values": true },
    "count": { "type": "integer", "doc_values": true, "index": "no" },
    "m1_rate": { "type": "float", "doc_values": true, "index": "no" },
    "m5_rate": { "type": "float", "doc_values": true, "index": "no" },
    "m15_rate": { "type": "float", "doc_values": true, "index": "no" },
    "max": { "type": "integer", "doc_values": true, "index": "no" },
    "mean": { "type": "integer", "doc_values": true, "index": "no" },
    "mean_rate": { "type": "float", "doc_values": true, "index": "no" },
    "median": { "type": "float", "doc_values": true, "index": "no" },
    "min": { "type": "float", "doc_values": true, "index": "no" },
    "p25": { "type": "float", "doc_values": true, "index": "no" },
    "p75": { "type": "float", "doc_values": true, "index": "no" },
    "p95": { "type": "float", "doc_values": true, "index": "no" },
    "p98": { "type": "float", "doc_values": true, "index": "no" },
    "p99": { "type": "float", "doc_values": true, "index": "no" },
    "p999": { "type": "float", "doc_values": true, "index": "no" },
    "std": { "type": "float", "doc_values": true, "index": "no" },
    "value": { "type": "float", "doc_values": true, "index": "no" },
    "value_boolean": { "type": "boolean", "doc_values": true, "index": "no" },
    "value_string": { "type": "string", "doc_values": true, "index": "no" }
    }
    

    View Slide

27. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    26
    Force merge
    $ curator optimize --delay 2 --max_num_segments 1 indices --older-
    than 1 --time-unit days --timestring %Y.%m.%d --prefix stagemonitor-
    metrics-
    • Force merge
    – optimize for long-term storage by merging Lucene segments on disk
    – happens automatically on writes
    – run manually after 24-48 hours
    – CPU intensive operation, run during off-peak hours
    Post Optimize: 2.2 GB => ~510 MB
    

    View Slide

28. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    27
    Elasticsearch @ MozDef SIEM https://www.elastic.co/elasticon/2015/sf/tackling-security-logs-
    with-the-elk-stack?q=mozilla
    Defender’s version of Metasploit and
    others
    Used in production at Mozilla, processes
    over 300M events per day
    Geolocate attackers
    Alerts
    Handle incidents in real time, collaborate
    with other administrators
    Integrates with other systems, e.g. to ban
    IP address ranges
    

    View Slide

29. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    28
    Dashboard
    

    View Slide

30. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    29
    Incident Handling
    

    View Slide

31. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    30
    Winter is Coming!
    

    View Slide

32. www.elastic.co
    Elastic proprietary and confidential.
    Please do not share or forward.
    31
    MozDef Architecture
    

    View Slide

33. stuff
    

    View Slide

34. www.elastic.co
    33
    Training schedule
    Operations, Hands-On Workshop AMSTERDAM, Netherlands December 1-4, 2015
    Development, Operations MADRID, Spain January 19-21, 2016
    Development, Operations, Kibana BERLIN, Germany January 25-28, 2016
    Development, Operations, Hands-
    On Workshop
    COPENHAGEN, Denmark January 26-29, 2016
    PARIS, France February 2016
    LONDON, United Kingdom February 2016
    AMSTERDAM, Netherlands February 2016
    LONDON, United Kingdom February 2016
    training.elastic.co
    

    View Slide

35. www.elastic.co
    34
    Elastic{ON} 2016
    $995 Early Bird Price until next Wednesday, Dec 2 (afterwards $1495)
    

    View Slide

36. www.elastic.co
    35
    demo
    timelion
    

    View Slide

37. q&a
    

    View Slide