Elasticon recap for Gotonight

In March 2015 I went to San Francisco to attend Elasticon 2015. In this presentation I give a summary of the presentations I watched. This presentation took place at Bol.com for the Gotonight.

Transcript

1. @jettroCoenradie elastic {ON} 2015 Recap
2. elastic {ON} 2015
3. 1300 Attendees
4. “WOW” –Steven Schuurman
5. Coming from 35 Countries
6. History lesson
7. The present
8. Announcements Each conference keynote has to have announcements
9. , more than Announcement Elasticsearch https://www.elastic.co/blog/elastic-you-know-for-more-than-search
10. Announcements Manage multiple clusters Provide elastic as a service within your company CloudES
11. Announcements For the past few years, Found has been busy building the best "Elasticsearch as a Service" out there.
12. Announcements https://www.elastic.co/blog/welcome-found Elastic Acquires Found https://www.found.no/elastic-acquires-found/
13. “We are thrilled and honoured to take our long-standing friendship with Elastic to the next level” – Alex Brasetvik
14. Customer cases
15. Customer cases: Verizon https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting “We offer technology products and solutions that transform the way our customers connect, collaborate and innovate” Store massive logging data Store in high rate Query in acceptable rate
16. Customer cases: Verizon 128 Nodes 8 cores - 64 Gb RAM - 6 x 1TB disk 10+ Bilion documents a day Over 500 Billion documents total
17. Customer cases: Verizon • For write performance use JBOD • Use Dedicated Master, Data and Client nodes • Use these nodes the right way • Number of nodes should not exceed 150 • To many indexes and types impact performance • Disable _all field by default Tips
18. Customer cases: Wikimedia Free text search through 888 wiki’s Near real time indexing Acceptable query performance
19. Customer cases: Wikimedia 31 Nodes 1800 indexes - 6500 shards (5 Mb - 50 Gb) 200+ Million documents (bigger than logs) 870 million searches in december 2014
20. Customer cases: Wikimedia • Don’t expose query-string query • Use instrumentation to monitor your queries • Use our plugins :-) • Swift • Experimental highlighter • Monitoring and alerting Tips https://github.com/wikimedia
21. Customer cases: Microsoft
22. Customer cases: Microsoft MSN 54 Nodes - 42 data / 3 master / 9 clients x 4 for each region 24k hits/s response within 30-50 ms
23. Customer cases: Microsoft MS Dynamics CRM - Social Listening 42 Nodes - 30 data / 3 master / 9 clients Running Ubuntu on Azure After filtering ~ 2 Million items a day
24. Customer cases: Microsoft Microsoft Azure Search PAAS - Search as a service Use Azure Portal to start 600 Clusters ~ 1000 nodes http://azure.microsoft.com/en-us/services/search/
25. Customer cases: Microsoft extensions shard allocation master election http://azure.microsoft.com/en-us/services/search/ DSL queries Language analysers
26. Customer cases: NASA http://www.jpl.nasa.gov “Examples of JPL's missions: from the Mars Curiosity rover to Cassini at Saturn, the Spitzer Space Telescope and the fleet of satellites that monitor our home planet” Store measurements Analysing these measurement
27. Customer cases: NASA
28. Customer cases: NASA SMAP - Soil Moisture Active Passive http://smap.jpl.nasa.gov/mission/why-it-matters/ Monitor Drought Predict Floods Assist Crop Productivity Weather Forecasting
29. Customer cases: NASA
30. Roadmap What is to come
31. –Shay Banon “We don’t prioritise progress over perfection. We pursue perfection in progress.”
32. Roadmap: Github
33. Roadmap: Github
34. Roadmap: Kibana
35. Roadmap: Kibana
36. Roadmap: Kibana • Formatting output: numbers, currency, urls, video • Edit and save or pin filters • Choose your own colours in charts • Visualisation builder for non elastic experts • Create API for custom plugins
37. Roadmap: Logstash • Implement internal queues that survive a crash • Dead letter queue for problem messages • Use Bulk api to send messages to elasticsearch • Create a logstash cluster • Provide API for pipeline configuration • Provide API to monitor pipeline
38. Roadmap: Elasticsearch • 1.4 -> 1.5 : Preventing you from hurting yourself • stricter field-type mappings • Two fields same name different type: same type • 1.5 has less lines of code without lost features. • Improving performance doc_values • Better error responses
39. Roadmap: Elasticsearch
40. Roadmap: Elasticsearch
41. Roadmap: Elasticsearch • Queries and Filters are merging • query = filter + scoring • Cluster state communication improvement • Only send diffs instead of full state • Reindex API • copy old index to new index • Changes API • copy changes to other cluster
42. Roadmap: Elasticsearch
43. Roadmap: Elasticsearch
44. Resilience The power or ability to return to the original form, position, etc., after being bent, compressed, or stretched; elasticity. http://dictionary.reference.com A design objective for buildings and infrastructure. It is the ability to absorb or avoid damage without suffering complete failure. http://en.wikipedia.org
45. Resilience fast slow node cluster master gone network disconnect timeouts corruption long GC kill -9 dead disk
46. Resilience • [Data corruption] Using checksums for segments improvements • [Data corruption] Check for problematic java versions • [Data corruption] Better shard allocation with multi disk setup • [Data corruption] Disk space aware allocation decider • [Out Of Memory] Circuit breaker improvement, better estimations • [Network] Refuse revived master • [external pressure] Time out long running queries
47. Resilience http://www.elastic.co/guide/en/elasticsearch/resiliency/current/ https://www.elastic.co/blog/elasticsearch-1-5-0-released https://www.elastic.co/downloads/past-releases/ elasticsearch-1-5-0
48. “SAAP : Security As A Plugin” –Uri Boness
49. Shield Leave elasticsearch as it is Lots of changes in elasticsearch to make it possible
50. Shield 1. Authentication - custom, ldap, active directory 2. Authorisation - Every endpoint is a configurable action 3. Securing the wire - SSL/TLS between nodes 4. Cluster fencing - client certificates and transport profiles 5. Audit trails - Log who did what
51. Shield 1.1 • Load balancing between ldap servers • Enable anonymous user • Update IP filters dynamically at runtime https://www.elastic.co/blog/shield-1-1-and-1-2-released
52. Shield x.x • Additional realms • Kerberos and SSO mechanisms • New native realm managed using API • Additional Authentication methods • Token/key authentication • Better Kibana integration
53. “Learned a lot, had fun while doing it.” –Jettro Coenradie
54. More information https://speakerdeck.com/elastic @jettroCoenradie jettro.coenradie@luminis.eu http://amsterdam.luminis.eu/news/ https://www.elastic.co/elasticon/2015/sf/videos?page=1