Tales from the field - Elastic Stack and Hadoop (v2.0)

Transcript

1. Tales from the Field: Elastic stack and Hadoop
2. Anoop Sunke Solutions Architect James Baiera Software Engineer
3. Housekeeping & Logistics Slides and recording will be available following the webinar Chat via IRC #elastic-webinar #elastic-webinar @ Freenode Click ”Join the Chat” link, create an IRC account Set higher youtube resolution
4. Elastic Company Update 3
5. 80M+ Product Downloads 3,000+ Subscription Customers 75,000+ Community Members Statistics since 2012, founding of Elastic 4
6. 2014 Millions of Downloads 40. 80. 2016 2015 2012 2013 Cumulative downloads of the Elastic Stack (Elasticsearch, Kibana, Beats, Logstash) and X-Pack 5
7. 6 Elastic Stack 100% open source No enterprise edition All new versions with 5.0
8. 7 X-Pack Single install Extensions for the Elastic Stack Subscription pricing Security Alerting Monitoring Reporting Graph
9. Elastic Stack Architecture 8
10. Elastic Stack Architecture
11. Hadoop in 60 seconds 10
12. Hadoop 2.x 11
13. Hadoop is great at • Massive scale • Storing and processing any and all data • Schema on read • Batch and Streaming, Multiple engines to analyze data • Powerful + Highly customizable programming frameworks • The ’Data lake’ paradigm • Cost effective 12
14. Hadoop is not so great at • Random access - HDFS is sequential read/write • Indexing data for fast access and search • Tight integrations into real-time visualization • Easy real-time • Simple developer friendly APIs except Apache Spark 13
15. Elastic stack is good at • Random access - HDFS is sequential read/write • Indexing data for fast access and search • Tight integrations into real-time visualization • Easy real-time • Simple developer friendly APIs except Apache Spark 14
16. The Marriage 15
17. 16 ES-Hadoop Elasticsearch for Hadoop Two-way connector Index Hadoop data in Elasticsearch Enable real-time search capabilities Visualize data in Kibana Read/Write directly to/from Kafka Support for Spark, Storm MapReduce, and more Elasticsearch for Apache Hadoop
18. Elasticsearch for Apache Hadoop
19. Certified to work 18
20. ES-Hadoop compute integrations
21. Use Case #1 20
22. Lambda Architecture
23. Common requirements for the fast data store 22 • High throughput ingest • Make data readable quickly • Flexible schema • Dashboards for Monitoring • Rich API’s for building custom UI’s • Fast aggregations
24. Lambda Architecture
25. Why ? • High scale Ingest rate - 1 Million+ Events Per Second • Data available in 1 second, default refresh interval • Simple data model – JSON • Flexible schema - Add fields anytime • Kibana - Native real time exploration, even at scale • API first data store – Fast Aggregations, Custom UI, Embedding • Index Index Index ! • Out of the box functionality - sharding, rebalancing, easy scaling • One stone, two mangoes 24
26. Financial Services Company
27. Bioinformatics Company
28. Use Case #2 27
29. Raw Real-time Architecture
30. Why ? • Real-time monitoring and alerts • Quick insights and analytics • Serving/ Presentation layer • Long-term in Hadoop • Elasticsearch as a transient data store 29
31. Medical Devices Company
32. Use Case #3 31
33. Monitoring Hadoop
34. Use Case #4 33
35. HDFS as a backup store
36. ES-Hadoop Code Samples 35
37. Apache Hive
38. 37 Apache Pig
39. Apache Spark - Scala 38
40. ES-Hadoop Internals 39
41. Partition-to-Partition Architecture
42. Dynamic Runtime Matching
43. Failure Handling
44. Questions? 43 Log into IRC to ask questions • #elastic-webinar @ Freenode • Click ”Join the Chat” link, create an IRC account
45. THANK YOU Web : www.elastic.co Products : https://www.elastic.co/products Forums : https://discuss.elastic.co/ Community : https://www.elastic.co/community/meetups Twitter : @elastic