Tales from the field: Elastic Stack and Hadoop

Hadoop is a great platform for storing and processing massive amounts of data. Elasticsearch is an ideal solution for real-time analytics and visualization of the same data. Join us to learn how you can leverage the full power of both platforms to maximize the value of your data. Learn about the ES-Hadoop connector and some common design patterns from real-world implementations.

Transcript

1. 1 Elastic stack and Hadoop Anoop Sunke Solutions Architect
2. 2 Software that makes massive amounts of structured and unstructured data usable for search, logging, analytics, and more in mission critical systems and applications
3. 3 Elastic Cloud Security Monitoring Alerting Graph X-Pack Kibana User Interface Elasticsearch Store, Index, & Analyze Ingest Logstash Beats + Elastic Stack Our Product Portfolio
4. 4 Elastic Architecture
5. 5 IT Operations Application Management Security Analytics Marketing Insights Business Development Customer Sentiment Website/App Search Internal/Intranet Search URL Search Internal Systems/Applications External Systems/Applications Developers IT/Ops Business Users Solving Many Use Cases Within Any Organization Log Analysis Analytics Search Security
6. 6 Hadoop in 60 seconds
7. 7 Hadoop 2.x 7 Hadoop Distributed File System (HDFS) YARN Map / Reduce Other
8. 8 Hadoop is great at • Massive scale • Storing and processing any and all data • Batch and Streaming • Multiple engines to analyze data • Powerful + Highly customizable programming frameworks • The ’Data lake’ paradigm 8
9. 9 Hadoop is not so great at • Random access - HDFS is sequential read/write • Indexing data for fast access and search • Tight integrations into real-time visualization • Easy real-time • Out of the box functionality • Simple developer friendly APIs • Interactive Ad-Hoc query • Data exploration 9
10. 10 The Marriage
11. 11 Elasticsearch for Apache Hadoop ™ aka ES-Hadoop
12. 12 Certified to work 12
13. 13 ES-Hadoop
14. 14 ES-Hadoop compute integrations Library / API ES-Hadoop exposed as Map/Reduce Input / OutputFormat Cascading Tap / Sink Apache Pig Loader / Storage Apache Hive (EXTERNAL) TABLE Apache Storm Spout / Bolt Apache Spark RDD, DataFrame, DataSource
15. 15 Use Case #1
16. 16 Lambda Architecture
17. 17 Lambda Architecture
18. 18 Why ? • Index Index Index ! • Native, Real-time data access – Kibana platform • Easy powerful APIs • Real-time even at scale • Simple data model - JSON • Built-in Analyzer • Out of the box functionality – Relevance, FTS • Embedding Elasticserach in custom applications • One stone, two mangoes 18
19. 19 Financial Services Company
20. 20 Bioinformatics Company
21. 21 Use Case #2
22. 22 Raw Real-time Architecture
23. 23 Why ? • Real-time monitoring and alerts • Quick insights and analytics • Elasticsearch as a transient data store • Long-term in Hadoop 23
24. 24 Medical Devices Company
25. 25 Use Case #3
26. 26 Monitoring Hadoop
27. 27 Use Case #4
28. 28 HDFS as a backup store
29. 29 ES-Hadoop Code Samples
30. 30 Apache Hive CREATE EXTERNAL TABLE playlist (name STRING, year:long) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource'='buckethead/albums','es.query'='?q=pikes'); SELECT name FROM playlist WHERE year > 2010; CREATE EXTERNAL TABLE playlist(name STRING, year:long) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource'='buckethead/albums'); INSERT INTO TABLE playlist VALUES ('Buildor', 2016), ('Florrmat', 2016);
31. 31 Apache Spark - Scala 31 import org.elasticsearch.spark._ val sc = new SparkContext(new SparkConf()) val rdd = sc.esRDD("radio/artists", "?me*") import org.elasticsearch.spark._ case class Artist(name: String, albums: Int) val u2 = Artist("U2", 12) val bh = Map("name"->"Buckethead","albums" -> 95, "age" -> 45) sc.makeRDD(Seq(u2, h2)).saveToEs("radio/artists")
32. 32 32 Apache Pig A = LOAD 'radio/artists' USING org.elasticsearch.hadoop.pig.EsStorage('es.query=?q=me*'); DUMP A; A = LOAD 'src/artists.dat' USING PigStorage() AS (id:long, name, url:chararray, picture: chararray); B = FOREACH A GENERATE name, TOTUPLE(url, picture) AS links; STORE B INTO 'radio/artists' USING org.elasticsearch.hadoop.pig.EsStorage();
33. 33 Apache Spark - Java 33
34. 34 ES-Hadoop Internals
35. 35 Partition-to-Partition Architecture
36. 36 Dynamic Runtime Matching
37. 37 Failure Handling
38. 38 Co-location
39. 39 Website: www.elastic.co Products: https://www.elastic.co/products Forums: https://discuss.elastic.co/ Community: https://www.elastic.co/community/meetups Twitter: @elastic Thank You