What's the Scoop on ES-Hadoop? Spark, Streaming & More

Elasticsearch is an industry-leading solution for search and real-time analytics at scale. Apache Spark has shaped into a powerhouse for processing massive data, both in batch and streaming contexts. Elasticsearch for Apache Hadoop (ES-Hadoop) is a two-way connector that provides the tools needed to marry these two together in perfect data harmony.

This talk aims to introduce the audience to the basics of ES-Hadoop’s native Spark Integration, touch upon the other features that the connector brings to the table (including native integrations with Hive, Storm, Pig, Cascading, and MapReduce), shed some light on the internals of how it works, as well as highlight what’s to come.

James Baiera l Software Engineer l Elastic
Anoop Sunke l Solutions Architect l Elastic

Transcript

1. Elastic 2017-03-09 What’s the Scoop with ES- Hadoop? James Baiera and Anoop Sunke @JimmyThaHat && @apsunke
2. 2 James Baiera • Software Engineer @ Elastic • ES-Hadoop Maintainer • @jbaiera on Github • @JimmyThaHat on Twitter
3. Agenda 3 45 minute block 1 Introduction to ES-Hadoop 2 Anatomy of a Job Execution 3 Connector Feature Tour and What’s to Come 4 User Success Stories and Use Cases 5 Q&A
4. Hadoop Overview In two minutes or less
5. Hadoop Ecosystem
6. Hadoop Ecosystem Hadoop Distributed File System (HDFS) YARN Map / Reduce Other
7. Introduction to ES-Hadoop
8. { } Elasticsearch for Apache Hadoop is an open-source, stand-alone, self-contained, small library that allows Hadoop jobs to interact with Elasticsearch.
9. Elasticsearch for Apache Hadoop is an open-source, stand-alone, self-contained, small library that allows Hadoop jobs to interact with Elasticsearch. { }
10. Elasticsearch for Apache Hadoop is an open-source, stand-alone, self-contained, small library that allows Hadoop jobs to interact with Elasticsearch. { }
11. Elasticsearch for Apache Hadoop is an open-source, stand-alone, self-contained, small library that allows Hadoop jobs to interact with Elasticsearch. { }
12. Elasticsearch for Apache Hadoop is an open-source, stand-alone, self-contained, small library that allows Hadoop jobs to interact with Elasticsearch. { }
13. Birds Eye View 13
15. ES-Hadoop Integrations 15 Library / API ES-Hadoop Exposed As MapReduce Input/OutputFormat Cascading Tap/Sink Apache Pig Storage (Load and Store) Apache Hive EXTERNAL Table Apache Storm Spout/Bolt Apache Spark RDD, DStream, Dataframe, Dataset, DataSource
16. • Latest Spark 2.1 support • Adaptive I/O for error handling, re-routing, backpressure • Push-down processing on either platform • Co-location, rack awareness • Elastic security compatible - basic authentication, SSL/TLS, PKI • Hadoop Kerberos security compatible • Hadoop distribution agnostic ES-Hadoop Features
17. Why?
18. Peeling Back the Layers ES-Hadoop Functional Deep Dive
19. Pushdown
20. Projections and Predicates 20 ID : Long First Name : String Age : Integer Profile : Text 347 Martha 32 Has a fancy vineyard 348 Mary 37 Friends with Peter and Paul 349 Geoff 25 Spells name with a ‘G’ 350 Travis 23 Loves jazz music 351 Jeremiah 39 Bullfrog, Good Friend 352 Mark 42 Hates cold spaghetti
21. Predicates 21 ID : Long First Name : String Age : Integer Profile : Text 347 Martha 32 Has a fancy vineyard 348 Mary 37 Friends with Peter and Paul 349 Geoff 25 Spells name with a ‘G’ 350 Travis 23 Loves jazz music 351 Jeremiah 39 Bullfrog, Good Friend 352 Mark 42 Hates cold spaghetti Predicate: ID > 347 && ID <= 350
22. Projections 22 ID : Long FirstName : String Age : Integer Profile : Text 347 Martha 32 Has a fancy vineyard 348 Mary 37 Friends with Peter and Paul 349 Geoff 25 Spells name with a ‘G’ 350 Travis 23 Loves jazz music 351 Jeremiah 39 Bullfrog, Good Friend 352 Mark 42 Hates cold spaghetti Projection: Select (ID, FirstName, Age)
23. Final View 23 ID : Long First Name : String Age : Integer Profile : Text 347 Martha 32 Has a fancy vineyard 348 Mary 37 Friends with Peter and Paul 349 Geoff 25 Spells name with a ‘G’ 350 Travis 23 Loves jazz music 351 Jeremiah 39 Bullfrog, Good Friend 352 Mark 42 Hates cold spaghetti Select (ID, FirstName, Age) where ID > 347 && ID <= 350
24. val df = sqlContext.read().format("es").load("spark/users") df.printSchema() // root //|-- name: string (nullable = true) //|-- id: long (nullable = true) //|-- profile: string (nullable = true) val filter = df.filter( df("name").equalTo("James").and(df("id").gt(200))) Predicate Pushdown Spark SQL Reading 24
25. Spark SQL Reading Predicate Pushdown And-Predicate Equal-To Greater-Than “name” “James” “id” 200
26. { "query" : { "bool" : { "must" : [ { "match_all" : {}} ], "filter" : [ { "bool":{"filter":[ {"match" : { "name" : "James"}}, {"range" : { "id" : { "gt" : 200 }}} ] } } } } Predicate Pushdown Spark SQL Reading 26
27. val df = sqlContext.read().format("es").load("spark/users") df.createOrReplaceTempView(“myIndex”) df.printSchema() // root //|-- name: string (nullable = true) //|-- id: long (nullable = true) //|-- profile: string (nullable = true) val names = sqlContext.sql("SELECT name, id FROM myIndex") Spark SQL Reading 27 Projection Pushdown
28. { "_source": [ "name", "id" ], "query" : { "bool":{ "must" : [ { "match_all" : {}} ] } } } Projection Pushdown Spark SQL Reading 28
29. val df = sqlContext.read().format("es").load("spark/users") df.createOrReplaceTempView(“myIndex”) df.printSchema() // root //|-- name: string (nullable = true) //|-- id: long (nullable = true) //|-- profile: string (nullable = true) val names = df.sqlContext.sql( "SELECT name FROM myIndex WHERE id >=1 AND id <= 10") Spark SQL Reading 29 Projection Pushdown + Predicate Pushdown
30. { "_source": [ "name", "id" ], "query" : { "bool":{ "must" : [ { "match_all" : {}} ], "filter" : [ { "bool":{"filter":[ { "range" : { "id" : { "gte" : 1 }}}, { "range" : { "id" : { "lte" : 10 }}} ]} }] } } } Projection Pushdown + Predicate Pushdown Spark SQL Reading 30
31. Automatic Pushdown Support in ES-Hadoop 31 Library / API Projection Predicate MapReduce Manual Manual Cascading Automatic Manual Apache Pig Automatic Manual Apache Hive Automatic Manual Apache Storm Manual Manual Apache Spark (SQL) Automatic (RDD) Manual (SQL) Automatic (RDD) Manual
32. Node Discovery
33. Master Master Master Client Data Data Data Data Data
34. Master Master Master Client Data Data Data Data Data Driver
35. Master Master Master Client Data Data Data Data Data Driver Initial Nodes Set
36. Master Master Client Data Data Data Data Data Driver Master Discovery
37. Master Master Client Data Data Data Data Data Driver Master Discovery Master Master Master Client
38. Master Master Client Data Data Data Data Driver Master Discovery Wow, look at all these nodes. Data
39. Master Master Client Data Data Data Data Data Driver Master Discovery Don’t need to talk to these...
40. Master Master Client Data Data Data Data Data Driver Master Discovery Nope, don’t need this one either...
41. Data Data Data Data Data Driver
42. Reading from Elasticsearch
43. Data Data Data Data Data Data Data Data Data Driver Finding Partitions Reading index “logs/data”...
44. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 Reading index “logs/data”... = Primary = Replica # #
45. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # 1 1 1 1 1 # Reading index “logs/data”...
46. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 Reading index “logs/data”... Finding shard sizes… (NEW in 5.0!) Repeat for each unique shard = Primary = Replica # #
47. Finding Partitions Reading index “logs/data”... Subdividing shards… (NEW in 5.0!) 1 2 3
48. Finding Partitions Reading index “logs/data”... Subdividing shards… (NEW in 5.0!) 1 2 3
49. NEW IN 5.0 OF ELASTICSEARCH
50. curl -XGET localhost:9200/idx/t/_search?scroll=1m -d' { "slice": { "id": 0, "max": 2 }, "query": { "match" : { "title" : "elasticsearch" } } }' New in Elasticsearch 5.0 Sliced Scrolls 50
51. Finding Partitions Reading index “logs/data”... Subdividing shards… (NEW in 5.0!) 1 2 3
52. Finding Partitions Reading index “logs/data”... Subdividing shards… (NEW in 5.0!) 1 2 3 You must be this tall to ride. 100,000 Documents
53. Finding Partitions Reading index “logs/data”... Subdividing shards… (NEW in 5.0!) 1 2 - 1 2 - 2 3 You must be this tall to ride. 100,000 Documents
54. Finding Partitions Reading index “logs/data”... Subdividing shards… (NEW in 5.0!) 1 2 - 1 3 2 - 2 100,000 Documents You must be this tall to ride.
55. Driver Job Execution Launching Tasks… 1 2-1 2-2 3
56. Job Execution Data Data Data Data Data 1 2 3 2 1 3 # # = Primary = Replica 3 1 2-1 2-2
57. Job Execution Data Data Data Data Data 1 2 3 2 1 3 3 1 2-1 2-2 Rack 1 Rack 2
58. Job Execution Data Data Data Data Data 1 2 3 2 1 3 # # = Primary = Replica 3 1 2-1 2-2
59. Communication Failure Data Data Data Data Data 1 2 3 2 1 3 # # = Primary = Replica !!!! 3 1 2-1 2-2
60. Communication Failure Data Data Data Data Data 1 2 3 2 1 3 # # = Primary = Replica 3 1 2-1 2-2
61. Writing to Elasticsearch
62. Finding Partitions Writing to index ... = Primary = Replica # # Data 1 Data 1 Data Data 1 Data
63. Finding Partitions Writing to index ... = Primary = Replica # # Data 1 Data 1 Data Data 1 Data 1
64. Finding Partitions Writing to index ... = Primary = Replica # # Data 1 Data Data Data 1 1 Data 1 2
65. Finding Partitions Writing to index ... = Primary = Replica # # Data 1 Data 1 Data Data 1 Data 1 2 3 3
66. Finding Partitions Writing to index ... = Primary = Replica # # Data 1 Data 1 Data Data 1 Data 1 2 3 3
67. Finding Partitions Writing to index ... = Primary = Replica # # Data 1 Data 1 Data Data 1 Data 2 2 1
68. Data Data Data Data Data Driver Finding Partitions Writing to index ...
69. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # Writing to index ...
70. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # 1 1 1 1 1 # Writing to index ...
71. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # 1 1 1 1 1 # Writing to index ... # # #
72. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # 1 1 1 1 1 # Writing to index ... # # #
73. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # 1 1 1 1 1 # Writing to index ... # # #
74. Data Data Data Data Data Driver Finding Partitions 1 2 3 2 1 3 = Primary = Replica # # Writing to index ...
75. Job Execution Data Data Data Data Data 1 2 3 2 1 3 # # = Primary = Replica
76. Feature Tour Code Examples
77. CREATE EXTERNAL TABLE artists ( id BIGINT, name STRING, links STRUCT<url:STRING, picture:STRING>) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES( 'es.resource' = 'radio/artists', 'es.query' = '?q=me*'); -- stream data from Elasticsearch SELECT * FROM artists; Reading from Elasticsearch Apache Hive 77
78. -- load data from HDFS into Pig using a schema A = LOAD 'src/test/resources/artists.dat' USING PigStorage() AS (id:long, name, genre, url:chararray, picture:chararray); -- transform data B = FOREACH A GENERATE name, genre, TOTUPLE(url, picture) AS links; -- save the result to Elasticsearch STORE B INTO 'radio/{genre}' USING org.elasticsearch.hadoop.pig.EsStorage(‘es.mapping.id=name’); Writing to Elasticsearch Apache Pig 78
79. import org.elasticsearch.spark._ val conf = ... val sc = new SparkContext(conf) val rdd = sc.esRDD("radio/artists") rdd.take(10) Reading from Elasticsearch Apache Spark 79
80. val sc = new SparkContext(conf) val ssc = new StreamingContext(sc, Seconds(1)) ssc.socketTextStream("127.0.0.1", 9999) .foreachRDD(EsSpark.saveToEs(_, "netcat/data")) ssc.start() DStreams: The Old Way Apache Spark Streaming 80
81. import org.elasticsearch.spark.streaming._ val sc = new SparkContext(conf) val ssc = new StreamingContext(sc, Seconds(1)) ssc.socketTextStream("127.0.0.1", 9999) .saveJsonToEs("netcat/data") ssc.start() DStream Native Integration (NEW IN 5.0) Apache Spark Streaming 81
82. import org.elasticsearch.spark.streaming._ val sc = new SparkContext(conf) val ssc = new StreamingContext(sc, Seconds(1)) val jobConf = Map("es.ingest.pipeline" -> "hadoop_test") ssc.socketTextStream("127.0.0.1", 9999) .saveJsonToEs("netcat/data", jobConf) ssc.start() Configurations Added for Ingest (NEW IN 5.0) Ingest Node Support 82
83. •Support for Unicode Index/Type Names •Source Filtering for RDD’s and MapReduce •User Specified HTTP Headers •Spark Structured Streaming Integration Coming Soon™
84. Common Use Cases Anoop Sunke
85. Anoop Sunke • Solutions Architect @ Elastic • Ex-Hortonworks, Microsoft • @apsunke on twitter
86. Use Case #1 Lambda Architecture 86
87. Lambda Architecture 87
88. Lambda Architecture 88
89. Why ? 89 • High scale Ingest rate - 1 Million+ Events Per Second • Data available in 1 second, default refresh interval • Simple data model – JSON • Flexible schema - Add fields anytime • Kibana - Native real time exploration, even at scale • REST API first data store – Fast Aggregations, Custom UI, Embedding • Index Index Index ! • Out of the box functionality - sharding, rebalancing, easy scaling
90. Financial Services company 90
91. Bioinformatics Company 91
92. Use Case #2 Raw Real-time 92
93. Raw real-time Architecture 93
94. Medical Devices Company 94
95. Use Case #3 Monitoring Hadoop 95
96. Monitoring Hadoop 96
97. Use Case #4 HDFS as a backup store 97
98. HDFS as a backup store 98
99. Thank You!
100. 100 Visit us on our forums! https://discuss.elastic.co/c/elasticsearch-and-hadoop
101. www.elastic.c o
102. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.