Getting started with Apache Spark (Developer's edition)

The slides from the talk at Confoo Montreal at 2019.

Transcript

1. Getting started (Developer’s edition)
2. R E D V E N T U R E S Majid Fatemian mfatemian@redventures.com aws CERTIFIED - SAA @majidfn
3. TECH & DATA-DRIVEN CUSTOMER ACQUISITION SALES & MARKETING R E D V E N T U R E S
4. MACHINE
 LEARNING PRODUCT
 RECOMMENDATION DATA
 SCIENCE REALTIME
 ANALYSIS USER
 EXPERIENCE DATA
 WAREHOUSE DATA
5. WHY THIS TALK?
6. UNIFIED COMPUTING ENGINE for large-scale distributed data processing
7. EXAMPLE
8. PROBLEM ➤ 10K log files ➤ JSON ➤ S3 ➤ Average latency for each user
9. PROBLEM ➤ 10K log files ➤ JSON ➤ S3 ➤ Average Latency for each user {"userId": 1, "url": "page1", "latency": 30}
 {"userId": 1, "url": "page2", "latency": 24} {"userId": 4, "url": "page1", "latency": 40} {"userId": 4, "url": "page5", "latency": 53}
10. SOLUTION - IMPERATIVE / PROCEDURAL // Import AWS SDK. // Define Log Struct. users = map[user][sum][count] for file in s3.ListFiles("s3://logs-bucket") { content = s3.ReadFile(file) for line in content { log = Unmarshal(line) users[log.UserId].sum += log.latency users[log.UserId].count++ } } for user in users { print user, sum / count }
11. SOLUTION - APACHE SPARK (DECLARATIVE) logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId).avg(logs.latency)
12. ABSTRACTION That’s all it’s about
13. DISTRIBUTED
14. SPARK CLUSTER
15. DRIVER EXECUTOR EXECUTOR EXECUTOR
16. DRIVER EXECUTOR EXECUTOR EXECUTOR
17. DRIVER EXECUTOR EXECUTOR EXECUTOR JVM JVM JVM JVM
18. DRIVER EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT JVM JVM JVM JVM
19. DRIVER EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT CLUSTER
 MANAGER
20. DRIVER EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT CLUSTER
 MANAGER Spark
 Mesos
 YARN
 Kubernetes (exp) {
21. DRIVER EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT EXECUTOR SLOT SLOT CLUSTER
 MANAGER
22. JOB / STAGE / TASK
23. logs = spark.read.json("s3://logs-bucket") logs.groupBy(logs.userId).avg(logs.latency)
24. STAGE 1 STAGE 2 STAGE 3 STAGE 4 JOB #1
25. STAGE 1 STAGE 2 STAGE 3 STAGE 4 JOB #1 TASK #1 … #2 #3
26. DRIVER EXECUTOR EXECUTOR EXECUTOR
27. PARTITION
31. DRIVER EXECUTOR EXECUTOR EXECUTOR
32. DRIVER EXECUTOR SLOT EXECUTOR EXECUTOR
33. UNIFIED
34. SQL Streaming Machine Learning Graph
35. Spark Core Engine Spark
 SQL Spark
 Streaming MLlib GraphX
36. Spark Core Engine Spark
 SQL Spark
 Streaming MLlib GraphX
37. Spark Core Engine Spark
 SQL Spark
 Streaming MLlib GraphX R SQL Python Scala Java
38. UNIFIED ➤ Supports wide range of tasks ➤ Interactive and iterative ad-hoc queries ➤ Advanced ML ➤ Graph processing ➤ Streaming ➤ Consistent set of APIs across languages
39. COMPUTING Engine
40. COMPUTING ➤ Limits its scope to computing / analytics ➤ No storage ➤ Apache Hadoop: storage + computation (HDFS) (MapReduce)
41. DATA SOURCES
42. DATA SOURCES - JSON FROM S3 spark .format("json") .read("s3a://logs-bucket")
43. DATA SOURCES - JSON FROM FILE SYSTEM spark .format("json") .read("/path/logs-folder")
44. DATA SOURCES - CSV FROM S3 spark .format("csv") .read("s3a://logs-bucket-csvs")
45. DATA SOURCES - SQL DB - JDBC spark .read .jdbc(jdbcUrl, "users", conProps)
46. DATA SOURCES - CASSANDRA spark .format("org.apache.spark.sql.cassandra") .options(Map( "table" -> "X", "keyspace" -> "Y")) .read()
47. OTHER CONNECTORS ➤ MongoDB ➤ Kafka ➤ ElasticSearch ➤ Neo4j ➤ Redis ➤ SnowFlake ➤ Network socket
48. OTHER FILE FORMATS ➤ Avro ➤ Parquet ➤ Zip files ➤ LZO compressed
49. DATA SINKS
50. DATA SINKS - JSON TO S3 spark .format("json") .write("s3a://logs-result")
51. DATA SINKS - SQL DB - JDBC spark .write .jdbc(jdbcUrl, "users", conProps)
52. API
53. Spark Core Engine RDD APIs DataFrame | SQL | DataSet APIs Structured Streaming | Spark ML | GraphFrames
54. Spark Core Engine
55. Spark Core Engine RDD APIs
56. RDD Resilient Distributed Dataset
57. RDD ➤ Main abstraction in Spark ➤ Collections of distributed items ➤ Parallelism ➤ Fault tolerance
58. Spark Core Engine RDD APIs DataFrame APIs
59. DATAFRAME
60. DATAFRAME ➤ Structured API ➤ Table: rows & columns ➤ Spreadsheet, but distributed
61. logs = spark.read.json("s3a://logs-bucket")
62. logs = spark.read.json("s3a://logs-bucket")
63. EXECUTOR EXECUTOR EXECUTOR
64. EXECUTOR EXECUTOR EXECUTOR
65. TRANSFORMATIONS
66. TRANSFORMATIONS ➤ filter ➤ map ➤ join ➤ groupByKey ➤ sortByKey ➤ …
67. TRANSFORMATIONS ➤ Data is immutable in Spark
68. logs = spark.read.json("s3a://logs-bucket")
69. logs = spark.read.json("s3a://logs-bucket")
70. late_logs = logs.where(“latency > 30”) logs = spark.read.json("s3a://logs-bucket")
71. late_logs = logs.where(“latency > 30”)
72. NARROW TRANSFORMATIONS One input partition contributes to one output In-memory Like (usually): where, filter, map
73. WIDE TRANSFORMATIONS One input Partition contributes to many outputs (shuffle) Write results to disk Like (usually): groupByKey, join, aggregation
74. ACTIONS
75. ACTIONS ➤ count ➤ collect ➤ take ➤ saveAsTextFile ➤ …
76. LAZY Evaluation
77. logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency)
78. logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency) .show()
79. logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency) .show() Action
80. logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency) .show() Optimization Logical
 Plan Optimization Physical
 Plan
81. logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency) .show() CSV Read GroupBy Avg DATAFRAME DATAFRAME DATAFRAME Show
82. Spark Core Engine RDD APIs DataFrame | SQL APIs
83. SQL
84. logs = spark.read.json("...")
85. logs = spark.read.json("...") logs.createOrReplaceTempView("logs")
86. logs = spark.read.json("...") logs.createOrReplaceTempView("logs") sqlDF = spark.sql(""" SELECT userId, AVG(latency) FROM logs GROUP BY userId """)
87. logs = spark.read.json("...") logs.createOrReplaceTempView("logs") sqlDF = spark.sql(""" SELECT userId, AVG(latency) FROM logs GROUP BY userId """) sqlDF.show()
88. Spark Core Engine RDD APIs DataFrame | SQL | DataSet APIs Structured Streaming | Spark ML | GraphFrames
89. STRUCTURED STREAMING
90. PROBLEM ➤ 10K log files & growing ➤ JSON ➤ S3 ➤ Average latency for each user
91. Scan Files Aggregate Write to Destination Batch
92. Scan Files Aggregate Write to Destination Scan New Files Stateful
 Aggregate Update Destination Batch Continuous
93. STRUCTURED STREAMING ➤ Fault tolerance ➤ Checkpointing ➤ Continuous DataFrame
94. BATCH AGGREGATION ➤ Re-Calculate / Scan the whole dataset logs = spark.read.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency)
95. CONTINUOUS AGGREGATION logs = spark.readStream.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency)
96. CONTINUOUS AGGREGATION logs = spark.readStream.json("s3a://logs-bucket") logs.groupBy(logs.userId) .avg(logs.latency)
97. ALL ABOUT ABSTRACTION ➤ Abstracting cluster management ➤ Abstracting state management ➤ Abstracting workload distribution ➤ Abstracting data sources ➤ Abstracting data sinks
98. UNIFIED COMPUTING ENGINE for large-scale distributed data processing
99. WHEN TO USE APACHE SPARK? ➤ Data could be partitioned ➤ Processing could be parallelized ➤ Data set is big(!) enough
100. HOW TO RUN SPARK
101. HOW TO RUN SPARK ➤ AWS Elastic Map Reduce (EMR) ➤ DataBricks + Community Edition ➤ Inside the code (Local mode) ➤ Docker
102. QUESTIONS?
103. EXTRA
104. GROWTH OF DATA ➤ Moore's law slowed down ➤ Multiple cores, parallel processing, distributed computing ➤ Growth of data ➤ Storage is cheap, processing is not
105. HISTORY OF SPARK ➤ 2009 by Matei Zaharia ➤ 2013 donated to Apache foundation ➤ 2014 top level Apache project ➤ 85% written in Scala
106. ➤ 1 ➤ 2 AWS S3 AUTH sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", ACCESS_KEY) sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", SECRET_KEY) sc.textFile("s3a://%s/.../..." % MOUNT_NAME) sc.textFile("s3a://%s:%s@%s/.../..." % ACCESS_KEY, ENCODED_SECRET_KEY, BUCKET_NAME)
107. DATA SOURCES - SQL DB - JDBC spark .read .jdbc(jdbcUrl, "users", conProps) val jdbcUrl = s"jdbc:mysql://${h}:${p}/${db}" conProps.put("user", s"${u}") conProps.put("password", s"${pwd}")