Spark Streaming + Kafka 0.10: an integration story by Joan Viladrosa Riera at Big Data Spain 2017

Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.

https://www.bigdataspain.org/2017/talk/spark-streaming-kafka-0-10-an-integration-story

Big Data Spain 2017
16th - 17th Kinépolis Madrid

Transcript

2. Joan Viladrosa, Billy Mobile Apache Spark Streaming + Kafka 0.10: An Integration Story #EUstr5
3. About me Joan Viladrosa Riera @joanvr joanviladrosa joan.viladrosa@billymob.com 2 #EUstr5 Degree In Computer Science Advanced Programming Techniques System Interfaces and Integration Co-Founder, Educabits Educational Big data solutions using AWS cloud Big Data Developer, Trovit Hadoop and MapReduce Framework SEM keywords optimization Big Data Architect & Tech Le BillyMobile Full architecture with Hadoop: Kafka, Storm, Hive, HBase, Spark, D
4. Apache Kafka #EUstr5
5. What is Apache Kafka? • Publish - Subscribe Message System 4 #EUstr5
6. What is Apache Kafka? • Publish - Subscribe Message System • Fast • Scalable • Durable • Fault-tolerant What makes it great? 5 #EUstr5
7. What is Apache Kafka? As a central point Producer Producer Producer Producer Kafka Consumer Consumer Consumer Consumer 6 #EUstr5
8. What is Apache Kafka? A lot of different connectors Apache Storm Apache Spark My Java App Logger Kafka Apache Storm Apache Spark My Java App Monitoring Tool 7 #EUstr5
9. Kafka Terminology Topic: A feed of messages Producer: Processes that publish messages to a topic Consumer: Processes that subscribe to topics and process the feed of published messages Broker: Each server of a kafka cluster that holds, receives and sends the actual data 8 #EUstr5
10. Kafka Topic Partitions 0 1 2 3 4 5 6 Partition 0 Partition 1 Partition 2 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 Topic: Old New writes 9 #EUstr5
11. Kafka Topic Partitions 0 1 2 3 4 5 6 Partition 0 7 8 9 Old New 1 0 1 1 1 2 1 3 1 4 1 5 Producer writes Consumer A (offset=6) Consumer B (offset=12) reads reads 10 #EUstr5
12. Kafka Topic Partitions 0 1 2 3 4 5 6 P0 P1 P2 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 0 1 2 3 4 5 6 P3 P4 P5 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 0 1 2 3 4 5 6 P6 P7 P8 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 Broker 1 Broker 2 Broker 3 Consumers & Producers 11 #EUstr5
13. Kafka Topic Partitions 0 1 2 3 4 5 6 P0 P1 P2 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 0 1 2 3 4 5 6 P3 P4 P5 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 0 1 2 3 4 5 6 P6 P7 P8 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 7 8 Broker 1 Broker 2 Broker 3 Consumers & Producers More Storage More Parallelism 12 #EUstr5
14. Kafka Semantics In short: consumer delivery semantics are up to you, not Kafka • Kafka doesn’t store the state of the consumers* • It just sends you what you ask for (topic, partition, offset, length) • You have to take care of your state 13 #EUstr5
15. Apache Kafka Timeline may-2016 nov-2015 nov-2013 nov-2012 New Producer New Consumer Security Kafka Streams Apache Incubator Project 0.7 0.8 0.9 0.10 14 #EUstr5
16. Apache Spark Streaming #EUstr5
17. • Process streams of data • Micro-batching approach What is Apache Spark Streaming? 16 #EUstr5
18. • Process streams of data • Micro-batching approach • Same API as Spark • Same integrations as Spark • Same guarantees & semantics as Spark What makes it great? What is Apache Spark Streaming? 17 #EUstr5
19. What is Apache Spark Streaming? Relying on the same Spark Engine: “same syntax” as batch jobs https://spark.apache.org/docs/latest/streaming-programming-guide.html 18
20. How does it work? • Discretized Streams https://spark.apache.org/docs/latest/streaming-programming-guide.html 19
21. How does it work? • Discretized Streams https://spark.apache.org/docs/latest/streaming-programming-guide.html 20
22. How does it work? 21 https://databricks.com/blog/2015/07/30/diving-into-apache-spark-streamings-execution-model.html
23. How does it work? 22 https://databricks.com/blog/2015/07/30/diving-into-apache-spark-streamings-execution-model.html
24. Spark Streaming Semantics As in Spark: • Not guarantee exactly-once semantics for output actions • Any side-effecting output operations may be repeated • Because of node failure, process failure, etc. So, be careful when outputting to external sources Side effects 23 #EUstr5
25. Spark Streaming Kafka Integration #EUstr5
26. Spark Streaming Kafka Integration Timeline dec-2016 jul-2016 jan-2016 sep-2015 jun-2015 mar-2015 dec-2014 sep-2014 Fault Tolerant WAL + Python API Direct Streams + Python API Improved Streaming UI Metadata in UI (offsets) + Graduated Direct Receivers Native Kafka 0.10 (experimental) 1.1 1.2 1.3 1.4 1.5 1.6 2.0 2.1 25 #EUstr5
27. Kafka Receiver (≤ Spark 1.1) Executor Driver Launch jobs on data Continuously receive data using High Level API Update offsets in ZooKeeper Receiver 26 #EUstr5
28. Kafka Receiver with WAL (Spark 1.2) HDFS Executor Driver Launch jobs on data Continuously receive data using High Level API Update offsets in ZooKeeper WAL Receiver 27 #EUstr5
29. Application Driver Executor Spark Context Jobs Computation checkpointed Receiver Input stream Block metadata Block metadata written to log Block data written both memory + log Streaming Context Kafka Receiver with WAL (Spark 1.2) 28 #EUstr5
30. Kafka Receiver with WAL (Spark 1.2) Restarted Driver Restarted Executor Restarted Spark Context Relaunch Jobs Restart computation from info in checkpoints Restarted Receiver Resend unacked data Recover Block metadata from log Recover Block data from log Restarted Streaming Context 29 #EUstr5
31. Kafka Receiver with WAL (Spark 1.2) HDFS Executor Driver Launch jobs on data Continuously receive data using High Level API Update offsets in ZooKeeper WAL Receiver 30 #EUstr5
32. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor Driver 31 #EUstr5
33. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor Driver 1. Query latest offsets and decide offset ranges for batch 32 #EUstr5
34. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor 1. Query latest offsets and decide offset ranges for batch 2. Launch jobs using offset ranges Driver topic1, p1, (2000, 2100) topic1, p2, (2010, 2110) topic1, p3, (2002, 2102) 33 #EUstr5
35. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor 1. Query latest offsets and decide offset ranges for batch 2. Launch jobs using offset ranges Driver topic1, p1, (2000, 2100) topic1, p2, (2010, 2110) topic1, p3, (2002, 2102) 3. Reads data using offset ranges in jobs using Simple API 34 #EUstr5
36. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor Driver 2. Launch jobs using offset ranges 3. Reads data using offset ranges in jobs using Simple API 1. Query latest offsets and decide offset ranges for batch topic1, p1, (2000, 2100) topic1, p2, (2010, 2110) topic1, p3, (2002, 2102) 35 #EUstr5
37. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor Driver 2. Launch jobs using offset ranges 3. Reads data using offset ranges in jobs using Simple API 1. Query latest offsets and decide offset ranges for batch topic1, p1, (2000, 2100) topic1, p2, (2010, 2110) topic1, p3, (2002, 2102) 36 #EUstr5
38. Direct Kafka Integration w/o Receivers or WALs (Spark 1.3) Executor Driver 2. Launch jobs using offset ranges 3. Reads data using offset ranges in jobs using Simple API 1. Query latest offsets and decide offset ranges for batch 37 #EUstr5
39. Direct Kafka API benefits • No WALs or Receivers • Allows end-to-end exactly- once semantics pipelines * * updates to downstream systems should be idempotent or transactional • More fault-tolerant • More efficient • Easier to use. 38 #EUstr5
40. Spark Streaming UI improvements (Spark 1.4) 39
41. Kafka Metadata (offsets) in UI (Spark 1.5) 40
42. What about Spark 2.0+ and new Kafka Integration? This is why we are here, right? 41 #EUstr5
43. Spark 2.0+ new Kafka Integration spark-streaming-kafka-0-8 spark-streaming-kafka-0-10 Broker Version 0.8.2.1 or higher 0.10.0 or higher Api Stability Stable Experimental Language Support Scala, Java, Python Scala, Java Receiver DStream Yes No Direct DStream Yes Yes SSL / TLS Support No Yes Offset Commit Api No Yes Dynamic Topic Subscription No Yes 42 #EUstr5
44. What’s really New with this New Kafka Integration? • New Consumer API * Instead of Simple API • Location Strategies • Consumer Strategies • SSL / TLS • No Python API :( 43 #EUstr5
45. Location Strategies • New consumer API will pre-fetch messages into buffers • So, keep cached consumers into executors • It’s better to schedule partitions on the host with appropriate consumers 44 #EUstr5
46. Location Strategies - PreferConsistent Distribute partitions evenly across available executors - PreferBrokers If your executors are on the same hosts as your Kafka brokers - PreferFixed Specify an explicit mapping of partitions to hosts 45 #EUstr5
47. Consumer Strategies • New consumer API has a number of different ways to specify topics, some of which require considerable post-object-instantiation setup. • ConsumerStrategies provides an abstraction that allows Spark to obtain properly configured consumers even after restart from checkpoint. 46 #EUstr5
48. Consumer Strategies - Subscribe subscribe to a fixed collection of topics - SubscribePattern use a regex to specify topics of interest - Assign specify a fixed collection of partitions • Overloaded constructors to specify the starting offset for a particular partition. • ConsumerStrategy is a public class that you can 47 #EUstr5
49. SSL/TTL encryption • New consumer API supports SSL • Only applies to communication between Spark and Kafka brokers • Still responsible for separately securing Spark inter-node communication 48 #EUstr5
50. How to use New Kafka Integration on Spark 2.0+ Scala Example Code Basic usage val kafkaParams = Map[String, Object]( "bootstrap.servers" -> "broker01:9092,broker02:9092", "key.deserializer" -> classOf[StringDeserializer], "value.deserializer" -> classOf[StringDeserializer], "group.id" -> "stream_group_id", "auto.offset.reset" -> "latest", "enable.auto.commit" -> (false: java.lang.Boolean) ) val topics = Array("topicA", "topicB") val stream = KafkaUtils.createDirectStream[String, String]( streamingContext, PreferConsistent, Subscribe[String, String](topics, kafkaParams) ) stream.map(record => (record.key, record.value)) 49 #EUstr5
51. How to use New Kafka Integration on Spark 2.0+ Java Example Code Getting metadata st r eam . f or eachRD D { r dd = > val of f set Ranges = r dd. asI nst anceO f [ HasO f f set Ranges] . of f set Ranges r dd. f or eachPart i t i on { i t er = > val osr: O f f set Range = of f set Ranges( TaskCont ext . get . part i t i onI d) / / get any needed dat a f r om t he of f set range val t opi c = osr . t opi c val kaf kaPart i t i onI d = osr . part i t i on val begi n = osr . f r om O f f set val end = osr . unt i l O f f set } } 50 #EUstr5
52. RDD Topic Kafka or Spark RDD Partitions? Kafka Spark 51 1 2 3 4 1 2 3 4
53. RDD Topic Kafka or Spark RDD Partitions? Kafka Spark 52 1 2 3 4 1 2 3 4
54. How to use New Kafka Integration on Spark 2.0+ Java Example Code Getting metadata st r eam . f or eachRD D { r dd = > val of f set Ranges = r dd. asI nst anceO f [ HasO f f set Ranges] . of f set Ranges r dd. f or eachPart i t i on { i t er = > val osr: O f f set Range = of f set Ranges( TaskCont ext . get . part i t i onI d) / / get any needed dat a f r om t he of f set range val t opi c = osr . t opi c val kafkaParti ti onI d = osr . parti ti on val begi n = osr . f r om O f f set val end = osr . unt i l O f f set } } 53 #EUstr5
55. How to use New Kafka Integration on Spark 2.0+ Java Example Code Store offsets in Kafka itself: Commit API st r eam . f or eachRD D { r dd = > val of f set Ranges = r dd. asI nst anceO f [ HasO f f set Ranges] . of f set Ranges / / D O YO UR STUFF w i t h DATA st r eam . asI nst anceO f [ CanCom m i t O f f set s] . com m i t Async( of f set Ranges) } } 54 #EUstr5
56. Kafka + Spark Semantics - At most once - At least once - Exactly once 55 #EUstr5
57. Kafka + Spark Semantics • We don’t want duplicates • Not worth the hassle of ensuring that messages don’t get lost • Example: Sending statistics over UDP 1. Set spark.task.maxFailures to 1 2. Make sure spark.speculation is false (the default) 3. Set Kafka param auto.offset.reset to “largest” 4. Set Kafka param enable.auto.commit to true At most once 56 #EUstr5
58. Kafka + Spark Semantics • This will mean you lose messages on restart • At least they shouldn’t get replayed. • Test this carefully if it’s actually important to you that a message never gets repeated, because it’s not a common use case. At most once 57 #EUstr5
59. Kafka + Spark Semantics • We don’t want to loose any record • We don’t care about duplicates • Example: Sending internal alerts on relative rare occurrences on the stream 1. Set spark.task.maxFailures > 1000 2. Set Kafka param auto.offset.reset to “smallest” 3. Set Kafka param enable.auto.commit to false At least once 58 #EUstr5
60. Kafka + Spark Semantics • Don’t be silly! Do NOT replay your whole log on every restart… • Manually commit the offsets when you are 100% sure records are processed • If this is “too hard” you’d better have a relative short retention log • Or be REALLY ok with duplicates. For example, you are outputting to an external system that handles duplicates for you (HBase) At least once 59 #EUstr5
61. Kafka + Spark Semantics • We don’t want to loose any record • We don’t want duplicates either • Example: Storing stream in data warehouse 1. We need some kind of idempotent writes, or whole-or-nothing writes (transactions) 2. Only store offsets EXACTLY after writing data 3. Same parameters as at least once Exactly once 60 #EUstr5
62. Kafka + Spark Semantics • Probably the hardest to achieve right • Still some small chance of failure if your app fails just between writing data and committing offsets… (but REALLY small) Exactly once 61 #EUstr5
63. Apache Kafka Apacke Spark at Billy Mobile 62 15B records monthly 35T B weekly retention log 6K events/second x4 growth/year
64. Our use cases • Input events from Kafka • Enrich events with some external data sources • Finally store it to Hive We do NOT want duplicates We do NOT want to lose events ETL to Data Warehouse 63
65. Our use cases • Hive is not transactional • Neither idempotent writes • Writing files to HDFS is “atomic” (whole or nothing) • A relation 1:1 from each partition-batch to file in HDFS • Store to ZK the current state of the batch • Store to ZK offsets of last finished batch ETL to Data Warehouse 64
66. Our use cases • Input events from Kafka • Periodically load batch- computed model • Detect when an offer stops converting (or too much) • We do not care about losing some events (on restart) • We always need to process the “real-time” stream Anomalies detector 65
67. Our use cases • It’s useless to detect anomalies on a lagged stream! • Actually it could be very bad • Always restart stream on latest offsets • Restart with “fresh” state Anomalies detector 66
68. Our use cases • Input events from Kafka • Almost no processing • Store it to HBase – (has idempotent writes) • We do not care about duplicates • We can NOT lose a single event Store to Entity Cache 67
69. Our use cases • Since HBase has idempotent writes, we can write events multiple times without hassle • But, we do NOT start with earliest offsets… – That would be 7 days of redundant writes…!!! • We store offsets of last finished batch • But obviously we might re-write some events on restart or failure Store to Entity Cache 68
70. Lessons Learned • Do NOT use checkpointing – Not recoverable across code upgrades – Do your own checkpointing • Track offsets yourself – In general, more reliable: HDFS, ZK, RMDBS... • Memory usually is an issue – You don’t want to waste it – Adjust batchDuration – Adjust maxRatePerPartition 69
71. Further Improvements • Dynamic Allocation spark.dynamicAllocation.enabled vs spark.streaming.dynamicAllocation.enabled https://issues.apache.org/jira/browse/SPARK-12133 But no reference in docs... • Graceful shutdown • Structured Streaming 70
72. Thank you very much! Questions? @joanvr joanviladrosa joan.viladrosa@billymob.com