Samza in LinkedIn: How LinkedIn Processes Billions of Events Everyday in Real-time

Slide 1

Slide 1 text

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events every day

Slide 2

Slide 2 text

Neha Narkhede ¨  Co-founder and Head of Engineering @ Stealth Startup ¨  Lead, Streams Infrastructure @ LI (Kafka & Samza) ¨  Apache Kafka committer and PMC member ¨  Reach out at @nehanarkhede

Slide 3

Slide 3 text

Agenda ¨  Real-time Data Integration ¨  Introduction to Logs & Apache Kafka ¨  Logs & Stream processing ¨  Apache Samza ¨  Stateful stream processing

Slide 4

Slide 4 text

Agenda ¨ Real-time Data Integration ¨  Introduction to Logs & Apache Kafka ¨  Logs & Stream processing ¨  Apache Samza ¨  Stateful stream processing

Slide 5

Slide 5 text

Increase in diversity of data 1980+ 2000+ 2010+ Siloed data feeds Database data (users, products, orders etc) IoT sensors Events (clicks, impressions, pageviews) Application logs (errors, service calls) Application metrics (CPU usage, requests/sec)

Slide 6

Slide 6 text

Explosion in diversity of systems ¨  Live Systems ¤  Voldemort ¤  Espresso ¤  GraphDB ¤  Search ¤  Samza ¨  Batch ¤  Hadoop ¤  Teradata

Slide 7

Slide 7 text

Data integration disaster Oracle Oracle Oracle User Tracking Hadoop Log Search Monitoring Data Warehous e Social Graph Rec. Engine Search Email Voldemort Voldemort Voldemort Espresso Espresso Espresso Logs Operational Metrics Production Services ... Security

Slide 8

Slide 8 text

Centralized service Oracle Oracle Oracle User Tracking Hadoop Log Search Monitorin g Data Warehous e Social Graph Rec Engine & Life Search Email Voldemort Voldemort Voldemort Espresso Espresso Espresso Logs Operational Metrics Production Services ... Security Data Pipeline

Slide 9

Slide 9 text

Agenda ¨  Real-time Data Integration ¨ Introduction to Logs & Apache Kafka ¨  Logs & Stream processing ¨  Apache Samza ¨  Stateful stream processing

Slide 10

Slide 10 text

Kafka at 10,000 ft ¨  Distributed from ground up ¨  Persistent ¨  Multi-subscriber Cluster of brokers Producer Producer Producer Producer Producer Producer Producer Consumer Producer Consumer Producer Consumer

Slide 11

Slide 11 text

Key design principles ¨  Scalability of a file system ¤  Hundreds of MB/sec/server throughput ¤  Many TBs per server ¨  Guarantees of a database ¤  Messages strictly ordered ¤  All data persistent ¨  Distributed by default ¤  Replication model ¤  Partitioning model

Slide 12

Slide 12 text

Apache Kafka @ LinkedIn ¨  175 TB of in-flight log data per colo ¨  Low-latency: ~1.5ms ¨  Replicated to each datacenter ¨  Tens of thousands of data producers ¨  Thousands of consumers ¨  7 million messages written/sec ¨  35 million messages read/sec ¨  Hadoop integration

Slide 13

Slide 13 text

The data structure every systems engineer should know Logs

Slide 14

Slide 14 text

The Log ¨  Ordered ¨  Append only ¨  Immutable 0 1 2 3 4 5 6 7 8 9 10 11 12 1st record next record written

Slide 15

Slide 15 text

The Log: Partitioning 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 0 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 1 Partition 2 13 14 15 16

Slide 16

Slide 16 text

Logs: pub/sub done right 0 1 2 3 4 5 6 7 8 9 10 11 12 writes Data source Destination system A (time = 7) Destination system B (time = 11) reads reads

Slide 17

Slide 17 text

Logs for data integration User updates proﬁle with new job Newsfeed KAFKA Search Hadoop Standardization engine

Slide 18

Slide 18 text

Agenda ¨  Real-time Data Integration ¨  Introduction to Logs & Apache Kafka ¨ Logs & Stream processing ¨  Apache Samza ¨  Stateful stream processing

Slide 19

Slide 19 text

Stream processing = f(log) Log A Job 1 Log B

Slide 20

Slide 20 text

Stream processing = f(log) Log A Job 1 Job 2 Log B Log C Log D Log E

Slide 21

Slide 21 text

Apache Samza at LinkedIn User updates proﬁle with new job Newsfeed KAFKA Search Hadoop Standardization engine

Slide 22

Slide 22 text

Latency spectrum of data systems Synchronous (milliseconds) RPC Batch (Hours) Latency Asynchronous processing (seconds to minutes)

Slide 23

Slide 23 text

Agenda ¨  Real-time Data Integration ¨  Introduction to Logs & Apache Kafka ¨  Logs & Stream processing ¨ Apache Samza ¨  Stateful stream processing

Slide 24

Slide 24 text

Samza API public interface StreamTask { void process (IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator); } getKey(), getMsg() sendMsg(topic, key, value) commit(), shutdown()

Slide 25

Slide 25 text

Samza Architecture (Logical view) Task 1 Task 2 Task 3 Log A Log B partition 0 partition 1 partition 2 partition 0 partition 1

Slide 26

Slide 26 text

Samza Architecture (Logical view) Task 1 Task 2 Task 3 Log A Log B partition 0 partition 1 partition 2 partition 0 partition 1 Samza container 1 Samza container 2

Slide 27

Slide 27 text

Samza Architecture (Physical view) Samza container 1 Samza container 2 Host 1 Host 2

Slide 28

Slide 28 text

Samza Architecture (Physical view) Samza container 1 Samza container 2 Host 1 Host 2 Samza YARN AM Node manager Node manager

Slide 29

Slide 29 text

Samza Architecture (Physical view) Samza container 1 Samza container 2 Host 1 Host 2 Samza YARN AM Node manager Node manager Kafka Kafka

Slide 30

Slide 30 text

Map Reduce Map Reduce YARN AM Node manager Node manager HDFS HDFS Host 1 Host 2 Samza Architecture: Equivalence to Map Reduce

Slide 31

Slide 31 text

M/R Operation Primitives ¨  Filter records matching some condition ¨  Map record = f(record) ¨  Join Two/more datasets by key ¨  Group records with same key ¨  Aggregate f(records within the same group) ¨  Pipe job 1’s output => job 2’s input

Slide 32

Slide 32 text

M/R Operation Primitives on streams ¨  Filter records matching some condition ¨  Map record = f(record) ¨  Join Two/more datasets by key ¨  Group records with same key ¨  Aggregate f(records within the same group) ¨  Pipe job 1’s output => job 2’s input Requires state maintenance

Slide 33

Slide 33 text

Agenda ¨  Real-time Data Integration ¨  Introduction to Logs & Apache Kafka ¨  Logs & Stream processing ¨  Apache Samza ¨ Stateful stream processing

Slide 34

Slide 34 text

Example: Newsfeed User 567 posted "Hello World" Status update log Fan out messages to followers Push notiﬁcation log 567 -> [123, 679, 789, ...] 999 -> [156, 343, ... ] User 989 posted "Blah Blah" User ... posted "..." External connection DB Refresh user 123's newsfeed Refresh user 679's newsfeed Refresh user ...'s newsfeed

Slide 35

Slide 35 text

Disk 100-500K msg/sec/node 100-500K msg/sec/node 1-5K queries/sec ?? ex: Cassandra, MongoDB, etc Remote state Samza task partition 0 Samza task partition 1 Local state vs Remote state: Remote ❌  Performance ❌  Isolation ❌  Limited APIs

Slide 36

Slide 36 text

Local LevelDB/RocksDB Samza task partition 0 Samza task partition 1 Local LevelDB/RocksDB Local state: Bring data closer to computation

Slide 37

Slide 37 text

Local LevelDB/RocksDB Samza task partition 0 Samza task partition 1 Local LevelDB/RocksDB Local state: Bring data closer to computation Disk Change log stream

Slide 38

Slide 38 text

Example Revisited: Newsfeed User 567 posted "Hello World" Status update log New connection log Fan out messages to followers Push notiﬁcation log 567 -> [123, 679, 789, ...] 999 -> [156, 343, ... ] User 123 followed 567 User 890 followed 234 User ... followed ... User 989 posted "Blah Blah" User ... posted "..." Refresh user 123's newsfeed Refresh user 679's newsfeed Refresh user ...'s newsfeed

Slide 39

Slide 39 text

Fault tolerance? Samza container 1 Samza container 2 Host 1 Host 2 Samza YARN AM Node manager Node manager Kafka Kafka

Slide 40

Slide 40 text

Local LevelDB/RocksDB Samza task partition 0 Samza task partition 1 Local LevelDB/RocksDB Durable change log Fault tolerance in Samza

Slide 41

Slide 41 text

Slow jobs Log A Job 1 Job 2 Log B Log C Log D Log E ❌  Drop data ❌  Backpressure ❌  Queue ❌ In memory ✅ On disk (KAFKA)

Slide 42

Slide 42 text

Summary ¨  Real time data integration is crucial for the success and adoption of stream processing ¨  Logs form the basis for real time data integration ¨  Stream processing = f(logs) ¨  Samza differentiator => performance & fault- tolerant stateful stream processing

Slide 43

Slide 43 text

Thank you! ¨  Logs ¤  http://bit.ly/the_log ¨  Apache Kafka ¤  http://kafka.apache.org ¨  Apache Samza ¤  http://samza.incubator.apache.org ¨  Me ¤  @nehanarkhede ¤  http://www.linkedin.com/in/nehanarkhede