×
Copy
Open
Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
Using Apache Kafka from Go Nikolay Stoitsev, Software Engineer
Slide 2
Slide 2 text
Trillion+ Messages/Day PBs Data Volume Tens of Thousands Topics
Slide 3
Slide 3 text
Message Message Message Message Message Message Topic
Slide 4
Slide 4 text
Partition - ordered, immutable sequence Message Message Message Message Message Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
Slide 5
Slide 5 text
Offset Message Message Message Message Message Partition 0 Offset: 1
Slide 6
Slide 6 text
Offset Message Message Message Message Message Partition 0 Offset: 2
Slide 7
Slide 7 text
Offset Message Message Message Message Message Partition 0 Offset: 2 Offset: 3 Consumer: 1 Consumer: 2
Slide 8
Slide 8 text
Kafka Cluster
Slide 9
Slide 9 text
Broker Broker Broker
Slide 10
Slide 10 text
Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Partitioned and Replicated
Slide 11
Slide 11 text
Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Producers Producer Producer
Slide 12
Slide 12 text
Consumer Consumer Consumer Consumer Consumer Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Producers Producer Producer Consumer Group 1 Consumer Group 2
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Latency vs. Durability
Slide 15
Slide 15 text
At-most-once cluster for logging
Slide 16
Slide 16 text
At-least-one cluster for message bus
Slide 17
Slide 17 text
Publishing to lossy kafka for logging
Slide 18
Slide 18 text
log.Printf("failed to fetch URL %s", url)
Slide 19
Slide 19 text
Structured logging
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
https://github.com/uber-go/zap
Slide 22
Slide 22 text
import "go.uber.org/zap"
Slide 23
Slide 23 text
SugaredLogger
Slide 24
Slide 24 text
Logger
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
Publishing to lossless kafka
Slide 31
Slide 31 text
Avro
Slide 32
Slide 32 text
https://github.com/confluentinc/kafka-rest
Slide 33
Slide 33 text
*net/http.Client
Slide 34
Slide 34 text
Consume from kafka
Slide 35
Slide 35 text
https://github.com/Shopify/sarama
Slide 36
Slide 36 text
AsyncProducer and SyncProducer
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
https://github.com/bsm/sarama-cluster
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
Heartbeat
Slide 43
Slide 43 text
Handles rebalancing
Slide 44
Slide 44 text
Handles coordinator failure
Slide 45
Slide 45 text
Thanks! Nikolay Stoitsev