Using Apache Kafka from Go

At Uber, we use Kafka and Golang a lot. In this talk, I go over the basic concepts in Kafka and the libraries we use to publish and consume messages.

Transcript

1. Using Apache Kafka from Go Nikolay Stoitsev, Software Engineer
2. Trillion+ Messages/Day PBs Data Volume Tens of Thousands Topics
3. Message Message Message Message Message Message Topic
4. Partition - ordered, immutable sequence Message Message Message Message Message Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
5. Offset Message Message Message Message Message Partition 0 Offset: 1
6. Offset Message Message Message Message Message Partition 0 Offset: 2
7. Offset Message Message Message Message Message Partition 0 Offset: 2 Offset: 3 Consumer: 1 Consumer: 2
8. Kafka Cluster
9. Broker Broker Broker
10. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Partitioned and Replicated
11. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Producers Producer Producer
12. Consumer Consumer Consumer Consumer Consumer Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Producers Producer Producer Consumer Group 1 Consumer Group 2
14. Latency vs. Durability
15. At-most-once cluster for logging
16. At-least-one cluster for message bus
17. Publishing to lossy kafka for logging
18. log.Printf("failed to fetch URL %s", url)
19. Structured logging
21. https://github.com/uber-go/zap
22. import "go.uber.org/zap"
23. SugaredLogger
24. Logger
30. Publishing to lossless kafka
31. Avro
32. https://github.com/confluentinc/kafka-rest
33. *net/http.Client
34. Consume from kafka
35. https://github.com/Shopify/sarama
36. AsyncProducer and SyncProducer
39. https://github.com/bsm/sarama-cluster
42. Heartbeat
43. Handles rebalancing
44. Handles coordinator failure
45. Thanks! Nikolay Stoitsev